METHOD FOR ESTIMATING A CAMERA MOTION AND FOR DETERMINING A THREE-DIMENSIONAL MODEL OF A REAL ENVIRONMENT

• US 20140293016A1
• Filed: 08/31/2011
• Published: 10/02/2014
• Est. Priority Date: 08/31/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Metaio GmbH (Apple Inc.)
• Inventors: Lieberknecht, Sebastian, Benhimane, Selime, Huber, Andrea
• Granted Patent: US 9,525,862 B2
• Time in Patent Office: Days
• US Class Current: 348/50
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects

Abstract

A method for estimating a camera motion and for determining a three-dimensional model of an environment is provided that includes the steps of: providing intrinsic parameters of a camera; providing a set of reference two-dimensional imaged points captured by the camera at a first camera pose and reference depth samples; determining a three-dimensional model of the environment; providing a set of current two-dimensional imaged points captured by the camera at a second camera pose and current depth samples associated to the set of current two-dimensional imaged points and determining a current three-dimensional model; estimating a camera motion between the first camera pose and the second camera pose; determining a similarity measure between the three-dimensional model and the current three-dimensional model, and if it is determined that the similarity measure meets a first condition, updating the three-dimensional model of the environment and adding the set of current two-dimensional imaged points to the set of reference two-dimensional imaged points.

Claims

1. A method for estimating a camera motion and for determining a three-dimensional model of an environment, comprising the steps of:

a. providing intrinsic parameters of a camera;

b. providing a set of reference two-dimensional imaged points captured by the camera at a first camera pose and reference depth samples associated to the set of reference two-dimensional imaged points;

c. determining a three-dimensional model of the environment using the reference depth samples and the intrinsic parameters;

d. providing a set of current two-dimensional imaged points captured by the camera at a second camera pose and current depth samples associated to the set of current two-dimensional imaged points and determining a current three-dimensional model using the current depth samples and the intrinsic parameters;

e. estimating a camera motion between the first camera pose and the second camera pose by means of an image registration method;

f. determining based on the estimated camera motion a similarity measure between the three-dimensional model and the current three-dimensional model, and

if it is determined that the similarity measure meets a first condition, updating the three-dimensional model of the environment using at least one point of the current three-dimensional model and adding the set of current two-dimensional imaged points to the set of reference two-dimensional imaged points.

2. The method according to claim 1, further comprising repeating steps d) to f) using the three-dimensional model and the set of reference two-dimensional imaged points resulting from a previous iteration of steps d) to f).

3. The method according to claim 2, further comprising recording the similarity measure and/or the variance of the similarity measure over time and if it is determined that the similarity measure and/or the variance of the similarity measure in a defined time frame meets a second condition, the three-dimensional model is no longer updated and the current set of two-dimensional imaged points are no longer added to the reference set of two-dimensional imaged points.

4. The method according to claim 2, further comprising determining the frequency of the similarity measure meeting the first condition and if it is determined that the frequency of updates falls below a defined threshold, the three-dimensional model is no longer updated and the current set of two-dimensional imaged points are no longer added to the reference set of two-dimensional imaged points.

5. The method according to claim 1, wherein step f) further comprises rendering the three-dimensional model into the coordinate system of the current three-dimensional model and determining the similarity measure between the three-dimensional model and the current three-dimensional model.

6. The method according to claim 1, wherein the similarity measure is indicative of the overlap between the three-dimensional model and the current three-dimensional model.

7. The method according to claim 1, wherein for at least one point in the current three-dimensional model it is determined whether an associated three-dimensional point exists in the three-dimensional model.

8. The method according to claim 7, wherein it is determined whether the difference between the depth of the point in the current three-dimensional model and the depth of the determined associated three-dimensional point in the three-dimensional model is above a defined threshold.

9. The method according to claim 7, wherein the similarity measure is defined such that the number of points in the current three-dimensional model for which no associated three-dimensional point in the three-dimensional model is determined must exceed a certain threshold.

10. The method according to claim 7, wherein only points in the current three-dimensional model for which no associated three-dimensional point in the three-dimensional model is determined are used for updating the three-dimensional model.

11. The method according to claim 1, further comprising:

determining based on the estimated camera motion the depth of a three-dimensional point in the three-dimensional model and the depth of that three-dimensional point from the current depth samples associated to the set of current two-dimensional im-aged points;

determining a refined depth of that three-dimensional point from the depth of the three-dimensional point in the three-dimensional model and the depth of the associated three-dimensional point in the current three-dimensional model; and

updating the depth of the three-dimensional point in the current three-dimensional point cloud and/or in the three-dimensional model with the determined refined depth.

16. The method according to claim 1, wherein at least one of the current depth samples is discarded if it is determined that the depth value is above a defined threshold.

17. The method according to claim 1, wherein the image registration method is feature-based or intensity-based.

18. The method according to claim 17, wherein in the feature-based image registration method two-dimensional image features from the set of reference two-dimensional imaged points are extracted and the camera motion is estimated by determining two-dimensional correspondences based on the extracted image features.

19. The method according to claim 1, wherein the three-dimensional model and/or current three-dimensional model further comprise additional information which is determined from the set of reference two-dimensional imaged points and/or set of current two-dimensional imaged points using the intrinsic parameters.

20. The method according to claim 19, wherein the additional information is determined by associating the colour of at least a corresponding reference two-dimensional imaged point to the three-dimensional model.

12. A method for estimating a camera motion and for determining a three-dimensional model of an environment, comprising the steps of:

a. providing intrinsic parameters of a camera;

b. providing a set of reference two-dimensional imaged points captured by the camera at a first camera pose and reference depth samples associated to the set of reference two-dimensional imaged points;

c. determining a three-dimensional model of the environment using the reference depth samples and the intrinsic parameters;

d. providing a set of current two-dimensional imaged points captured by the camera at a second camera pose and current depth samples associated to the set of current two-dimensional imaged points and determining a current three-dimensional model using the current depth samples and the intrinsic parameters;

e. estimating a camera motion between the first camera pose and the second camera pose by means of an image registration method;

f. determining based on the estimate of the camera motion a similarity measure between at least one three-dimensional point in the three-dimensional model and an associated three-dimensional point in the current three-dimensional model, and

if it is determined that the similarity measure meets a first condition, refining the current three-dimensional model and/or the three-dimensional model.

13. The method according to claim 12, wherein the associated three-dimensional point in the current three-dimensional model is determined based on the estimated camera motion.

14. The method according to claim 12, wherein the similarity measure is defined as a difference between the depth of the three-dimensional point in the three-dimensional model and the depth of the associated three-dimensional point in the current three-dimensional model.

15. The method according to claim 12, wherein a refined depth of the three-dimensional point is determined from the depth of the three-dimensional point in the three-dimensional model and the depth of the associated three-dimensional point in the current three-dimensional model and is used for updating the depth of the three-dimensional point in the current three-dimensional model and/or in the three dimensional model.

21. A non-transitory computer readable medium comprising software code sections which are adapted to perform a method for estimating a camera motion and for determining a three-dimensional model of an environment, comprising the steps of:

a. providing intrinsic parameters of a camera;

b. providing a set of reference two-dimensional imaged points captured by the camera at a first camera pose and reference depth samples associated to the set of reference two-dimensional imaged points;

g. determining a three-dimensional model of the environment using the reference depth samples and the intrinsic parameters;

h. providing a set of current two-dimensional imaged points captured by the camera at a second camera pose and current depth samples associated to the set of current two-dimensional imaged points and determining a current three-dimensional model using the current depth samples and the intrinsic parameters;

i. estimating a camera motion between the first camera pose and the second camera pose by means of an image registration method;

j. determining based on the estimated camera motion a similarity measure between the three-dimensional model and the current three-dimensional model, and

if it is determined that the similarity measure meets a first condition, updating the three-dimensional model of the environment using at least one point of the current three-dimensional model and adding the set of current two-dimensional imaged points to the set of reference two-dimensional imaged points.