Method for estimating a camera motion and for determining a three-dimensional model of a real environment

• US 9,525,862 B2
• Filed: 08/31/2011
• Issued: 12/20/2016
• Est. Priority Date: 08/31/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Metaio GmbH (Apple Inc.)
• Inventors: Benhimane, Selim, Lieberknecht, Sebastian, Huber, Andrea
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): Duley, Janese
• Application Number:
  US14/240,995
• Publication Number:
  US 20140293016A1
• Time in Patent Office: 1,938 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects

Abstract

A method for estimating a camera motion and for determining a three-dimensional model of an environment is provided that includes the steps of: providing intrinsic parameters of a camera; providing a set of reference two-dimensional imaged points captured by the camera at a first camera pose and reference depth samples; determining a three-dimensional model of the environment; providing a set of current two-dimensional imaged points captured by the camera at a second camera pose and current depth samples associated to the set of current two-dimensional imaged points and determining a current three-dimensional model; estimating a camera motion between the first camera pose and the second camera pose; determining a similarity measure between the three-dimensional model and the current three-dimensional model, and if it is determined that the similarity measure meets a first condition, updating the three-dimensional model of the environment and adding the set of current two-dimensional imaged points to the set of reference two-dimensional imaged points.

Claims

1. A method for estimating a camera motion and for determining a three-dimensional model of an environment, comprising the steps of:

obtaining a set of reference two-dimensional imaged points captured by a camera at a first camera pose and reference depth samples associated to the set of reference two-dimensional imaged points;

determining a three-dimensional model of the environment using the reference depth samples and intrinsic parameters of the camera;

providing a set of current two-dimensional imaged points captured by the camera at a second camera pose and current depth samples associated to the set of current two-dimensional imaged points and determining a current three-dimensional model using the current depth samples and the intrinsic parameters;

estimating a camera motion between the first camera pose and the second camera pose by means of an image registration method;

determining based on the estimated camera motion a similarity measure between the three-dimensional model and the current three-dimensional model;

in response to determining that the similarity measure meets a first condition, updating the three-dimensional model of the environment using at least one point of the current three-dimensional model; and

in response to updating the three-dimensional model using at least one point of the current three-dimensional model, adding the set of current two-dimensional imaged points to the set of reference two-dimensional imaged points.

2. The method according to claim 1, further comprising repeating providing a set of current two-dimensional points, estimating, and determining a similarity measure, using the three-dimensional model and the set of reference two-dimensional imaged points resulting from a previous iteration of providing a set of current two-dimensional points, estimating, and determining a similarity measure.

3. The method according to claim 2, further comprising recording the similarity measure data over time and if it is determined that the similarity measure data in a defined time frame meets a second condition, the three-dimensional model is no longer updated and the current set of two-dimensional imaged points are no longer added to the reference set of two-dimensional imaged points.

4. The method according to claim 2, further comprising determining a frequency of the similarity measure meeting the first condition and if it is determined that the frequency of updates falls below a defined threshold, the three-dimensional model is no longer updated and the current set of two-dimensional imaged points are no longer added to the reference set of two-dimensional imaged points.

5. The method according to claim 1, wherein determining the similarity measure further comprises rendering the three-dimensional model into a coordinate system of the current three-dimensional model and determining the similarity measure between the three-dimensional model and the current three-dimensional model.

6. The method according to claim 1, wherein the similarity measure is indicative of an overlap between the three-dimensional model and the current three-dimensional model.

7. The method according to claim 1, further comprising determining whether an associated three-dimensional point exists in the three-dimensional model for at least one point in the current three-dimensional model.

8. The method according to claim 7, further comprising determining whether a difference between the depth of the at least one point in the current three-dimensional model and the depth of the determined associated three-dimensional point in the three-dimensional model is above a defined threshold.

9. The method according to claim 7, wherein the similarity measure is defined such that a number of points in the current three-dimensional model for which no associated three-dimensional point in the three-dimensional model is determined must exceed a certain threshold.

10. The method according to claim 7, wherein only points in the current three-dimensional model for which no associated three-dimensional point in the three-dimensional model is determined are used for updating the three-dimensional model.

11. The method according to claim 1, further comprising:

determining, based on the estimated camera motion, a depth of a three-dimensional point in the three-dimensional model and the depth of the three-dimensional point from the current depth samples associated to the set of current two-dimensional imaged points;

determining a refined depth of the three-dimensional point from the depth of the three-dimensional point in the three-dimensional model and the depth of the associated three-dimensional point in the current three-dimensional model; and

updating the depth of the three-dimensional point in at least one of the current three-dimensional point cloud, and in the three-dimensional model with the determined refined depth.

16. The system according to claim 1, wherein at least one of the current depth samples is discarded if it is determined that the depth value is above a defined threshold.

17. The system according to claim 1, wherein the image registration method is feature-based or intensity-based.

18. The system according to claim 17, wherein in the feature-based image registration method, two-dimensional image features from the set of reference two-dimensional imaged points are extracted and the camera motion is estimated by determining two-dimensional correspondences based on the extracted image features.

19. The system according to claim 1, wherein at least one of the three-dimensional model and current three-dimensional model further comprise additional information which is determined from at least one of the set of reference two-dimensional imaged points and the set of current two-dimensional imaged points using the intrinsic parameters.

20. The system according to claim 19, wherein the additional information is determined by associating the color of at least a corresponding reference two-dimensional imaged point to the three-dimensional model.

12. A system for estimating a camera motion and for determining a three-dimensional model of an environment, comprising:

a processor; and

a memory coupled to the processor and comprising computer code which, when executed by the processor, causes the system to;

obtain a set of reference two-dimensional imaged points captured by a camera at a first camera pose and reference depth samples associated to the set of reference two-dimensional imaged points;

determine a three-dimensional model of the environment using the reference depth samples and intrinsic parameters of the camera;

provide a set of current two-dimensional imaged points captured by the camera at a second camera pose and current depth samples associated to the set of current two-dimensional imaged points and determining a current three-dimensional model using the current depth samples and the intrinsic parameters;

estimate a camera motion between the first camera pose and the second camera pose by means of an image registration method;

determine based on the estimate of the camera motion a similarity measure between at least one three-dimensional point in the three-dimensional model and an associated three-dimensional point in the current three-dimensional model;

in response to determining that the similarity measure meets a first condition, updating the three-dimensional model of the environment using at least one point of the current three-dimensional model; and

in response to updating the three-dimensional model using at least one point of the current three-dimensional model, adding the set of current two-dimensional imaged points to the set of reference two-dimensional imaged points.

13. The system according to claim 12, wherein the associated three-dimensional point in the current three-dimensional model is determined based on the estimated camera motion.

14. The system according to claim 12, wherein the similarity measure is defined as a difference between a depth of the three-dimensional point in the three-dimensional model and a depth of the associated three-dimensional point in the current three-dimensional model.

15. The system according to claim 12, wherein a refined depth of the three-dimensional point is determined from the depth of the three-dimensional point in the three-dimensional model and the depth of the associated three-dimensional point in the current three-dimensional model and is used for updating the depth of the three-dimensional point in at least one of the current three-dimensional model and in the three dimensional model.

21. A non-transitory computer readable medium comprising software code sections which are adapted to

obtain a set of reference two-dimensional imaged points captured by a camera at a first camera pose and reference depth samples associated to the set of reference two-dimensional imaged points;

determine a three-dimensional model of the environment using the reference depth samples and intrinsic parameters of the camera;

provide a set of current two-dimensional imaged points captured by the camera at a second camera pose and current depth samples associated to the set of current two-dimensional imaged points and determining a current three-dimensional model using the current depth samples and the intrinsic parameters;

estimate a camera motion between the first camera pose and the second camera pose by means of an image registration method;

determine based on the estimated camera motion a similarity measure between the three-dimensional model and the current three-dimensional model;

in response to determining that the similarity measure meets a first condition, updating the three-dimensional model of the environment using at least one point of the current three-dimensional model; and

in response to updating the three-dimensional model using at least one point of the current three-dimensional model, adding the set of current two-dimensional imaged points to the set of reference two-dimensional imaged points.