METHOD FOR DETERMINING CORRESPONDENCES BETWEEN A FIRST AND A SECOND IMAGE, AND METHOD FOR DETERMINING THE POSE OF A CAMERA

• US 20140037140A1
• Filed: 01/27/2011
• Published: 02/06/2014
• Est. Priority Date: 01/27/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Metaio GmbH (Apple Inc.)
• Inventors: Ulbricht, Daniel, Benhimane, Selim
• Granted Patent: US 9,235,894 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/757 Matching configurations of points or features

Abstract

A method for determining correspondences between a first and a second image, comprising the steps providing a first image and a second image of the real environment, defining a warping function between the first and second image, determining the parameters of the warping function between the first image and the second image by means of an image registration method, determining a third image by applying the warping function with the determined parameters to the first image, determining a matching result by matching the third image and the second image, and determining correspondences between the first and the second image using the matching result and the warping function with the determined parameters. The method may be used in a keyframe based method for determining the pose of a camera based on the determined correspondences.

Claims

1. A method for determining correspondences between a first and a second image, comprising the steps of:

a) providing a first image and a second image of the real environment;

b) defining a warping function between the first and second image;

c) determining the parameters of the warping function between the first image and the second image by means of an image registration method;

d) determining a third image by applying the warping function with the determined parameters to the first image;

e) determining a matching result by matching the third image and the second image; and

f) determining correspondences between the first and the second image using the matching result and the warping function with the determined parameters.

2. The method according to claim 1, wherein the definition of the warping function is based on an assumption about the geometry of the real environment.

3. The method according to claim 2, wherein the geometry of the real environment is assumed to be planar.

4. The method according to claim 1, wherein the definition of the warping function is based on changes in the real environment between capturing the first image and capturing the second image.

5. The method according to claim 2, wherein the assumption about the geometry of the real environment and/or the changes in the real environment between capturing the first image and the second image are acquired from at least one of an environment model, a three-dimensional reconstruction method, a time-of-flight camera, stereovision and structured-light-based approach.

6. The method according to claim 1, wherein the definition of the warping function is based on an assumed camera motion between capturing the first image and capturing the second image.

7. The method according to claim 1, wherein the definition of the warping function is based on illumination changes in the real environment or in the image between the first image and the second image.

8. The method according to claim 1, wherein the image registration method is selected according to the definition of the warping function.

9. The method according to claim 1, wherein the image registration method is based on an iterative minimization process, wherein a first set of pixels in the first image is compared with a computed set of pixels in the second image and the computed set of pixels in the second image used for the comparison varies at each iteration.

10. The method according to claim 9, wherein the comparison in the image registration method is based on image intensity differences.

11. The method according to claim 9, wherein an initial estimate of the parameters of the warping function is provided to the iterative minimization process.

12. The method according to claim 11, wherein the initial estimate of the parameters of the warping function is determined using an estimate of camera motion between capturing the first image and capturing the second image.

13. The method according to claim 1, wherein step e) is further comprising:

extracting features from the third image;

extracting features from the second image; and

determining the matching result by matching the extracted features from the third and the second image.

14. The method according to claim 13, further comprising:

providing initial correspondences between the first image and the second image; and

determining the matching result only for the extracted features that are not present in the initial correspondences.

15. The method according to claim 14, wherein the initial correspondences is determined using an image matching method.

19. A computer program product adapted to be loaded into the internal memory of a digital computer and comprising software code sections by means of which the method according to claim 1, is performed when said product is running on said computer.

20. The method according to claim 4, wherein the assumption about the geometry of the real environment and/or the changes in the real environment between capturing the first image and the second image are acquired from at least one of an environment model, a three-dimensional reconstruction method, a time-of-flight camera, stereovision and structured-light-based approach.

16. A method for determining the pose of a camera, comprising the steps of:

a) providing at least one reference image and a reference pose of the camera in a common coordinate system while capturing the at least one reference image;

b) capturing a current image;

c) determining correspondences between the current image and the at least one reference image, wherein the first image is set to be the reference image or the current image, and the second image is set to be the current image or the reference image, respectively; and

d) determining the pose of the camera based on the determined correspondences.

17. The method according to claim 16, wherein step b) is further comprising the step of providing an initial estimate of the current pose of the camera in the common coordinate system while capturing the current image, and in step c) the current image is matched with at least one reference image selected from a set of reference images, wherein the selection is based on the distance of the reference pose to the estimate of the current pose.

18. The method according to claim 17, wherein the initial estimate of the pose of the camera is acquired by using at least one of an optical tracking method, inertial sensor information and stereo vision.