Method of matching image features with reference features

• US 9,400,941 B2
• Filed: 08/31/2011
• Issued: 07/26/2016
• Est. Priority Date: 08/31/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Metaio GmbH (Apple Inc.)
• Inventors: Kurz, Daniel, Meier, Peter
• Primary Examiner(s): Chawan, Sheela C
• Application Number:
  US14/240,988
• Publication Number:
  US 20150161476A1
• Time in Patent Office: 1,791 Days
• Field of Search: 382/190, 382/100, 382/141, 382/145, 382/181, 382/209, 382/212, 382/214, 382/304, 382/276, 382/302, 382/103, 382/278, 382/107
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/30181 Earth observation
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]

Abstract

A method of matching image features with reference features comprises the steps of providing a current image, providing a set of reference features, wherein each of the reference features comprises at least one first parameter which is at least partially indicative of a position and/or orientation of the reference feature with respect to a global coordinate system, wherein the global coordinate system is an earth coordinate system or an object coordinate system, or at least partially indicative of a position of the reference feature with respect to an altitude, detecting at least one feature in the current image in a feature detection process, associating with the detected feature at least one second parameter which is at least partially indicative of a position and/or orientation of the detected feature, or which is at least partially indicative of a position of the detected feature with respect to an altitude, and matching the detected feature with a reference feature by determining a similarity measure.

Claims

1. A method of matching image features with reference features, comprising:

providing a current image captured by a capturing device;

providing a set of reference features, wherein each of the reference features comprises at least one first parameter which is at least partially indicative of a position and/or orientation of the reference feature with respect to a global coordinate system, wherein the global coordinate system is an earth coordinate system or an object coordinate system, or which is at least partially indicative of a position of the reference feature with respect to an altitude;

detecting at least one feature in the current image in a feature detection process;

associating, with the detected feature, at least one second parameter which is at least partially indicative of a position and/or orientation of the detected feature with respect to the global coordinate system, or which is at least partially indicative of a position of the detected feature with respect to an altitude; and

matching the detected feature with at least one of the reference features of the set of reference features by determining a similarity measure between the at least one first parameter and the at least one second parameter.

2. The method according to claim 1, including defining a search space with a reduced number of reference features within the set of reference features when matching the detected feature, wherein the search space is determined based on the at least one second parameter.

3. The method according to claim 1, wherein the global coordinate system provides information about a geo-referenced position and/or orientation of a feature.

4. The method according to claim 1, wherein at least one of the at least one first and second parameters is at least partially indicative of a longitude and a latitude.

5. The method according to claim 1, wherein at least one of the at least one first and second parameters is at least partially indicative of a position with respect to a relative altitude relative to the capturing device, or an absolute altitude.

6. The method according to claim 1, wherein the position and/or orientation of the detected feature is described by an equation that describes a geometric shape, wherein the geometric shape comprises at least one of a point, a ray, a straight line, a circle, a cone, or a cylinder.

7. The method according to claim 6, wherein matching the detected feature with the set of reference features comprises only considering those reference features of the set of reference features as possible matches that have a position whose distance to the geometric shape is below a threshold.

8. The method according to claim 1, wherein at least one of the at least one first and second parameters is part of a spatial data structure in a feature descriptor domain and/or in a spatial domain, wherein the spatial data structure includes a bounding volume, such as axis-aligned boxes, oriented boxes, spheres, arbitrary volumes, or a grid, or any kind of tree, such as Quadtrees, Octrees, BSP-trees, kd-trees, R-trees.

9. The method according to claim 1, wherein the at least one second parameter is determined using an at least partial position and/or orientation of the capturing device with respect to the global coordinate system when capturing the current image.

10. The method according to claim 9, wherein the at least one second parameter is determined by further using a set of intrinsic parameters of the capturing device and a position of the detected feature in the current image.

11. The method according to claim 1, wherein the at least one second parameter is determined using a depth of at least one element in the current image being a part of the detected feature.

12. The method according to claim 1, wherein the at least one second parameter is determined using a measurement of a gravity vector in a coordinate system associated to the capturing device and a depth of at least one element in the current image being a part of the detected feature.

13. The method according to claim 12, wherein the position of the detected feature is at least partially described by a plane having a normal being or corresponding to the gravity vector, with the plane indicative of the altitude.

14. The method according to claim 1, further comprising calculating a 3-dimensional position of the detected feature with respect to the global coordinate system or a coordinate system of the capturing device by combining a determined distance between the detected feature and the capturing device and the orientation of the capturing device.

15. The method according to claim 1, wherein the set of reference features is taken from at least one reference image which has been recorded with a second capturing device different from the capturing device.

16. The method of claim 15, wherein the capture time of the at least one reference image is at least one day older than the capture time of the current image.

17. The method according to claim 1, further comprising providing at least a partial position, orientation and/or altitude of the capturing device, thereby deriving an uncertainty information associated with the at least one second parameter of the detected feature.

18. The method according to claim 17, wherein the uncertainty information is used to influence the matching process.

19. The method according to claim 18, wherein the uncertainty information influences the matching process by changing a respective weight of at least one of one or more individual degrees of freedom regarding the position and/or orientation depending on the uncertainty.

20. The method according to claim 18, wherein the uncertainty information influences the selection of reference features as matching candidates to be matched.

21. The method according to claim 20, further providing a weight based on which the selection of reference features as matching candidates is influenced, wherein a higher uncertainty decreases the weight and a lower uncertainty increases the weight.

22. The method according to claim 17, wherein the uncertainty information is determined based on a type of sensor for providing the position, orientation and/or altitude of the capturing device.

23. The method according to claim 1, further including using a matching result for one or more of:

detection, classification, or localization of an object in the current image;

localization of the capturing device in the global coordinate system;

initial pose estimation of the capturing device; and

initializing pose tracking of the capturing device.

24. The method according to claim 1, wherein the method is used in an augmented reality application.

25. A non-transitory computer readable medium comprising software code sections adapted to perform a method of matching image features with reference features, comprising:

providing a current image captured by a capturing device;

providing a set of reference features, wherein each of the reference features comprises at least one first parameter which is at least partially indicative of a position and/or orientation of the reference feature with respect to a global coordinate system, wherein the global coordinate system is an earth coordinate system or an object coordinate system, or which is at least partially indicative of a position of the reference feature with respect to an altitude;

detecting at least one feature in the current image in a feature detection process;

associating with the detected feature at least one second parameter which is at least partially indicative of a position and/or orientation of the detected feature with respect to the global coordinate system, or which is at least partially indicative of a position of the detected feature with respect to an altitude; and

matching the detected feature with at least one of the reference features of the set of reference features by determining a similarity measure between the at least one first parameter and the at least one second parameter.