Method of providing a descriptor for at least one feature of an image and method of matching features

• US 10,062,169 B2
• Filed: 12/11/2014
• Issued: 08/28/2018
• Est. Priority Date: 10/19/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Kurz, Daniel, Benhimane, Selim
• Primary Examiner(s): Patel, Nirav G
• Application Number:
  US14/567,668
• Publication Number:
  US 20150092987A1
• Time in Patent Office: 1,356 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 10/768 using context analysis, e.g. recognition aided by known co-occurring patterns
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

A method of providing a descriptor for at least one feature of an image comprises the steps of providing an image captured by a capturing device and extracting at least one feature from the image, and assigning a descriptor to the at least one feature, the descriptor depending on at least one parameter which is indicative of an orientation, wherein the at least one parameter is determined from the orientation of the capturing device measured by a tracking system. The invention also relates to a method of matching features of two or more images.

Claims

1. A method of providing a descriptor for at least one feature of an image, comprising:

providing an image captured by a capturing device and extracting at least one feature from the image; and

assigning a descriptor to the at least one feature, the descriptor depending on at least one parameter indicative of an orientation with respect to a direction of a gravitational force in 3D space,

wherein the at least one parameter is determined from at least one tracking system, and

wherein the at least one parameter is determined from depth information associated with the at least one feature.

2. The method of claim 1, further comprising determining at least one 3D position related to the at least one feature according to the depth information associated with the at least one feature and the at least one tracking system, wherein the at least one parameter is further determined from the at least one 3D position.

3. The method of claim 1, further comprising:

providing at least one surrounding pixel of the at least one feature in the image and depth information associated with the at least one surrounding pixel;

determining at least one 3D position related to the at least one surrounding pixel according to the depth information associated with the at least one surrounding pixel and the at least one tracking system,

wherein the at least one parameter is further determined from the at least one 3D position associated with the at least one surrounding pixel.

4. The method of claim 1, wherein the orientation with respect to the direction of the gravitational force in 3D space is represented by a 3D vector originating from a respective 3D position, related to the at least one feature, in 3D space in a world coordinate system.

5. The method of claim 1, wherein the at least one parameter is determined from an orientation of the capturing device in reference to a common coordinate system.

6. The method of claim 1, wherein the tracking system comprises a sensor, in particular an inertial sensor, or an accelerometer, or a gyroscope or a compass, which is associated with the capturing device.

7. The method of claim 1, wherein the tracking system comprises a mechanical and/or electromagnetic and/or acoustic and/or optical tracking system.

8. The method of claim 1, wherein the capturing device comprises a range data capturing device, particularly a laser scanner, time-of-flight camera, or a stereo camera, which provides image pixels with an associated depth and/or 3D position.

9. The method of claim 1, further comprising the step of normalizing a neighborhood of the at least one feature with respect to the orientation.

10. The method of claim 9, wherein in the step of normalizing the neighborhood of the at least one feature, the orientation provides an angle for rotating the image.

11. The method of claim 9, wherein in the step of normalizing the neighborhood of the at least one feature, the orientation is used to warp the entire image or parts of the image to one or more reference orientations to correct for perspective distortions in particular feature neighborhoods.

12. The method of claim 11, wherein the at least one parameter is further determined according to depth information associated with the at least one feature and depth information associated with at least two other pixels in the image.

13. A method of matching features of two or more images, comprising:

extracting at least one first feature of a first image and at least one second feature of a second image;

providing a first descriptor for the first feature and a second descriptor for the second feature, wherein at least one of the first and second descriptors is provided according to the method of claim 1; and

comparing the first and second descriptors in a matching process for the first and second features.

14. The method of claim 13, wherein in the matching process it is determined based on a similarity measure whether the first and second features correspond with each other.

15. The method of claim 14, wherein the similarity measure comprises a multi-stage method that successively computes different measures to reject wrong matches as early as possible.

16. The method of claim 1, wherein the method is implemented in a process of stereo matching, particularly wide-baseline stereo matching, camera tracking, image retrieval, object recognition, visual search, pose estimation, visual surveillance, scene reconstruction, motion estimation, panorama stitching, or image restoration.

18. The method of claim 13, wherein the method is implemented in a process of stereo matching, particularly wide-baseline stereo matching, camera tracking, image retrieval, object recognition, visual search, pose estimation, visual surveillance, scene reconstruction, motion estimation, panorama stitching or image restoration.

17. A non-transitory computer readable medium comprising software code which, when executed by a processor, causes the processor to:

provide an image captured by a capturing device and extract at least one feature from the image; and

assign a descriptor to the at least one feature, the descriptor depending on at least one parameter indicative of an orientation with respect to a direction of a gravitational force in 3D space,

wherein at least one parameter is determined from at least one tracking system, and

wherein the at least one parameter is determined from depth information associated with the at least one feature.

19. The computer readable medium of claim 17, wherein the software code further causes the process to determine at least one 3D position related to the at least one feature according to the depth information associated with the at least one feature and the at least see tracking system, wherein the at least one parameter is further determined from the at least one 3D position.

20. A system for providing a descriptor for at least one feature of an image, comprising:

a processor; and

a memory coupled to the processor and comprising computer code which, when executed by the processor, causes the processor to;

provide an image captured by a capturing device and extract at least one feature from the image; and

assign a descriptor to the at least one feature, the descriptor depending on at least one parameter indicative of an orientation with respect to a direction of a gravitational force in 3D space,

wherein the at least one parameter is determined from at least one tracking system, and

wherein the at least one parameter is determined from depth information associated with the at least one feature.