Method of providing a descriptor for at least one feature of an image and method of matching features

• US 10,650,546 B2
• Filed: 07/26/2018
• Issued: 05/12/2020
• Est. Priority Date: 10/19/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Kurz, Daniel, Benhimane, Selim
• Primary Examiner(s): Patel, Nirav G
• Application Number:
  US16/046,715
• Publication Number:
  US 20180357786A1
• Time in Patent Office: 656 Days
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 10/768 using context analysis, e.g. recognition aided by known co-occurring patterns
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

A method of providing a descriptor for at least one feature of an image comprises the steps of providing an image captured by a capturing device and extracting at least one feature from the image, and assigning a descriptor to the at least one feature, the descriptor depending on at least one parameter which is indicative of an orientation, wherein the at least one parameter is determined from the orientation of the capturing device measured by a tracking system. The invention also relates to a method of matching features of two or more images.

Claims

1. A method comprising:

extracting a feature from an image captured by a capturing device;

projecting a gravitational force onto a plane of the image;

determining a direction, with respect to the feature, of the gravitational force projected onto the plane of the image at a location of the feature in the image; and

assigning a descriptor to the feature based on the direction, with respect to the feature, of the gravitational force projected onto the plane of the image at the location of the feature.

2. The method of claim 1, wherein the gravitational force is projected onto the plane of the image based on intrinsic parameters of the capturing device.

3. The method of claim 2, further comprising determining an orientation of the gravitational force in a coordinate system of the capturing device, wherein the gravitational force is projected onto the plane of the image based further on the orientation.

4. The method of claim 3, further comprising receiving orientation data from an accelerometer, a gyrometer, a compass, or a combination thereof, wherein the orientation of the capturing device is determined based on the orientation data.

5. The method of claim 4, wherein the direction, with respect to the feature, of the gravitational force projected onto the plane of the image at the location of the feature in the image is determined based on a three dimensional location of the feature.

6. The method of claim 5, wherein the three dimensional location of the feature is determined based on depth information received from the capturing device.

7. The method of claim 6, wherein the capture device includes a laser scanner, a time-of-flight camera, a stereo camera, or a combination thereof.

8. An apparatus comprising:

one or more processors; and

a memory storing instructions executable by the one or more processors to;

extract a feature from an image captured by a capturing device;

project a gravitational force onto a plane of the image;

determine a direction, with respect to the feature, of the gravitational force projected onto the plane of the image at a location of the feature in the image; and

assign a descriptor to the feature based on the direction, with respect to the feature, of the gravitational force projected onto the plane of the image at the location of the feature in the image.

9. The apparatus of claim 8, further comprising the capturing device.

10. The apparatus of claim 8, wherein the instructions are executable by the one or more processors to project the gravitational force onto the plane of the image based on intrinsic parameters of the capturing device.

11. The apparatus of claim 8, further comprising a sensor including:

an accelerometer;

a gyrometer;

a compass;

or

a combination thereof.

12. The apparatus of claim 11, wherein the instructions are further executable by the one or more processors to:

determine an orientation of the gravitational force with respect to a coordinate system of the capturing device based on orientation data from the sensor; and

project the gravitational force onto the plane of the image based further on the orientation.

13. The apparatus of claim 8, wherein the instructions are executable by the one or more processors to determine the direction, with respect to the feature, of the gravitational force projected onto the plane of the image at the location in the image based on a three dimensional location of the feature.

14. The apparatus of claim 13, further comprising a three dimensional sensor comprising a laser scanner, a time-of-flight camera, a stereo camera, or a combination thereof, wherein the instructions are further executable by the one or more processors to determine the three dimensional location of the feature based on data from the three dimensional sensor.

15. A computer readable storage device storing instructions executable by one or more processors to:

extract a feature from an image captured by a capturing device;

project a gravitational force onto a plane of the image;

determine a direction, with respect to the feature, of the gravitational force projected onto the plane of the image at a location of the feature in the image; and

assign a descriptor to the feature based on the direction, with respect to the feature, of the gravitational force projected onto the plane of the image at the location of the feature in the image.

16. The computer readable storage device of claim 15, wherein the instructions are executable by the one or more processors to project the gravitational force onto the plane of the image based on intrinsic parameters of the capturing device.

17. The computer readable storage device of claim 15, wherein the instructions are further executable by the one or more processors to:

determine an orientation of the gravitational force in a coordinate system of the capturing device based on orientation data from an accelerometer, a gyrometer, a compass, or a combination thereof; and

project the gravitational force onto the plane of the image based further on the orientation.

18. The computer readable storage device of claim 15, wherein the instructions are executable by the one or more processors to determine the direction, with respect to the feature, of the gravitational force projected onto the plane of the image based on a three dimensional location of the feature.

19. The computer readable storage device of claim 18, wherein the instructions are further executable by the one or more processors to determine the three dimensional location of the feature based on data from a three dimensional sensor comprising a laser scanner, a time-of-flight camera, a stereo camera, or a combination thereof.

20. The computer readable storage device of claim 15, wherein the instructions are further executable by the one or more processors to match the feature to a second feature from a second image based on the descriptor and a second descriptor of the second feature, and wherein the second descriptor is based on a second direction, with respect to the second feature, of the gravitational force projected onto a second plane of the second image at a second location of the second feature in the second image.