METHOD OF PROVIDING IMAGE FEATURE DESCRIPTORS

• US 20150294189A1
• Filed: 07/23/2012
• Published: 10/15/2015
• Est. Priority Date: 07/23/2012
• Status: Abandoned Application

• Current Assignee: Apple Inc.
• Original Assignee: Selim Benhimane
• Inventors: Kurz, Daniel, Olszamowski, Thomas, Benhimane, Selim
• Application Number:
  US14/417,046
• Publication Number:
  US 20150294189A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/2113 by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/771 Feature selection, e.g. selecting representative features from a multi-dimensional feature space
• CPC Class Codes: G06V 10/772 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries

Abstract

A method of providing a set of feature descriptors configured to be used in matching an object in an image of a camera is provided. The method includes: a) providing at least two images of a first object; b) extracting in at least two of the images at least one feature from the respective image, c) providing at least one descriptor for an extracted feature, and storing the descriptors; d) matching descriptors in the first set of descriptors; e) computing a score parameter based on the result of the matching process; f) selecting at least one descriptor based on its score parameter; g) adding the selected descriptor(s) to a second set of descriptors; and h) updating the score parameter of descriptors in the first set based on a selection process and to the result of the matching process.

Claims

1. A method of providing a set of feature descriptors configured to be used in matching at least one feature of an object in an image of a camera, comprising the steps of:

a) providing at least two images of a first object or of multiple instances of a first object, wherein the multiple instances provide different appearances or different versions of an object;

b) extracting in at least two of the images at least one feature from the respective image;

c) providing at least one descriptor for an extracted feature, and storing the descriptors for a plurality of extracted features in a first set of descriptors;

d) matching a plurality of the descriptors of the first set of descriptors against a plurality of the descriptors of the first set of descriptors;

e) computing a score parameter for a plurality of the descriptors based on the result of the matching process;

f) selecting among the descriptors at least one descriptor based on its score parameter in comparison with score parameters of other descriptors;

g) adding the selected at least one descriptor to a second set of descriptors;

h) updating the score parameter of a plurality of the descriptors in the first set of descriptors according to any preceding selection process and to the result of the matching process; and

i) performing steps f) and g) at least a second time;

wherein the second set of descriptors is configured to be used in matching at least one feature of the first object or of a second object in an image of a camera.

2. The method according to claim 1, wherein steps h) and i) are repeatedly processed until the number of descriptors in the second set of descriptors has reached a particular value or the number of descriptors in the second set of descriptors stops varying.

3. The method according to claim 1, wherein step g) is preceded by modifying the at least one selected descriptor based on the selection process, wherein the modification of the selected descriptor comprises updating the descriptor as a combination of the selected descriptor and other descriptors in the first set of descriptors.

5. The method according to claim 1, wherein the usage of the result of the matching process in the update step h) is restricted to the result of the matching process of the at least one selected descriptor, or the result of the matching process of the descriptors that match with the at least one selected descriptor.

6. The method according to claim 1, wherein step d) includes determining for each of the descriptors which were matched whether they were correctly or incorrectly matched, and step e) includes computing the score parameter dependent on whether the descriptors were correctly or incorrectly matched.

7. The method according to claim 1, wherein

the score parameter is indicative of the number of matches the respective descriptor has been correctly matched with any other of the descriptors;

in step f) at least one descriptor with a score parameter indicative of the highest number of matches within the first set of descriptors is selected; and

step h) reduces the score parameter of the at least one selected descriptor and the score parameter of the descriptors that match with the at least one selected descriptor.

18. The method according to claim 1, wherein step a) includes providing the different images of the first object under different conditions which includes changes from one of the images to another one of the images in at least one of the following:

viewpoint, illumination, camera parameters such as focal length, focus, exposure time, signal-to-noise-ratio.

19. The method according to claim 1, wherein step a) includes providing the different images of the first object by using a synthetic view creation algorithm creating the different images by respective virtual cameras as respective synthetic views, and

wherein the synthetic view creation algorithm includes a spatial transformation which projects a 3D model onto the image plane of a respective synthetic camera, and a rendering method is applied which is capable to simulate properties of a real camera, particularly such as defocus, motion blur, noise, exposure time, brightness, contrast, and to also simulate different environments, particularly such as by using virtual light sources, shadows, reflections, lens flares, blooming, environment mapping.

22. A method of matching at least one feature of an object in an image of a camera, comprising:

providing at least one image with an object captured by a camera;

extracting current features from the at least one image and providing a set of current feature descriptors with at least one current feature descriptor provided for an extracted feature;

providing a second set of descriptors according to claim 1; and

comparing the set of current feature descriptors with the second set of descriptors for matching at least one feature of the object in the at least one image.

28. The method according to claim 22, further including determining a position and orientation of the camera which captures the at least one image with respect to the object based on correspondences of feature descriptors determined in the matching process.

29. The method according to claim 1, wherein the method of providing a set of feature descriptors is a method of providing a set of feature descriptors configured to be used in localizing an object in an image of a camera in an augmented reality application.

30. The method according to claim 22, wherein the method of matching an object in an image of a camera is a method of localizing an object in an image of a camera in an augmented reality application.

4. (canceled)

8. (canceled)

9. A method of providing at least two sets of feature descriptors configured to be used in matching at least one feature of an object in an image of a camera, comprising the steps of:

a) providing at least two images of a first object or of multiple instances of a first object, wherein the multiple instances provide different appearances or different versions of an object, wherein each of the images is generated by a respective camera having a known orientation with respect to gravity when generating the respective image,

b) extracting in at least two of the images at least one feature from the respective image;

c) providing;

c1) at least one descriptor for an extracted feature, and storing the descriptors for a plurality of extracted features in multiple sets of descriptors with at least a first set of descriptors and a second set of descriptors, wherein the first set of descriptors contains descriptors of features which were extracted from images captured by the respective cameras having the orientations corresponding to a first orientation zone with respect to gravity, and the second set of descriptors contains descriptors of features which were extracted from images captured by the respective cameras having the orientations corresponding to a second orientation zone with respect to gravity;

or;

c2) at least one descriptor for an extracted feature, storing the descriptors for a plurality of extracted features in multiple sets of descriptors with at least a first set of descriptors and a second set of descriptors, and calculating for each descriptor an orientation of a ray pointing from the respective camera center towards the respective extracted feature that is described by the descriptor with respect to gravity, wherein the first set of descriptors contains descriptors of features having the orientations corresponding to a first orientation zone with respect to gravity, and the second set of descriptors contains descriptors of features having the orientations corresponding to a second orientation zone with respect to gravity;

d) matching a plurality of the descriptors of the first set of descriptors against a plurality of the descriptors of the first set of descriptors, and matching a plurality of the descriptors of the second set of descriptors against a plurality of the descriptors of the second set of descriptors;

e) computing a score parameter for a plurality of the descriptors based on the result of the matching process;

f) selecting within the first set of descriptors at least one descriptor based on its score parameter in comparison with score parameters of other descriptors, and selecting within the second set of descriptors at least another descriptor based on its score parameter in comparison with score parameters of other descriptors;

g) adding the at least one selected descriptor from the first set to a third set of descriptors and adding the at least one selected descriptor from the second set to a fourth set of descriptors;

h) updating the score parameter of a plurality of descriptors in the first and/or second set of descriptors according to any preceding selection process and to the result of the matching process,

i) performing steps f) and g) at least a second time;

wherein the third and/or fourth set of descriptors are configured to be used in matching at least one feature of the first object or of a second object in an image of a camera.

10. The method according to claim 9, wherein steps h) and i) are repeatedly processed until the number of descriptors in the third and/or fourth set of descriptors has reached a particular value or the number of descriptors in the third and/or fourth set of descriptors stops varying.

11. The method according to claim 9 or 10, wherein step g) is preceded by modifying the at least one selected descriptor based on the selection process, and

wherein the modification of the selected descriptor comprises updating the descriptor as a combination of the selected descriptor and other descriptors in the first or second set of descriptors.

13. The method according to claim 9, wherein when step c1) is performed, step a) further includes calculating an orientation angle between the principal axis and a provided gravity vector of the camera that corresponds to the respective image;

and

wherein for the calculated orientation angle it is determined whether it corresponds to the first or second orientation zone, and if it corresponds to the first orientation zone the descriptor of the extracted feature of the respective image is stored in the first set of descriptors, and if it corresponds to the second orientation zone the descriptor of the extracted feature of the respective image is stored in the second set of descriptors.

15. The method according to claim 9, wherein step c2) further includes calculating for each descriptor an orientation angle between the ray pointing from the respective camera center towards the respective extracted feature that is described by the descriptor and a provided gravity vector of the camera which provides the respective image;

and

wherein for the calculated orientation angle it is determined whether it corresponds to the first or second orientation zone, and if it corresponds to the first orientation zone the respective descriptor is stored in the first set of descriptors and if it corresponds to the second orientation zone the respective descriptor is stored in the second set of descriptors.

23. A method of matching at least one feature of an object in an image of a camera, comprising:

providing at least one image with an object captured by a camera;

extracting current features from the at least one image and providing a set of current feature descriptors with at least one current feature descriptor provided for an extracted feature,

providing a third and a fourth set of descriptors according to claim 9;

comparing the set of current feature descriptors with the third and/or fourth set of descriptors for matching at least one feature of the object in the at least one image.

24. The method according to claim 23, further comprising:

providing a gravity vector of the camera which captures the at least one image;

determining an orientation of the camera which captures the at least one image with respect to gravity and associating the determined orientation of the camera with the first orientation zone or with the second orientation zone; and

comparing the set of current feature descriptors with the third set of descriptors, if the determined orientation of the camera is associated with the first orientation zone, and comparing the set of current feature descriptors with the fourth set of descriptors, if the determined orientation of the camera is associated with the second orientation zone.

26. The method according to claim 23, further comprising:

providing a gravity vector of the camera which captures the at least one image;

calculating for each descriptor an orientation angle between a ray pointing from the camera center towards the respective extracted feature that is described by the descriptor and the gravity vector of the camera which captures the at least one image, and associating the orientation angle of each descriptor with the first orientation zone or with the second orientation zone; and

comparing one of the current feature descriptors with the third set of descriptors, if the determined orientation angle of that current feature descriptor is associated with the first orientation zone, and comparing one of the current feature descriptors with the fourth set of descriptors, if the determined orientation angle of that current feature descriptor is associated with the second orientation zone.

27. The method according to claim 23, wherein the gravity vector is provided from a gravity sensor associated with the camera which captures the at least one image.

12. (canceled)

14. (canceled)

16-17. -17. (canceled)

20-21. -21. (canceled)

25. (canceled)

31. A non-transitory computer readable medium comprising software code sections adapted to perform a method of providing a set of feature descriptors configured to be used in matching at least one feature of an object in an image of a camera, which method comprises:

a) providing at least two images of a first object or of multiple instances of a first object, wherein the multiple instances provide different appearances or different versions of an object;

b) extracting in at least two of the images at least one feature from the respective image;

c) providing at least one descriptor for an extracted feature, and storing the descriptors for a plurality of extracted features in a first set of descriptors;

d) matching a plurality of the descriptors of the first set of descriptors against a plurality of the descriptors of the first set of descriptors;

e) computing a score parameter for a plurality of the descriptors based on the result of the matching process;

f) selecting among the descriptors at least one descriptor based on its score parameter in comparison with score parameters of other descriptors;

g) adding the selected at least one descriptor to a second set of descriptors;

h) updating the score parameter of a plurality of the descriptors in the first set of descriptors according to any preceding selection process and to the result of the matching process; and

i) performing steps f) and g) at least a second time;

wherein the second set of descriptors is configured to be used in matching at least one feature of the first object or of a second object in an image of a camera.