Object recognition using incremental feature extraction

• US 8,625,902 B2
• Filed: 07/28/2011
• Issued: 01/07/2014
• Est. Priority Date: 07/30/2010
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Baheti, Pawan Kumar, Vaddadi, Sundeep, Swaminathan, Ashwin, Reznik, Yuriy, Hamsici, Onur C., Chari, Murali Ramaswamy, Hong, John H., Lee, Chong Uk
• Primary Examiner(s): Allison, Andrae S
• Application Number:
  US13/193,294
• Publication Number:
  US 20120027290A1
• Time in Patent Office: 894 Days
• Field of Search: 382/162, 382/165, 382/190, 382/195, 382/225, 382/275, 382/279, 382/305, 382/306, 707/6, 707/104, 707/736, 707/771, 707/E17.014
• US Class Current: 382/190
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 30/2504 Coarse or fine approaches, e.g. resolution of ambiguities or multiscale approaches

Abstract

In one example, an apparatus includes a processor configured to extract a first set of one or more keypoints from a first set of blurred images of a first octave of a received image, calculate a first set of one or more descriptors for the first set of keypoints, receive a confidence value for a result produced by querying a feature descriptor database with the first set of descriptors, wherein the result comprises information describing an identity of an object in the received image, and extract a second set of one or more keypoints from a second set of blurred images of a second octave of the received image when the confidence value does not exceed a confidence threshold. In this manner, the processor may perform incremental feature descriptor extraction, which may improve computational efficiency of object recognition in digital images.

Claims

1. A method of determining identities for objects in an image, the method comprising:

extracting a first set of one or more keypoints from a first set of blurred images of a first octave of a received image;

calculating a first set of one or more descriptors for the first set of keypoints;

receiving a confidence value for a result produced by querying a feature descriptor database with the first set of descriptors, wherein the result comprises information describing an identity of an object in the received image; and

extracting a second set of one or more keypoints from a second set of blurred images of a second octave of the received image when the confidence value does not exceed a confidence threshold.

2. The method of claim 1, further comprising:

determining that the confidence value does not exceed the confidence threshold;

extracting the second set of keypoints from a second set of blurred images of the second octave based on the determination;

calculating a second set of one or more descriptors from the second set of keypoints; and

querying the feature descriptor database with a set of descriptors comprising the first set of descriptors and the second set of descriptors.

3. The method of claim 1, further comprising:

estimating a scale value for the object in the received image;

selecting a minimum octave of the feature descriptor database based on the estimated scale; and

providing an indication of a minimum octave in the feature descriptor database to cause a search of the feature descriptor database to search feature descriptors at or above the selected minimum octave in the feature descriptor database.

4. The method of claim 3, wherein estimating the scale value comprises:

analyzing a distribution of the first set of keypoints across a scale space;

determining a cutoff point corresponding to a scale level in the distribution such that approximately 90% of the keypoints fall below the scale level; and

estimating the scale value as a function of the determined cutoff point.

5. The method of claim 3, further comprising obtaining depth information for the object, wherein estimating the scale value comprises estimating the scale value based at least in part on the depth information for the object.

6. The method of claim 1, further comprising:

analyzing sensor information from one or more sensors associated with a camera that captured the received image; and

determining an octave level for at least one of the first octave and the second octave based on the analysis of the sensor information.

7. The method of claim 6, wherein analyzing the sensor information comprises analyzing global positioning system (GPS) information to determine whether the camera was located in an outdoor environment when the received image was captured.

8. The method of claim 6, wherein analyzing the sensor information comprises analyzing GPS information to determine locations of objects relatively near the camera when the received image was captured, and determining sizes for the objects from descriptive data for the objects.

9. The method of claim 6, wherein analyzing the sensor information comprises determining whether the camera was located in an indoor environment based on network data indicative of whether a device comprising the camera was communicatively coupled to a wireless network when the received image was captured.

10. The method of claim 6, wherein analyzing the sensor information comprises calculating depth information indicative of distances between one or more objects in the received image and the camera when the received image was captured.

11. The method of claim 6, wherein analyzing sensor information comprises estimating a depth value for the object using data provided by an active probing sensor.

12. The method of claim 1, further comprising sending the one or more descriptors to a server to cause the server to query the feature descriptor database using the one or more descriptors, wherein receiving the confidence value comprises receiving the confidence value from the server in response to the query.

13. An apparatus for determining identities for objects in an image, the apparatus comprising a processor configured to extract a first set of one or more keypoints from a first set of blurred images of a first octave of a received image, calculate a first set of one or more descriptors for the first set of keypoints, receive a confidence value for a result produced by querying a feature descriptor database with the first set of descriptors, wherein the result comprises information describing an identity of an object in the received image, and extract a second set of one or more keypoints from a second set of blurred images of a second octave of the received image when the confidence value does not exceed a confidence threshold.

14. The apparatus of claim 13, further comprising a camera configured to capture the image and to provide the image to the processor.

15. The apparatus of claim 13, wherein the processor is further configured, when the processer determines that the confidence value does not exceed the confidence threshold, to extract the second set of keypoints from a second set of blurred images of the second octave based on the determination, calculate a second set of one or more descriptors from the second set of keypoints, and query the feature descriptor database with a set of descriptors comprising the first set of descriptors and the second set of descriptors.

16. The apparatus of claim 15, wherein the processor is further configured to estimate a scale value for the object in the received image, select a minimum octave of the feature descriptor database based on the estimated scale, and search descriptors in the feature descriptor database, wherein the searched descriptors correspond to octaves in the feature descriptor database at or above the selected minimum octave.

17. The apparatus of claim 16, wherein to estimate the scale value, the processor is configured to analyze a distribution of the first set of keypoints across a scale space, determine a cutoff point corresponding to a scale level in the distribution such that approximately 90% of the keypoints fall below the scale level, and calculate the scale value as a function of the determined cutoff point.

18. The apparatus of claim 16, further comprising a depth estimation unit configured to obtain depth information for the object, wherein the processor is configured to estimate the scale value based at least in part on the depth information for the object.

19. The apparatus of claim 13, wherein the processor is further configured to analyze sensor information from one or more sensors associated with a camera that captured the received image, and determine an octave level for at least one of the first octave and the second octave based on the analysis of the sensor information.

20. The apparatus of claim 19, further comprising a global positioning system (GPS) unit configured to determine location information for the apparatus, wherein to analyze the sensor information, the processor is configured to determine whether the camera was located in an outdoor environment when the received image was captured based on the location information determined by the GPS unit.

21. The apparatus of claim 19, further comprising a GPS unit configured to determine location information relative to the apparatus, wherein to analyze the sensor information, the processor is configured to analyze the location information determined by the GPS unit to determine locations of objects relatively near the camera when the received image was captured, and to determine sizes for the objects from descriptive data for the objects.

22. The apparatus of claim 19, further comprising a wireless network interface, wherein to analyze the sensor information, the processor is configured to determine whether the camera was located in an indoor environment based on network data indicative of whether the wireless network interface was communicatively coupled to a wireless network when the received image was captured.

23. The apparatus of claim 19, further comprising a camera array comprising at least two cameras including the camera that captured the received image, wherein to analyze the sensor information, the processor is configured to calculate depth information from images captured by the camera array indicative of distances between one or more objects in the received image and the camera when the received image was captured.

24. The apparatus of claim 19, wherein the one or more sensors comprise an active probing sensor configured to estimate a depth value for the object.

25. The apparatus of claim 13, wherein the processor is configured to send the one or more descriptors to a server to cause the server to query the feature descriptor database using the one or more descriptors, and to receive the confidence value from the server in response to the query.

26. The apparatus of claim 13, wherein the apparatus comprises at least one of:

an integrated circuit;

a microprocessor; and

a wireless communication device that includes the processor.

27. An apparatus for determining identities for objects in an image, the apparatus comprising:

means for extracting a first set of one or more keypoints from a first set of blurred images of a first octave of a received image;

means for calculating a first set of one or more descriptors for the first set of keypoints;

means for receiving a confidence value for a result produced by querying a feature descriptor database with the first set of descriptors, wherein the result comprises information describing an identity of an object in the received image; and

means for extracting a second set of one or more keypoints from a second set of blurred images of a second octave of the received image when the confidence value does not exceed a confidence threshold.

28. The apparatus of claim 27, further comprising:

means for determining that the confidence value does not exceed the confidence threshold;

means for extracting the second set of keypoints from a second set of blurred images of the second octave based on the determination;

means for calculating a second set of one or more descriptors from the second set of keypoints; and

means for querying the feature descriptor database with a set of descriptors comprising the first set of descriptors and the second set of descriptors.

29. The apparatus of claim 28, further comprising:

means for estimating a scale value for the object in the received image;

means for selecting a minimum octave of the feature descriptor database based on the estimated scale; and

means for providing an indication of a minimum octave in the feature descriptor database to cause a search of the feature descriptor database to search feature descriptors at or above the selected minimum octave in the feature descriptor database.

30. The apparatus of claim 29, wherein the means for estimating the scale value comprises:

means for analyzing a distribution of the first set of keypoints across a scale space;

means for determining a cutoff point corresponding to a scale level in the distribution such that approximately 90% of the keypoints fall below the scale level; and

means for estimating the scale value as a function of the determined cutoff point.

31. The apparatus of claim 29, further comprising means for obtaining depth information for the object, wherein the means for estimating the scale value comprises means for estimating the scale value based at least in part on the depth information for the object.

32. The apparatus of claim 27, further comprising:

means for analyzing sensor information from one or more sensors associated with a camera that captured the received image; and

means for determining an octave level for at least one of the first octave and the second octave based on the analysis of the sensor information.

33. The apparatus of claim 32, further comprising means for receiving global positioning system (GPS) information, wherein the means for analyzing the sensor information comprises means for analyzing the GPS information to determine whether the camera was located in an outdoor environment when the received image was captured.

34. The apparatus of claim 32, further comprising means for receiving GPS information, wherein the means for analyzing the sensor information comprises:

means for analyzing the GPS information to determine locations of objects relatively near the camera when the received image was captured; and

means for determining sizes for the objects from descriptive data for the objects.

35. The apparatus of claim 32, further comprising means for communicating via at least one wireless network protocol, wherein the means for analyzing the sensor information comprises means for determining whether the camera was located in an indoor environment based on network data indicative of whether the means for communicating via the wireless network protocol was communicatively coupled to a wireless network when the received image was captured.

36. The apparatus of claim 32, further comprising means for capturing two or more images of a scene, wherein one of the two or more images comprises the received image, and wherein the means for analyzing the sensor information comprises means for calculating depth information from the two or more images of the scene indicative of distances between one or more objects in the received image and the camera when the received image was captured.

37. The apparatus of claim 32, wherein the means for analyzing the sensor information comprises means for estimating a depth value for the object using data provided by an active probing sensor.

38. The apparatus of claim 27, further comprising means for sending the one or more descriptors to a server to cause the server to query the feature descriptor database using the one or more descriptors, wherein the means for receiving the confidence value comprises means for receiving the confidence value from the server in response to the query.

39. A computer program product comprising a non-transitory computer-readable medium having stored thereon instructions that, when executed, cause a processor to:

extract a first set of one or more keypoints from a first set of blurred images of a first octave of a received image;

calculate a first set of one or more descriptors for the first set of keypoints;

receive a confidence value for a result produced by querying a feature descriptor database with the first set of descriptors, wherein the result comprises information describing an identity of an object in the received image; and

extract a second set of one or more keypoints from a second set of blurred images of a second octave of the received image when the confidence value does not exceed a confidence threshold.

40. The computer program product of claim 39, further comprising instructions that cause the processor to:

determine that the confidence value does not exceed the confidence threshold;

extract the second set of keypoints from a second set of blurred images of the second octave based on the determination;

calculate a second set of one or more descriptors from the second set of keypoints; and

query the feature descriptor database with a set of descriptors comprising the first set of descriptors and the second set of descriptors.

41. The computer program product of claim 40, further comprising instructions that cause the processor to:

estimate a scale value for the object in the received image;

select a minimum octave of the feature descriptor database based on the estimated scale; and

provide an indication of a minimum octave in the feature descriptor database to cause a search of the feature descriptor database to search feature descriptors at or above the selected minimum octave in the feature descriptor database.

42. The computer program product of claim 41, wherein the instructions that cause the processor to estimate the scale value comprise instructions that cause the processor to:

analyze a distribution of the first set of keypoints across a scale space;

determine a cutoff point corresponding to a scale level in the distribution such that approximately 90% of the keypoints fall below the scale level; and

estimate the scale value as a function of the determined cutoff point.

43. The computer program product of claim 41, further comprising instructions that cause the processor to obtain depth information for the object, wherein the instructions that cause the processor to estimate the scale value comprise instructions that cause the processor to estimate the scale value based at least in part on the depth information for the object.

44. The computer program product of claim 39, further comprising instructions that cause the processor to:

analyze sensor information from one or more sensors associated with a camera that captured the received image; and

determine an octave level for at least one of the first octave and the second octave based on the analysis of the sensor information.

45. The computer program product of claim 44, wherein the instructions that cause the processor to analyze the sensor information comprises instructions that cause the processor to analyze global positioning system (GPS) information to determine whether the camera was located in an outdoor environment when the received image was captured.

46. The computer program product of claim 44, wherein the instructions that cause the processor to analyze the sensor information comprises instructions that cause the processor to analyze GPS information to determine locations of objects relatively near the camera when the received image was captured, and determine sizes for the objects from descriptive data for the objects.

47. The computer program product of claim 44, wherein the instructions that cause the processor to analyze the sensor information comprises instructions that cause the processor to determine whether the camera was located in an indoor environment based on network data indicative of whether a device comprising the camera was communicatively coupled to a wireless network when the received image was captured.

48. The computer program product of claim 44, wherein the instructions that cause the processor to analyze the sensor information comprises instructions that cause the processor to calculate depth information indicative of distances between one or more objects in the received image and the camera when the received image was captured.

49. The computer program product of claim 44, wherein the instructions that cause the processor to analyze the sensor information comprise instructions that cause the processor to estimate a depth value for the object using data provided by an active probing sensor.

50. The computer program product of claim 39, further comprising instructions that cause the processor to send the one or more descriptors to a server to cause the server to query the feature descriptor database using the one or more descriptors, wherein the instructions that cause the processor to receive the confidence value comprise instructions that cause the processor to receive the confidence value from the server in response to the query.