FEEDBACK TO IMPROVE OBJECT RECOGNITION

• US 20120011142A1
• Filed: 07/08/2010
• Published: 01/12/2012
• Est. Priority Date: 07/08/2010
• Status: Abandoned Application

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Baheti, Pawan K., Swaminathan, Ashwin, Spindola, Serafin Diaz, Chari, Murali Ramaswamy
• Application Number:
  US12/832,918
• Publication Number:
  US 20120011142A1
• Time in Patent Office: Days
• US Class Current: 707/769
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]

Abstract

A database for object recognition is modified based on feedback information received from a mobile platform. The feedback information includes information with respect to an image of an object captured by the mobile platform. The feedback information, for example, may include the image, features extracted from the image, a confidence level for the features, posterior probabilities of the features belonging to an object in the database, GPS information, and heading orientation information. The feedback information may be used to improve the database pruning, add content to the database or update the database compression efficiency. The information feedback to the server by the mobile platform may be determined based on a search of a portion of the database performed by the mobile platform using features extracted from a captured query image.

Claims

1. A method of modifying a database of information of objects and images of the objects, the method comprising:

storing a database of information of objects and images of the objects;

receiving feedback information from a mobile platform, the received feedback information including information with respect to an image of an object captured by the mobile platform; and

updating the database using the received feedback information.

2. The method of claim 1, wherein updating the database comprises using the feedback information to perform at least one of improving the database pruning, learning user-generated content by adding the feedback information to the database, and updating the database compression efficiency.

3. The method of claim 1, wherein the received feedback information comprises at least one of:

the image, features extracted from the image, a confidence level for the features, posterior probabilities of the features belonging to an object in the database, GPS information, heading orientation information, scale information, and feature extraction parameters.

4. The method of claim 1, wherein updating the database comprises:

determining the received feedback information is related to an object that is not in the database; and

adding the object to the database.

5. The method of claim 4, wherein adding the object to the database comprises:

performing intra-object pruning for the object, the intra-object pruning comprising;

identifying a set of matching keypoint descriptors for a plurality of keypoint descriptors for the object;

removing one or more of the matching keypoint descriptors within each set of matching keypoint descriptors, wherein subsequent to the removal of the one or more of the matching keypoint descriptors there is at least one remaining keypoint descriptor in each set of matching keypoint descriptors;

performing inter-object pruning for the object with respect to the database, the inter-object pruning comprising;

characterizing discriminability of the remaining keypoint descriptors;

removing remaining keypoint descriptors with discriminability based on a threshold;

selecting descriptors for the object to be retained in the database; and

storing the descriptors in the database.

6. The method of claim 1, wherein updating the database comprises:

determining the received feedback information is related to an object that is in the database;

updating probabilities of keypoint descriptors stored in the database belonging to the object.

7. The method of claim 6, further comprising:

performing inter-object pruning for the object with respect to the database, the inter-object pruning comprising;

characterizing discriminability of the remaining keypoint descriptors using the updated probabilities;

removing remaining keypoint descriptors with discriminability based on a threshold;

selecting descriptors for the object to be retained in the database; and

storing the descriptors in the database.

8. The method of claim 6, further comprising:

determining to add the image of the object to the database;

performing intra-object pruning for the object, the intra-object pruning comprising;

identifying a set of matching keypoint descriptors for a plurality of keypoint descriptors for the object;

removing one or more of the matching keypoint descriptors within each set of matching keypoint descriptors, wherein subsequent to the removal of the one or more of the matching keypoint descriptors there is at least one remaining keypoint descriptor in each set of matching keypoint descriptors;

performing inter-object pruning for the object with respect to the database, the inter-object pruning comprising;

characterizing discriminability of the remaining keypoint descriptors using the updated probabilities;

removing remaining keypoint descriptors with discriminability based on a threshold;

selecting descriptors for the object to be retained in the database; and

storing the descriptors in the database.

9. The method of claim 1, wherein the received feedback information comprises at least one of GPS information, heading orientation information, and scale information, the method further comprising providing information from the database to the mobile platform based on the at least one of GPS information, heading orientation information, scale information, and feature extraction parameters.

10. The method of claim 1, wherein the received feedback information facilitates a personalized search.

11. The method of claim 1, wherein the received feedback information is used to build a collaborative search system.

12. The method of claim 1, wherein updating the database using the received feedback information comprises using the received feedback information to update the popularity of at least one of objects and views of the objects based on the number of times the at least one of objects and views is queried and the number of times a feature descriptor match occurs.

13. An apparatus comprising:

an external interface for receiving feedback information from a mobile platform, the received feedback information including information with respect to an image of an object captured by the mobile platform;

a processor connected to the external interface;

a database of information of objects and images of the objects;

memory connected to the processor; and

software held in the memory and run in the processor to update the database using the received feedback information.

14. The apparatus of claim 13, wherein the software run in the processor to update the database comprises software that causes the processor to at least one of improve the database pruning, learn user-generated content by adding the feedback information to the database, and update the database compression efficiency.

15. The apparatus of claim 13, wherein the received feedback information comprises at least one of:

the image, features extracted from the image, a confidence level for the features, posterior probabilities of the features belonging to an object in the database, GPS information, heading orientation information, scale information, and feature extraction parameters.

16. The apparatus of claim 13, wherein the software run in the processor to update the database comprises software that causes the processor to determine the received feedback information is related to an object that is not in the database;

and add the object to the database.

17. The apparatus of claim 16, wherein the software that causes the processor to add the object to the database comprises software that causes the processor to:

perform intra-object pruning for the object, the intra-object pruning comprising;

identify a set of matching keypoint descriptors for a plurality of keypoint descriptors for the object;

remove one or more of the matching keypoint descriptors within each set of matching keypoint descriptors, wherein subsequent to the removal of the one or more of the matching keypoint descriptors there is at least one remaining keypoint descriptor in each set of matching keypoint descriptors;

perform inter-object pruning for the object with respect to the database, the inter-object pruning comprising;

characterize discriminability of the remaining keypoint descriptors;

remove remaining keypoint descriptors with discriminability based on a threshold;

select descriptors for the object to be retained in the database; and

store the descriptors in the database.

18. The apparatus of claim 13, wherein the software run in the processor to update the database comprises software that causes the processor to determine the received feedback information is related to an object that is in the database and update probabilities of keypoint descriptors stored in the database belonging to the object.

19. The apparatus of claim 18, further comprising software that causes the processor to:

perform inter-object pruning for the object with respect to the database, the inter-object pruning comprising;

characterize discriminability of the remaining keypoint descriptors using the updated probabilities;

remove remaining keypoint descriptors with discriminability based on a threshold;

select descriptors for the object to be retained in the database; and

store the descriptors in the database.

20. The apparatus of claim 18, further comprising software that causes the processor to:

determine to add the image of the object to the database;

perform intra-object pruning for the object, the intra-object pruning comprising;

identify a set of matching keypoint descriptors for a plurality of keypoint descriptors for the object;

remove one or more of the matching keypoint descriptors within each set of matching keypoint descriptors, wherein subsequent to the removal of the one or more of the matching keypoint descriptors there is at least one remaining keypoint descriptor in each set of matching keypoint descriptors;

perform inter-object pruning for the object with respect to the database, the inter-object pruning comprising;

characterize discriminability of the remaining keypoint descriptors using the updated probabilities;

remove remaining keypoint descriptors with discriminability based on a threshold;

select descriptors for the object to be retained in the database; and

store the descriptors in the database.

21. The apparatus of claim 13, wherein the received feedback information comprises at least one of GPS information, heading orientation information, and scale information, the software further causes the processor to provide information from the database to the mobile platform based on the at least one of GPS information, heading orientation information, scale information, and feature extraction parameters.

22. The apparatus of claim 13, wherein the received feedback information facilitates a personalized search.

23. The apparatus of claim 13, wherein the received feedback information is used to build a collaborative search system.

24. The apparatus of claim 13, wherein the software run in the processor to update the database comprises software that causes the processor to use the received feedback information to update the popularity of at least one of objects and views of the objects based on the number of times the at least one of objects and views is queried and the number of times a feature descriptor match occurs.

25. A system comprising:

means for receiving feedback information from a mobile platform, the received feedback information including information with respect to an image of an object captured by the mobile platform; and

means for updating a database of information of objects and images of the objects using the received feedback information.

26. The system of claim 25, wherein the means for updating the database comprises means for using the feedback information to perform at least one of improving the database pruning, learning user-generated content by adding the feedback information to the database, and updating the database compression efficiency.

27. The system of claim 25, wherein the means for updating the database comprises:

means for determining the received feedback information is related to an object that is not in the database; and

means for adding the object to the database.

28. The system of claim 27, wherein the means for adding the object to the database comprises:

means for performing intra-object pruning for the object, the intra-object pruning comprising;

identifying a set of matching keypoint descriptors for a plurality of keypoint descriptors for the object;

removing one or more of the matching keypoint descriptors within each set of matching keypoint descriptors, wherein subsequent to the removal of the one or more of the matching keypoint descriptors there is at least one remaining keypoint descriptor in each set of matching keypoint descriptors;

means for performing inter-object pruning for the object with respect to the database, the inter-object pruning comprising;

characterizing discriminability of the remaining keypoint descriptors;

removing remaining keypoint descriptors with discriminability based on a threshold;

means for selecting descriptors for the object to be retained in the database; and

means for storing the descriptors in the database.

29. The system of claim 25, wherein the means for updating the database comprises:

means for determining the received feedback information is related to an object that is in the database;

means for updating probabilities of keypoint descriptors stored in the database belonging to the object.

30. The system of claim 29, wherein the means for updating the database comprises:

means for performing inter-object pruning for the object with respect to the database, the inter-object pruning comprising;

characterizing discriminability of the remaining keypoint descriptors using the updated probabilities;

removing remaining keypoint descriptors with discriminability based on a threshold;

means for selecting descriptors for the object to be retained in the database; and

means for storing the descriptors in the database.

31. The system of claim 29, wherein the means for updating the database comprises:

means for determining to add the image of the object to the database;

means for performing intra-object pruning for the object, the intra-object pruning comprising;

identifying a set of matching keypoint descriptors for a plurality of keypoint descriptors for the object;

removing one or more of the matching keypoint descriptors within each set of matching keypoint descriptors, wherein subsequent to the removal of the one or more of the matching keypoint descriptors there is at least one remaining keypoint descriptor in each set of matching keypoint descriptors;

means for performing inter-object pruning for the object with respect to the database, the inter-object pruning comprising;

characterizing discriminability of the remaining keypoint descriptors using the updated probabilities;

removing remaining keypoint descriptors with discriminability based on a threshold;

means for selecting descriptors for the object to be retained in the database; and

means for storing the descriptors in the database.

32. The system of claim 25, wherein the received feedback information comprises at least one of GPS information, heading orientation information, and scale information, the system further comprising means for providing information from the database to the mobile platform based on the at least one of GPS information, heading orientation information, scale information, and feature extraction parameters.

33. The system of claim 25, wherein the received feedback information facilitates a personalized search.

34. The system of claim 25, wherein the received feedback information is used to build a collaborative search system.

35. The system of claim 25, wherein the means for updating the database comprises means for using the received feedback information to update the popularity of at least one of objects and views of the objects based on the number of times the at least one of objects and views is queried and the number of times a feature descriptor match occurs.

36. A computer-readable medium including program code stored thereon, comprising:

program code to analyze received feedback information from a mobile platform, the received feedback information including information with respect to an image of an object captured by the mobile platform;

program code to update a database of information of objects and images of the objects using the received feedback information.

37. The computer-readable medium of claim 36, wherein the program code to update the database comprises program code to at least one of improve the database pruning, learn user-generated content by adding the feedback information to the database, and update the database compression efficiency.

38. The computer-readable medium of claim 36, wherein the program code to update the database comprises program code to determine the received feedback information is related to an object that is not in the database;

and add the object to the database.

39. The computer-readable medium of claim 38, wherein the program code to add the object to the database comprises:

program code to perform intra-object pruning for the object, the intra-object pruning comprising;

identify a set of matching keypoint descriptors for a plurality of keypoint descriptors for the object;

remove one or more of the matching keypoint descriptors within each set of matching keypoint descriptors, wherein subsequent to the removal of the one or more of the matching keypoint descriptors there is at least one remaining keypoint descriptor in each set of matching keypoint descriptors;

program code to perform inter-object pruning for the object with respect to the database, the inter-object pruning comprising;

characterize discriminability of the remaining keypoint descriptors;

remove remaining keypoint descriptors with discriminability based on a threshold;

program code to select descriptors for the object to be retained in the database; and

program code to store the descriptors in the database.

40. The computer-readable medium of claim 36, wherein the program code to update the database comprises program code to determine the received feedback information is related to an object that is in the database and update probabilities of keypoint descriptors stored in the database belonging to the object.

41. The computer-readable medium of claim 40, further comprising:

program code to perform inter-object pruning for the object with respect to the database, the inter-object pruning comprising;

characterize discriminability of the remaining keypoint descriptors using the updated probabilities;

remove remaining keypoint descriptors with discriminability based on a threshold;

program code to select descriptors for the object to be retained in the database; and

program code to store the descriptors in the database.

42. The computer-readable medium of claim 40, further comprising:

program code to determine to add the image of the object to the database;

program code to perform intra-object pruning for the object, the intra-object pruning comprising;

identify a set of matching keypoint descriptors for a plurality of keypoint descriptors for the object;

remove one or more of the matching keypoint descriptors within each set of matching keypoint descriptors, wherein subsequent to the removal of the one or more of the matching keypoint descriptors there is at least one remaining keypoint descriptor in each set of matching keypoint descriptors;

program code to perform inter-object pruning for the object with respect to the database, the inter-object pruning comprising;

characterize discriminability of the remaining keypoint descriptors using the updated probabilities;

remove remaining keypoint descriptors with discriminability based on a threshold;

program code to select descriptors for the object to be retained in the database; and

program code to store the descriptors in the database.

43. The computer-readable medium of claim 36, wherein the received feedback information comprises at least one of GPS information, heading orientation information, and scale information, the computer-readable medium further comprising program code to provide information from the database to the mobile platform based on the at least one of GPS information, heading orientation information, scale information, and feature extraction parameters.

44. The computer-readable medium of claim 36, wherein the received feedback information facilitates a personalized search.

45. The computer-readable medium of claim 36, wherein the received feedback information is used to build a collaborative search system.

46. The computer-readable medium of claim 36, wherein the program code to update the database comprises program code to use the received feedback information to update the popularity of at least one of objects and views of the objects based on the number of times the at least one of objects and views is queried and the number of times a feature descriptor match occurs.

47. A method comprising:

receiving a feature database from a server;

capturing a query image of an object;

extracting query features from the query image of the object;

performing a search of the feature database using the extracted query features; and

providing feedback information to the server based on the performed search of the feature database.

48. The method of claim 47, wherein the feedback information comprises at least one of:

the query image, features extracted from the query image, a confidence level for the features, posterior probabilities of the features belonging to an object in the database, GPS information, heading orientation information, scale information, and feature extraction parameters.

49. The method of claim 47, further comprising:

determining the object in the query image does not belong to the feature database; and

providing the query image, features extracted from the query image, a confidence level for the features, posterior probabilities of the features belonging to an object in the database to the server.

50. The method of claim 49, wherein the determining the object in the query image does not belong to feature database comprises:

determining the probabilities of features extracted from the query image belonging to an object stored in the feature database;

generating a confidence measure based on the determined probabilities; and

determining whether the confidence measure is greater than a threshold.

51. The method of claim 47, further comprising:

determining the object in the query image belongs to the feature database; and

providing an application context, an object identifier and view identifier to the server.

52. An apparatus comprising:

an external interface for receiving a feature database from a server;

a camera for capturing an image;

a processor connected to the external interface and camera;

memory connected to the processor; and

software held in the memory and run in the processor to extract query features from the captured image, to perform a search of the feature database using the extracted query features and to provide feedback information using the external interface based on the performed search of the feature database.

53. The apparatus of claim 52, wherein the feedback information comprises at least one of:

the query image, features extracted from the query image, a confidence level for the features, posterior probabilities of the features belonging to an object in the database, GPS information, heading orientation information, scale information, and feature extraction parameters.

54. The apparatus of claim 52, further comprising software held in the memory and run in the processor to:

determine the object in the query image does not belong to the feature database; and

provide the query image, features extracted from the query image, a confidence level for the features, posterior probabilities of the features belonging to an object in the database to the server.

55. The apparatus of claim 54, wherein the software to determine the object in the query image does not belong to feature database comprises software that causes the processor to:

determine the probabilities of features extracted from the query image belonging to an object stored in the feature database;

generate a confidence measure based on the determined probabilities; and

determine whether the confidence measure is greater than a threshold.

56. The apparatus of claim 52, further comprising software held in the memory and run in the processor to:

determine the object in the query image belongs to the feature database; and

provide an application context, the object identifier and view identifier to the server.

57. A system comprising:

means for receiving a feature database from a server;

means for capturing a query image;

means for extracting query features from the captured image;

means for performing a search of the feature database using the extracted query features; and

means for providing feedback information using an external interface based on the performed search of the feature database.

58. The system of claim 57, wherein the feedback information comprises at least one of:

the query image, features extracted from the query image, a confidence level for the features, posterior probabilities of the features belonging to an object in the database, GPS information, heading orientation information, scale information, and feature extraction parameters.

59. The system of claim 57, further comprising:

means for determining the object in the query image does not belong to the feature database; and

means for providing the query image, features extracted from the query image, a confidence level for the features, posterior probabilities of the features belonging to an object in the database to the server.

60. The system of claim 59, wherein the means for determining the object in the query image does not belong to feature database comprises:

means for determining the probabilities of features extracted from the query image belonging to an object stored in the feature database;

means for generating a confidence measure based on the determined probabilities; and

means for determining whether the confidence measure is greater than a threshold.

61. The system of claim 57, further comprising:

means for determining the object in the query image belongs to the feature database; and

means for providing an application context, the object identifier and view identifier to the server.

62. A computer-readable medium including program code stored thereon, comprising:

program code to extract query features from a captured image;

program code to perform a search of a feature database using the extracted query features;

program code to determine information to feedback to a server based on the performed search of the feature database; and

program code to transmit the determined information to the server.

63. The computer-readable medium of claim 62, further comprising:

program code to determine the object in the query image does not belong to the feature database; and

program code to provide the query image, features extracted from the query image, a confidence level for the features, posterior probabilities of the features belonging to an object in the database to the server.

64. The computer-readable medium of claim 63, wherein the program code to determine the object in the query image does not belong to feature database comprises:

program code to determine the probabilities of features extracted from the query image belonging to an object stored in the feature database;

program code to generate a confidence measure based on the determined probabilities; and

program code to determine whether the confidence measure is greater than a threshold.

65. The computer-readable medium of claim 62, further comprising:

program code to determine the object in the query image belongs to the feature database; and

program code to provide an application context, the object identifier and view identifier to the server.