OBJECT RECOGNITION SYSTEM WITH DATABASE PRUNING AND QUERYING

• US 20120011119A1
• Filed: 07/08/2010
• Published: 01/12/2012
• Est. Priority Date: 07/08/2010
• Status: Abandoned Application

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Baheti, Pawan K., Swaminathan, Ashwin, Spindola, Serafin Diaz, Ning, Xia
• Application Number:
  US12/832,796
• Publication Number:
  US 20120011119A1
• Time in Patent Office: Days
• US Class Current: 707/737
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 18/211 Selection of the most significant subset of features
• CPC Class Codes: G06F 18/2321 using statistics or function optimisation, e.g. modelling of probability density functions
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations
• CPC Class Codes: G06V 10/771 Feature selection, e.g. selecting representative features from a multi-dimensional feature space

Abstract

A database for object recognition is generated by performing at least one of intra-object pruning and inter-object pruning, as well as keypoint clustering and selection. Intra-object pruning removes similar and redundant keypoints within an object and different views of the same object, and may be used to generate and associate a significance value, such as a weight, with respect to remaining keypoint descriptors. Inter-object pruning retains the most informative set of descriptors across different objects, by characterizing the discriminability of the keypoint descriptors for all of the objects and removing keypoint descriptors with a discriminability that is less than a threshold. Additionally, a mobile platform may download a geographically relevant portion of the database and perform object recognition by extracting features from the query image and using determined confidence levels for each query feature during outlier removal.

Claims

1. A method of building a database for information of objects and images of the objects, the method comprising:

extracting keypoints and generating keypoint descriptors in a plurality of images of a plurality of objects;

performing intra-object pruning for at least one object, the intra-object pruning comprising;

identifying a set of matching keypoint descriptors for a plurality of keypoint descriptors in each image of the at least one object;

removing one or more of the matching keypoint descriptors within each set of matching keypoint descriptors, wherein subsequent to the removal of the one or more of the matching keypoint descriptors there is at least one remaining keypoint descriptor in each set of matching keypoint descriptors;

performing inter-object pruning for a plurality of objects, the inter-object pruning comprising;

characterizing discriminability of the remaining keypoint descriptors;

removing remaining keypoint descriptors with discriminability based on a threshold;

clustering keypoints in each image based on location and retaining a subset of keypoints in each cluster of keypoints;

associating remaining keypoints with an object identifier; and

storing the associated remaining keypoints and object identifier.

2. The method of claim 1, wherein identifying a set of matching keypoint descriptors comprises:

comparing each keypoint descriptor to a plurality of keypoint descriptors from each image of the same object to find a match between keypoint descriptors;

comparing each match between keypoint descriptors to a second threshold and placing keypoint descriptors in a set of matching keypoint descriptors based on the comparison to the second threshold.

3. The method of claim 1, wherein the intra-object pruning further comprises determining and assigning a significance for the remaining keypoint descriptors in each set of matching keypoint descriptors and wherein characterizing discriminability of the remaining keypoint descriptors is based on the assigned significance.

4. The method of claim 3, wherein weight is used to assign the significance for the remaining keypoint descriptors.

5. The method of claim 3, wherein the significance for the remaining keypoint descriptors is determined based on the number of keypoint descriptors in the set of matching keypoint descriptors before removing the one or more of the matching keypoint descriptors.

6. The method of claim 1, further comprising compressing the keypoint descriptors.

7. The method of claim 1, further comprising pruning keypoints in each image based on location by identifying keypoints having a same location and removing one or more keypoints having the same location, wherein subsequent to the removal of the one or more keypoints having the same location there is at least one remaining keypoint for the same location.

8. The method of claim 7, wherein pruning keypoints in each image based on location comprises retaining keypoint with a largest scale for each location.

9. The method of claim 1, wherein clustering keypoints in each image is performed before performing the intra-object pruning.

10. The method of claim 1, wherein clustering keypoints in each image is performed after performing the inter-object pruning.

11. The method of claim 1, wherein removing one or more of the matching keypoint descriptors within each set of matching keypoint descriptors comprises retaining at least one of the matching keypoint descriptors and removing the remaining keypoint descriptors.

12. The method of claim 1, wherein removing one or more of the matching keypoint descriptors within each set of matching keypoint descriptors comprises compounding the matching keypoint descriptors into the remaining keypoint descriptor and removing all of the matching keypoint descriptors.

13. The method of claim 1, wherein removing one or more of the matching keypoint descriptors within each set of matching keypoint descriptors comprises retaining at least one of the keypoint location, scale information, object and view association for each of the removed matching keypoint descriptors.

14. The method of claim 1, wherein characterizing discriminability of the remaining keypoint descriptors comprises:

quantifying a probability for each remaining keypoint descriptor of belonging to any of the plurality of objects; and

determining an entropy measure using the quantified probability for each remaining keypoint descriptor to determine discriminability measure, wherein the determined entropy measure is compared to a second threshold.

15. The method of claim 1, wherein characterizing discriminability of the remaining keypoint descriptors comprises determining a distance between remaining keypoint descriptors that do not belong to the same object, wherein the determined distance is compared to a second threshold.

16. The method of claim 1, wherein clustering keypoints in each image comprises retaining the subset of keypoints with largest scales in each cluster of keypoints.

17. A method of building a database for information of objects and images of the objects, the method comprising:

extracting keypoints and generating keypoint descriptors in a plurality of images of a plurality of objects;

performing inter-object pruning for a plurality of objects, the inter-object pruning comprising;

characterizing discriminability of the keypoint descriptors;

removing keypoint descriptors with discriminability based on a threshold;

clustering keypoints in each image based on location and retaining a subset of keypoints in each cluster of keypoints;

associating keypoints with an object identifier; and

storing the associated keypoints and object identifier.

18. A method of building a database for information of objects and images of the objects, the method comprising:

extracting keypoints and generating keypoint descriptors in a plurality of images of a plurality of objects;

performing intra-object pruning for at least one object, the intra-object pruning comprising;

identifying a set of matching keypoint descriptors for a plurality of keypoint descriptors in each image of the at least one object;

removing one or more of the matching keypoint descriptors within each set of matching keypoint descriptors, wherein subsequent to the removal of the one or more of the matching keypoint descriptors there is at least one remaining keypoint descriptor in each set of matching keypoint descriptors;

clustering keypoints in each image based on location and retaining a subset of keypoints based on scale in each cluster of keypoints;

associating remaining keypoints with an object identifier; and

storing the associated remaining keypoints and object identifier.

19. An apparatus comprising:

an external interface for receiving a plurality of images to be processed and stored in a database, the plurality of images containing a plurality of views of a plurality of objects;

a processor connected to the external interface;

memory connected to the processor; and

software held in the memory and run in the processor to extract keypoints and generate keypoint descriptors in the plurality of images, to perform intra-object pruning for at least one object, the intra-object pruning comprising;

identifying a set of matching keypoint descriptors for a plurality of keypoint descriptors in each image of the at least one object;

removing one or more of the matching keypoint descriptors within each set of matching keypoint descriptors, wherein subsequent to the removal of the one or more of the matching keypoint descriptors there is at least one remaining keypoint descriptor in each set of matching keypoint descriptors;

to perform inter-object pruning for a plurality of objects, the inter-object pruning comprising;

characterizing discriminability of the remaining keypoint descriptors;

removing remaining keypoint descriptors with discriminability based on a threshold;

to cluster keypoints in each image based on location and retain a subset of keypoints in each cluster of keypoints, to associate remaining keypoints with an object identifier; and

to store the associated remaining keypoints and object identifier in the database.

20. A system comprising:

means for receiving a plurality of images to be processed and stored in a database, the plurality of images containing a plurality of views of a plurality of objects;

means for extracting keypoints and generating keypoint descriptors in the plurality of images;

means for performing intra-object pruning comprising;

identifying a set of matching keypoint descriptors for a plurality of keypoint descriptors in each image of the plurality of objects;

removing one or more of the matching keypoint descriptors within each set of matching keypoint descriptors, wherein subsequent to the removal of the one or more of the matching keypoint descriptors there is at least one remaining keypoint descriptor in each set of matching keypoint descriptors;

means for performing inter-object pruning for a plurality of objects, the inter-object pruning comprising;

characterizing discriminability of the remaining keypoint descriptors;

removing remaining keypoint descriptors with discriminability based on a threshold;

means for clustering keypoints in each image based on location and retaining a subset of keypoints in each cluster of keypoints;

means for associating remaining keypoints with an object identifier; and

means for storing the associated remaining keypoints and object identifier in the database.

21. A computer-readable medium including program code stored thereon, comprising:

program code to extract keypoints and generate keypoint descriptors from a plurality of images;

program code to perform intra-object pruning including identifying a set of matching keypoint descriptors for a plurality of keypoint descriptors in each image of each object and removing one or more of the matching keypoint descriptors within each set of matching keypoint descriptors, wherein subsequent to the removal of the one or more of the matching keypoint descriptors there is at least one remaining keypoint descriptor in each set of matching keypoint descriptors;

program code to perform inter-object pruning for a plurality of objects in the plurality of images including characterizing discriminability of the remaining keypoint descriptors and removing remaining keypoint descriptors with discriminability based on a threshold;

program code to cluster keypoints in each image based on location and retain a subset of keypoints in each cluster of keypoints;

program code to associate remaining keypoints with an object identifier; and

program code to store the associated remaining keypoints and object identifier in a database.

22. A method of determining at least one best match between a query image and information related to images of objects in a database using extracted keypoint descriptors from the query image and keypoint descriptors in the database, the method comprising:

performing a search of the database using the keypoint descriptors from the query image to retrieve neighbors;

determining a quality of match for each retrieved neighbor with respect to associated keypoint descriptor from the query image;

using the determined quality of match for each retrieved neighbor to generate an object candidate set;

removing outliers from the object candidate set using the determined quality of match for each retrieved neighbor to provide the at least one best match; and

storing the at least one best match.

23. The method of claim 22, wherein the at least one best match is one of a best object match and a best view match.

24. The method of claim 22, wherein removing outliers from the object candidate set comprises filtering the object candidate set based on keypoint descriptor distance between the keypoint descriptors for the query image and keypoint descriptors of objects in the object candidate set.

25. The method of claim 24, wherein filtering the object candidate set comprises:

determining a number of keypoint descriptor matches for each object in each view in the object candidate set by identifying the number of keypoint descriptors of the object in the object candidate set that is less than a threshold distance from the keypoint descriptors from the query image; and

retaining a subset of objects in the object candidate set with a greatest number of keypoint descriptor matches.

26. The method of claim 22, wherein removing outliers from the object candidate set comprises filtering the object candidate set based on orientation.

27. The method of claim 26, wherein filtering the object candidate set based on orientation comprises:

determining keypoint descriptor orientation differences between keypoint descriptors from the query image and keypoint descriptors for each object;

computing a histogram of the keypoint descriptor orientation difference; and

retaining objects in the object candidate set having a subset of inliers that are within a threshold keypoint descriptor orientation difference.

28. The method of claim 22, wherein removing outliers from the object candidate set comprises filtering the object candidate set based on geometry, wherein a pose estimation of the object in the query image is provided.

29. The method of claim 28, wherein filtering the object candidate set based on geometry comprises fitting an affine model to matching keypoints descriptors pairs from the query image and objects in the object candidate set to determine a set of inliers of each object.

30. The method of claim 28, wherein filtering the object candidate set based on geometry comprises computing a homography and estimate a pose of the query image with respect to the object image.

31. The method of claim 22, wherein performing a search of an object database comprises:

determining a distance between the keypoint descriptors from the query image and keypoint descriptors from objects in the object database;

comparing the determined distance to a threshold;

storing keypoint descriptors and an associated object identification as nearest neighbors when the determined distance is less than the threshold.

32. The method of claim 22, wherein performing a search of an object database comprises:

determining a distance between the keypoint descriptors from the query image and keypoint descriptors from objects in the object database to determine a closest neighbor and next closest neighbor;

computing a ratio of a distance between the closest neighbor and the next closest neighbor for each keypoint descriptor from the query image; and

comparing the ratio to a threshold.

33. The method of claim 22, wherein determining the quality of match for each retrieved neighbor with respect to associated keypoint descriptor from the query image comprises at least one of computing posterior probabilities, computing distance ratios of distances between keypoint descriptors from the query image and retrieved neighbors and distances between two retrieved neighbors, and determining distances between keypoint descriptors from the query image and the retrieved neighbors.

34. The method of claim 33, wherein computing posterior probabilities comprises quantifying the probability for each query descriptor of belonging to any of a plurality of objects.

35. The method of claim 22, wherein the database contains keypoint descriptors with different weights for a plurality of objects and determining the quality of the match uses the weights of the keypoint descriptors.

36. The method of claim 22, wherein determining the quality of the match comprises computing confidence scores by determining entropy measures for the retrieved neighbors.

37. A mobile platform comprising:

a camera for capturing a query image;

a database of information with respect to reference objects and their images;

a processor connected to receive the query image;

memory connected to the processor;

a display connected to the memory; and

software held in the memory and run in the processor to extract keypoints and generate descriptors from the query image, to perform a search of the database using the keypoint descriptors from the query image to retrieve neighbors;

to determine a quality of match for each retrieved neighbor with respect to associated keypoint descriptor from the query image;

to use the determined quality of match for each retrieved neighbor to generate an object candidate set;

to remove outliers from the object candidate set using the determined quality of match for each retrieved neighbor to provide at least one best match, and to store the at least one best match.

38. A system for determining at least one best match between a query image and information related to images of objects in a database using extracted keypoint descriptors from the query image and keypoint descriptors in the database, the system comprising:

means for performing a search of the database using the keypoint descriptors from the query image to retrieve neighbors;

means for determining a quality of match for each retrieved neighbor with respect to associated keypoint descriptor from the query image;

means for using the determined quality of match for each retrieved neighbor to generate an object candidate set;

means for removing outliers from the object candidate set using the determined quality of match for each retrieved neighbor to provide the at least one best match; and

means for storing the at least one best match.

39. A computer-readable medium including program code stored thereon, comprising:

program code to perform a search of a database using extracted keypoint descriptors from a query image to retrieve neighbors;

program code to determine a quality of match for each retrieved neighbor with respect to associated keypoint descriptor from the query image;

program code to use the determined quality of match for each retrieved neighbor to generate an object candidate set;

program code to remove outliers from the object candidate set using the determined quality of match for each retrieved neighbor to provide at least one best match; and

program code to store the at least one best match.