Systems and methods for visual object matching

• US 8,577,131 B1
• Filed: 07/12/2011
• Issued: 11/05/2013
• Est. Priority Date: 07/12/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Li, Yuan, Adam, Hartwig
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): WOLDEMARIAM, AKLILU K
• Application Number:
  US13/181,015
• Time in Patent Office: 847 Days
• Field of Search: 382/159, 382/160, 382/173, 382/195, 382/224
• US Class Current: 382/159
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting

Abstract

Systems and methods for improving visual object recognition by analyzing query images are disclosed. In one example, a visual object recognition module may determine query images matching objects of a training corpus utilized by the module. Matched query images may be added to the training corpus as training images of a matched object to expand the recognition of the object by the module. In another example, relevant candidate image corpora from a pool of image data may be automatically selected by matching the candidate image corpora against user query images. Selected image corpora may be added to a training corpus to improve recognition coverage. In yet another example, objects unknown to a visual object recognition module may be discovered by clustering query images. Clusters of similar query images may be annotated and added into a training corpus to improve recognition coverage.

Claims

1. A method comprising:

receiving a plurality of query images;

matching the plurality of query images to a plurality of objects using a visual object recognition module, wherein the visual object recognition module is configured to utilize information from a training corpus to match query images to one or more training images of the plurality of objects, wherein the training corpus comprises one or more training images associated with the plurality of objects;

associating match scores with matches between the query images and one or more matched training images, wherein the visual object recognition module is configured to determine match scores indicating a level of similarity between the query images and one or more matched training images;

determining matched and unmatched query images based on the match scores associated with the query images, wherein query images with match scores above a score threshold are identified as matched query images and query images with match scores below the score threshold are identified as unmatched query images; and

based on the determination, expanding the training corpus utilized by the visual object recognition module to include the matched query images, wherein the matched query images are added as additional training images associated with the objects to which the matched training images are associated, and the matched query images are used as training images when re-training the visual object recognition module on the expanded training corpus.

2. The method of claim 1, wherein one or more of the matched query images is an image of one of the plurality of objects with a variation as compared to a training image in illumination of the object.

3. The method of claim 1, wherein one or more of the matched query images is an image of one of the plurality of objects with a variation as compared to a training image in pose of the object.

4. The method of claim 1, wherein one or more of the matched query images is an image of one of the plurality of objects with a variation as compared to a training image in shape of the object.

5. The method of claim 1, further comprising:

determining metadata associated with the one or more plurality of objects to which the matched training images are associated; and

associating the metadata with the matched query images added to the training corpus.

6. The method of claim 1, further comprising:

receiving a plurality of candidate image corpora;

matching the plurality of candidate image corpora to the unmatched query images using a second visual object recognition module, wherein the second visual object recognition module is configured to utilize information from a second training corpus comprising the unmatched query images to match images of the plurality of candidate image corpora to one or more of the unmatched query images;

determining match statistics associated with the plurality of candidate image corpora; and

based on the match statistics, selecting one or more candidate image corpora to add to the training corpus as training images.

7. The method of claim 6, further comprising, indexing the one or more selected candidate image corpora by associating metadata with images from the one or more selected candidate image corpora.

8. The method of claim 6, further comprising:

determining samples of images associated with the plurality of candidate image corpora, wherein the samples of images are used to determine match statistics associated with the plurality of candidate image corpora;

matching the samples of images to the unmatched query images;

determining match statistics associated with the samples of images; and

based on the match statistics associated with the samples of images, selecting one or more candidate image corpora to add to the training corpus as training images.

9. The method of claim 6, further comprising determining matched images within the plurality of candidate image corpora, wherein images from the plurality of candidate image corpora with a match score above a second threshold are identified as matched images, and wherein the match statistics comprise a percentage of matched images within the plurality of candidate image corpora.

10. The method of claim 9, further comprising selecting a candidate image corpus if the percentage of matched images within the candidate image corpus is above a percentage threshold.

11. The method of claim 9, further comprising selecting a candidate image corpus if a number of the matched images within the candidate image corpus is above a number threshold.

12. The method of claim 1, further comprising:

clustering the unmatched query images into groups of similar query images;

determining that a group of similar query images describes a given object; and

adding the group of similar query images to the training corpus as training images associated with the given object.

13. The method of claim 12, further comprising filtering the groups by removing groups having an amount of similar query images below a threshold.

14. The method of claim 12, further comprising:

constructing image templates for the unmatched query images, wherein the image templates comprise a set of descriptors of information for interest points identified within the unmatched query images; and

comparing the image templates to determine similar images within the unmatched query images.

15. The method of claim 12, further comprising:

determining metadata associated with the given object; and

indexing the group of similar query images by associating the metadata with the similar query images.

16. A non-transitory computer readable medium having stored therein instructions executable by a computing device to cause the computing device to perform functions of:

receiving a plurality of query images;

matching the plurality of query images to a plurality of objects using a visual object recognition module, wherein the visual object recognition module is configured to utilize information from a training corpus to match query images to one or more training images of the plurality of objects, wherein the training corpus comprises one or more training images associated with the plurality of objects;

associating match scores with matches between the query images and one or more matched training images, wherein the visual object recognition module is configured to determine match scores indicating a level of similarity between the query images and one or more matched training images;

determining matched and unmatched query images based on the match scores associated with the query images, wherein query images with match scores above a score threshold are identified as matched query images and query images with match scores below the score threshold are identified as unmatched query images; and

based on the determination, expanding the training corpus utilized by the visual object recognition module to include the matched query images, wherein the matched query images are added as additional training images associated with the objects to which the matched training images are associated, and the matched query images are used as training images when re-training the visual object recognition module on the expanded training corpus.

17. The computer readable medium of claim 16, further comprising instructions to perform functions of:

receiving a plurality of candidate image corpora;

matching the plurality of candidate image corpora to the unmatched query images using a second visual object recognition module, wherein the second visual object recognition module is configured to utilize information from a second training corpus comprising the unmatched query images to match images of the plurality of candidate image corpora to one or more of the unmatched query images;

determining match statistics associated with the plurality of candidate image corpora; and

based on the match statistics, selecting one or more candidate image corpora to add to the training corpus as training images.

18. The computer readable medium of claim 16, further comprising instructions to perform functions of:

clustering the unmatched query images into groups of similar query images;

determining that a group of similar query images describes a given object; and

adding the group of similar query images to the training corpus as training images associated with the given object.

19. A system comprising:

a memory;

a processor coupled to the memory; and

instructions stored in the memory and executable by the processor to perform functions of;

receiving a plurality of query images;

matching the plurality of query images to a plurality of objects using a visual object recognition module, wherein the visual object recognition module is configured to utilize information from a training corpus to match query images to one or more training images of the plurality of objects, wherein the training corpus comprises one or more training images associated with the plurality of objects;

associating match scores with matches between the query images and one or more matched training images, wherein the visual object recognition module is configured to determine match scores indicating a level of similarity between the query images and one or more matched training images;

determining matched and unmatched query images based on the match scores associated with the query images, wherein query images with match scores above a score threshold are identified as matched query images and query images with match scores below the score threshold are identified as unmatched query images; and

based on the determination, expanding the training corpus utilized by the visual object recognition module to include the matched query images, wherein the matched query images are added as additional training images associated with the objects to which the matched training images are associated, and the matched query images are used as training images when re-training the visual object recognition module on the expanded training corpus.

20. The system of claim 19, further comprising instructions for:

receiving a plurality of candidate image corpora;

matching the plurality of candidate image corpora to the unmatched query images using a second visual object recognition module, wherein the second visual object recognition module is configured to utilize information from a second training corpus comprising the unmatched query images to match images of the plurality of candidate image corpora to one or more of the unmatched query images;

determining match statistics associated with the plurality of candidate image corpora; and

based on the match statistics, selecting one or more candidate image corpora to add to the training corpus as training images.

21. The system of claim 19, further comprising instructions for:

clustering the unmatched query images into groups of similar query images;

determining that a group of similar query images describes a given object; and

adding the group of similar query images to the training corpus as training images associated with the given object.