Systems and methods for matching visual object components

• US 8,625,887 B2
• Filed: 07/13/2011
• Issued: 01/07/2014
• Est. Priority Date: 07/13/2011
• Status: Expired due to Fees

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Li, Yuan, Adam, Hartwig
• Primary Examiner(s): Koziol, Stephen R
• Assistant Examiner(s): CONNER, SEAN M
• Application Number:
  US13/182,076
• Publication Number:
  US 20130016899A1
• Time in Patent Office: 909 Days
• Field of Search: None
• US Class Current: 382/159
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06V 10/75 Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49

Abstract

Systems and methods for modeling the occurrence of common image components (e.g., sub-regions) in order to improve visual object recognition are disclosed. In one example, a query image may be matched to a training image of an object. A matched region within the training image to which the query image matches may be determined and a determination may be made whether the matched region is located within an annotated image component of the training image. When the matched region matches only to the image component, an annotation associated with the component may be identified. In another example, sub-regions within a plurality of training image corpora may be annotated as common image components including associated information (e.g., metadata). Matching sub-regions appearing in many training images of objects may be down-weighted in the matching process to reduce possible false matches to query images including common image components.

Claims

1. A method comprising:

receiving a query image;

matching the query image to an object using a visual object recognition module, wherein the visual object recognition module is configured to utilize information from a plurality of training image corpora to match the query image to a training image of the object;

determining a matched region within the training image to which the query image matches using the visual object recognition module; and

responsive to determining the matched region;

determining whether the matched region is located within an annotated image component of the training image, wherein the annotated image component is a sub-region of the training image and has an associated annotation describing the sub-region, and

when the matched region is located within the annotated image component of the training image, identifying an annotation associated with the annotated image component.

2. The method of claim 1, further comprising identifying the annotation associated with the annotated image component when a percentage of the matched region located within the annotated image component is greater than a percentage threshold.

3. The method of claim 1, further comprising identifying an annotation associated with an entirety of the training image when a percentage of the matched region located within the annotated image component is less than a percentage threshold.

4. The method of claim 1, further comprising:

receiving the query image from a client device; and

sending the identified annotation associated with the annotated image component to the client device.

5. The method of claim 1, further comprising:

receiving the plurality of training image corpora, wherein the plurality of training image corpora comprise a plurality of training images of a plurality of objects, wherein one or more training images comprise metadata related to the plurality of objects;

performing cross-corpus image region matching among the plurality of training image corpora; and

for matched regions within the plurality of training image corpora, annotating the matched regions as sub-regions within the training images comprising associated information.

6. The method of claim 1, further comprising

matching the query image to one or more training images of the plurality of objects;

determining matched regions within the one or more training images to which the query image matches;

determining match scores indicating a level of similarity between the query image and the matched regions; and

selecting the training image associated with a maximum match score as a match to the query image.

7. The method of claim 6, further comprising:

performing cross-corpus matching among the plurality of training image corpora to identify common image components appearing in more than one training image;

identifying image descriptors appearing in more than one training image, wherein image descriptors comprise information associated with image interest points within a training image;

assigning a weighting factor to image descriptors associated with the common image components, wherein the weighting factor influences the match scores between the query image and the matched regions; and

determining adjusted match scores based on the weighting factor.

8. The method of claim 7, wherein the common image components comprise one or more of the following identified within multiple training images:

similar text, similar fonts, similar patterns, and stock images.

9. The method of claim 7, further comprising:

performing a plurality of matching experiments on a validation set using a series of weighting factors, wherein the validation set is a set of query images comprising one or more of the common image components;

determining matching accuracies associated with results of the plurality of matching experiments performed using the series of weighting factors; and

selecting the weighting factor yielding a maximum matching accuracy.

10. The method of claim 6, further comprising:

assigning a weighting factor to image descriptors of the training images, wherein image descriptors comprise information associated with image interest points within a training image; and

determining adjusted match scores from the visual object recognition module based on the weighting factor.

11. The method of claim 10, wherein the weighting factors are computed using the following formula:

(1/x)*log(D/d),

wherein x is a total number of image descriptors in the training image, D is a total number of training images in the plurality of training image corpora, and d is a total number of training images in which the image descriptor appears.

12. The method of claim 10, further comprising the visual object recognition module computing the adjusted match scores using the following formula:

13. The method of claim 7, wherein performing cross-corpus matching further comprises:

using optical character recognition (OCR) to recognize characters in the plurality of training image corpora; and

comparing the recognized characters to identify common image components appearing in more than one training image.

14. A non-transitory computer readable medium having stored therein instructions executable by a computing device to cause the computing device to perform functions of:

receiving a query image;

matching the query image to an object using a visual object recognition module, wherein the visual object recognition module is configured to utilize information from a plurality of training image corpora to match the query image to a training image of the object;

determining a matched region within the training image to which the query image matches using the visual object recognition module; and

responsive to determining the matched region;

determining whether the matched region is located within an annotated image component of the training image, wherein the annotated image component is a sub-region of the training image and has an associated annotation describing the sub-region, and

when the matched region is located within the annotated image component of the training image, identifying an annotation associated with the annotated image component.

15. The computer readable medium of claim 14, further comprising instructions to perform functions of:

receiving a plurality of training image corpora, wherein the plurality of training image corpora comprise a plurality of training images of a plurality of objects, wherein one or more training images comprise metadata related to the plurality of objects;

performing cross-corpus image region matching among the plurality of training image corpora; and

for matched regions within the plurality of training image corpora, annotating the matched regions as sub-regions within the training images comprising associated information.

16. The computer readable medium of claim 14, further comprising instructions to perform functions of:

matching the query image to one or more training images of the plurality of objects;

determining matched regions within the one or more training images to which the query image matches;

performing cross-corpus matching among the plurality of training image corpora to identify common image components appearing in more than one training image;

identifying image descriptors appearing in more than one training image, wherein image descriptors comprise information associated with image interest points within a training image;

assigning a weighting factor to image descriptors associated with the common image components, wherein the weighting factor influences match scores between the query image and the matched regions;

determining match scores based on the weighting factor; and

selecting the training image associated with a maximum match score as a match to the query image.

17. A system comprising:

a memory;

a processor coupled to the memory; and

instructions stored in the memory and executable by the processor to perform functions of;

receiving a query image;

matching the query image to an object using a visual object recognition module, wherein the visual object recognition module is configured to utilize information from the plurality of training image corpora to match the query image to a training image of the object;

determining a matched region within the training image to which the query image matches using the visual object recognition module; and

responsive to determining the matched region;

determining whether the matched region is located within an annotated image component of the training image, wherein the annotated image component is a sub-region of the training image and has an associated annotation describing the sub-region, and

when the matched region is located within the annotated image component of the training image, identifying an annotation associated with the annotated image component.

18. The system of claim 17, further comprising instructions for:

receiving the plurality of training image corpora, wherein the plurality of training image corpora comprise a plurality of training images of a plurality of objects, wherein one or more training images comprise metadata related to the plurality of objects;

performing cross-corpus image region matching among the plurality of training image corpora; and

for matched regions within the plurality of training image corpora, annotating the matched regions as sub-regions within the training images comprising associated information.

19. The system of claim 17, further comprising instructions for:

matching the query image to one or more training images of the plurality of objects;

determining matched regions within the one or more training images to which the query image matches;

performing cross-corpus matching among the plurality of training image corpora to identify common image components appearing in more than one training image;

identifying image descriptors appearing in more than one training image, wherein image descriptors comprise information associated with image interest points within a training image;

assigning a weighting factor to image descriptors associated with the common image components, wherein the weighting factor influences match scores between the query image and the matched regions;

determining match scores based on the weighting factor; and

selecting the training image associated with a maximum match score as a match to the query image.

20. The system of claim 17, wherein the query image only matches a portion of the training image.