Systems and methods for matching visual object components

• US 9,117,146 B2
• Filed: 12/31/2013
• Issued: 08/25/2015
• Est. Priority Date: 07/13/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Li, Yuan, Adam, Hartwig
• Primary Examiner(s): Koziol, Stephen R
• Assistant Examiner(s): CONNER, SEAN M
• Application Number:
  US14/145,279
• Publication Number:
  US 20140112576A1
• Time in Patent Office: 602 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06V 10/75 Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49

Abstract

Systems and methods for modeling the occurrence of common image components (e.g., sub-regions) in order to improve visual object recognition are disclosed. In one example, a query image may be matched to a training image of an object. A matched region within the training image to which the query image matches may be determined and a determination may be made whether the matched region is located within an annotated image component of the training image. When the matched region matches only to the image component, an annotation associated with the component may be identified. In another example, sub-regions within a plurality of training image corpora may be annotated as common image components including associated information (e.g., metadata). Matching sub-regions appearing in many training images of objects may be down-weighted in the matching process to reduce possible false matches to query images including common image components.

Claims

1. A computer-implemented method comprising:

receiving a search query that includes a query image of an object;

identifying a training image of the object;

identifying a particular sub-region of the training image that a visual object recognition engine indicates as matching the query image;

determining, by one or more computers, that the particular sub-region of the training image of the object is located within an annotated sub-region of the training image, wherein the annotated sub-region is associated with an annotation; and

in response to determining that the particular sub-region of the training image of the object is located within the annotated sub-region of the training image, providing the annotation for output in response to the search query.

2. The method of claim 1, wherein identifying a training image of the object comprises:

identifying the object in the query image; and

identifying an object in the training image that corresponds to the object in the query image.

3. The method of claim 2, wherein identifying the object in the query image comprises:

removing background information in the query image to obtain a normalized query image; and

identifying the object in the normalized query image.

4. The method of claim 1, wherein determining that the particular sub-region of the training image of the object is located within an annotated sub-region of the training image comprises determining that the particular sub-region of the training image of the object is located within the annotated sub-region in response to determining that the particular sub-region of the training image of the object is completely located within the annotated sub-region of the training image.

5. The method of claim 1, wherein determining that the particular sub-region of the training image of the object is located within an annotated sub-region of the training image comprises determining that the particular sub-region of the training image of the object is located within the annotated sub-region in response to determining that a percentage of the particular sub-region of the training image of the object located within the annotated sub-region of the training image meets a threshold.

6. The method of claim 1, wherein determining that the particular sub-region of the training image of the object is located within an annotated sub-region of the training image comprises determining that a match score between the particular sub-region of the training image of the object and the annotated sub-region of the training image meets a threshold.

7. The method of claim 1,

wherein identifying a particular sub-region of the training image that a visual object recognition engine indicates as matching the query image comprises:

identifying the object in the particular sub-region of the training image of the object.

8. The method of claim 1, wherein the annotation includes metadata associated with an object in the particular sub-region of the training image.

9. A non-transitory computer-readable medium storing software having stored thereon instructions, which, when executed by one or more computers, cause the one or more computers to perform operations of:

receiving a search query that includes a query image of an object;

identifying a training image of the object;

identifying a particular sub-region of the training image that a visual object recognition engine indicates as matching the query image;

determining, by one or more computers, that the particular sub-region of the training image of the object is located within an annotated sub-region of the training image, wherein the annotated sub-region is associated with an annotation; and

in response to determining that the particular sub-region of the training image of the object is located within the annotated sub-region of the training image, providing the annotation for output in response to the search query.

10. The computer-readable medium of claim 9, wherein identifying a training image of the object comprises:

identifying the object in the query image; and

identifying an object in the training image that corresponds to the object in the query image.

11. The computer-readable medium of claim 10, wherein identifying the object in the query image comprises:

removing background information in the query image to obtain a normalized query image; and

identifying the object in the normalized query image.

12. The computer-readable medium of claim 9, wherein determining that the particular sub-region of the training image of the object is located within an annotated sub-region of the training image comprises determining that the particular sub-region of the training image of the object is located within the annotated sub-region in response to determining that the particular sub-region of the training image of the object is completely located within the annotated sub-region of the training image.

13. The computer-readable medium of claim 9, wherein determining that the particular sub-region of the training image of the object is located within an annotated sub-region of the training image comprises determining that the particular sub-region of the training image of the object is located within the annotated sub-region in response to determining that a percentage of the particular sub-region of the training image of the object located within the annotated sub-region of the training image meets a threshold.

14. The computer-readable medium of claim 9, wherein determining that the particular sub-region of the training image of the object is located within an annotated sub-region of the training image comprises determining that a match score between the particular sub-region of the training image of the object and the annotated sub-region of the training image meets a threshold.

15. A system comprising:

one or more processors and one or more computer storage media storing instructions that are operable, when executed by the one or more processors, to cause the one or more processors to perform operations comprising;

receiving a search query that includes a query image of an object;

identifying a training image of the object;

identifying a particular sub-region of the training image that a visual object recognition engine indicates as matching the query image;

determining, by one or more computers, that the particular sub-region of the training image of the object is located within an annotated sub-region of the training image, wherein the annotated sub-region is associated with an annotation; and

in response to determining that the particular sub-region of the training image of the object is located within the annotated sub-region of the training image, providing the annotation for output in response to the search query.

16. The system of claim 15, wherein identifying a training image of the object comprises:

identifying the object in the query image; and

identifying an object in the training image that corresponds to the object in the query image.

17. The system of claim 16, wherein identifying the object in the query image comprises:

removing background information in the query image to obtain a normalized query image; and

identifying the object in the normalized query image.

18. The system of claim 15, wherein determining that the particular sub-region of the training image of the object is located within an annotated sub-region of the training image comprises determining that the particular sub-region of the training image of the object is located within the annotated sub-region in response to determining that the particular sub-region of the training image of the object is completely located within the annotated sub-region of the training image.

19. The system of claim 15, wherein determining that the particular sub-region of the training image of the object is located within an annotated sub-region of the training image comprises determining that the particular sub-region of the training image of the object is located within the annotated sub-region in response to determining that a percentage of the particular sub-region of the training image of the object located within the annotated sub-region of the training image meets a threshold.

20. The system of claim 15, wherein determining that the particular sub-region of the training image of the object is located within an annotated sub-region of the training image comprises determining that a match score between the particular sub-region of the training image of the object and the annotated sub-region of the training image meets a threshold.