Label consistency for image analysis

• US 10,445,623 B2
• Filed: 04/14/2017
• Issued: 10/15/2019
• Est. Priority Date: 12/20/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Bengio, Samy, Dean, Jeffrey Adgate, Le, Quoc V., Shlens, Jonathon, Singer, Yoram
• Primary Examiner(s): Potts, Ryan P
• Application Number:
  US15/488,041
• Publication Number:
  US 20170220906A1
• Time in Patent Office: 914 Days
• Field of Search: None
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 10/758 Involving statistics of pixels or of feature values, e.g. histogram matching
• CPC Class Codes: G06V 10/759 Region-based matching
• CPC Class Codes: G06V 10/768 using context analysis, e.g. recognition aided by known co-occurring patterns
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 30/268 Lexical context

Abstract

Systems and techniques are disclosed for labeling objects within an image. The objects may be labeled by selecting an option from a plurality of options such that each option is a potential label for the object. An option may have an option score associated with. Additionally, a relation score may be calculated for a first option and a second option corresponding to a second object in an image. The relation score may be based on a frequency, probability, or observance corresponding to the co-occurrence of text associated with the first option and the second option in a text corpus such as the World Wide Web. An option may be selected as a label for an object based on a global score calculated based at least on an option score and relation score associated with the option.

Claims

1. A computer implemented method comprising:

receiving a first option for a first patch within an image, the first option being a first text label, and the first patch being a first portion of the image depicting a first object;

receiving a second option for a second patch within the image, the second option being a second text label, and the second patch being a second portion of the image depicting a second object;

generating a first option score for the first patch, the first option score being indicative of a likelihood that the first text label corresponds to the first object;

generating a first relation score based on a consistency between the first option and the second option, the first relation score based on a number of co-occurrences of the first text label and the second text label in text of a set of documents, wherein each document in the set of documents is different than the image; and

generating a first global score for the first patch based on the first option score and the first relation score, wherein the first global score is directly proportional to the first option score and the relation score such that an increase in the first option score or an increase in the first relation score generates an increase in the first global score.

2. The method of claim 1, further comprising designating the first option as a label for the first object in the image if the first global score exceeds a global score threshold.

3. The method of claim 1, further comprising:

receiving a third option for the first patch within the image;

generating a second option score for a first patch;

generating a second relation score based on a number of co-occurrences of the second option and the third option in text of a set of documents;

generating a second global score for the first patch based on the third option score and the second relation score;

comparing the first global score to the second global score; and

designating the first option as a label for the first object in the image based on the first global score exceeding the second global score.

4. The method of claim 1, wherein the set of documents is a set of web pages.

5. The method of claim 1, wherein first patch and the second patch partially overlap in the image.

6. The method of claim 1, wherein the first patch and the second patch are separate from each other in the image.

7. The method of claim 1, wherein generating a first global score for the first patch based on the first option score and the first relation score comprises summing the first option score with the first relation score.

8. A system comprising:

a processor configured to;

receive a first option for a first patch within an image, the first option being a first text label, and the first patch being a first portion of the image depicting a first object;

receive a second option for a second patch within the image, the second option being a second text label, and the second patch being a second portion of the image depicting a second object;

generate a first option score for the first patch, the first option score being indicative of the likelihood that the first text label corresponds to the first object;

generate a first relation score based on a consistency between the first option and the second option, the first relation score based on a number of co-occurrences of the first text label and the second text label in text of a set of documents, wherein each document in the set of documents is different than the image; and

generate a first global score for the first patch based on the first option score and the first relation score, wherein the first global score is directly proportional to the first option score and the relation score such that an increase in the first option score or an increase in the first relation score generates an increase in the first global score.

9. The system of claim 8, further configured to designate the first option as a label for the first object in the image if the first global score exceeds a global score threshold.

10. The system of claim 8, further configured to:

receive a third option for the first patch within the image;

generate a second option score for a first patch;

generate a second relation score based on a number of co-occurrences of the second option and the third option in text of a set of documents;

generate a second global score for the first patch based on the third option score and the second relation score;

compare the first global score to the second global score; and

designate the first option as a label for the first object in the image based on the comparison based on the first global score exceeding the second global score.

11. The system of claim 8, wherein the set of documents is a set of web pages.

12. The system of claim 8, wherein first patch and the second patch partially overlap in the image.

13. The system of claim 8, wherein the first patch and the second patch are separate from each other in the image.

14. The system of claim 8, wherein generating a first global score for the first patch based on the first option score and the first relation score comprises summing the first option score with the first relation score.