Image classification

• US 8,787,683 B1
• Filed: 06/05/2013
• Issued: 07/22/2014
• Est. Priority Date: 07/17/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Yee, Yangli Hector, Bengio, Samy, Rosenberg, Charles J., Murphy-Chutorian, Erik
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): TORRES, JOSE
• Application Number:
  US13/910,493
• Time in Patent Office: 412 Days
• Field of Search: 382/155, 382/159, 382/190, 382/224, 382/305, 707/E17.004, 707/E17.019, 707/E17.02, 707/E17.023
• US Class Current: 382/224
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/55 Clustering; Classification
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/18152 Extracting features based on a plurality of salient regional features, e.g. "bag of words"
• CPC Class Codes: G06V 30/19147 Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning

Abstract

An image classification system trains an image classification model to classify images relative to text appearing with the images. Training images are iteratively selected and classified by the image classification model according to feature vectors of the training images. An independent model is trained for unique n-grams of text. The image classification system obtains text appearing with an image and parses the text into candidate labels for the image. The image classification system determines whether an image classification model has been trained for the candidate labels. When an image classification model corresponding to a candidate label has been trained, the image classification subsystem classifies the image relative to the candidate label. The image is labeled based on candidate labels for which the image is classified as a positive image.

Claims

1. A computer-implemented method, comprising:

obtaining text associated with an image;

obtaining a feature vector representing visual features of the image;

parsing, in one or more processing devices, the text into candidate labels for the image, each candidate label being a unique n-gram of text;

determining, in the one or more processing devices, that an image classification model has been trained for an n-gram matching one or more of the candidate labels, wherein the image classification model has been trained to classify images based on the feature vector; and

assigning, in the processing device, at least one of the one or more candidate labels as a label for the image based on the feature vector and the image classification model for the n-gram matching the one or more candidate labels.

2. The method of claim 1, further comprising:

computing a classification score for the image to the one or more candidate labels based on the feature vector and the image classification model for the n-gram matching the one or more candidate labels; and

classifying the image as relevant to the one or more candidate labels based on the classification score.

3. The method of claim 2, wherein classifying the image to the one or more candidate labels based on the classification score comprises:

determining that the image has a classification score to a particular candidate label that meets or exceeds a relevance threshold; and

labeling the image with a label corresponding to the particular candidate label.

4. The method of claim 3, wherein the determining that the image has a classification score to a particular candidate label and labeling the image is performed for each of at least two candidate labels.

5. The method of claim 1, wherein obtaining text associated with the image comprises obtaining text appearing with the image on a Web page.

6. A non-transitory computer readable medium encoded with a computer program comprising instructions that when executed cause one or more computers to perform operations comprising:

obtaining text associated with an image;

obtaining a feature vector representing visual features of the image;

parsing the text into candidate labels for the image, each candidate label being a unique n-gram of the text;

determining that an image classification model has been trained for an n-gram matching one or more of the candidate labels, wherein the image classification model has been trained to classify images based on the feature vector; and

assigning at least one of the one or more candidate labels as a label for the image based on the feature vector and the image classification model for the n-gram matching the one or more candidate labels.

7. The computer readable medium of claim 6, wherein the instructions cause the one or more computers to perform operations comprising:

computing a classification score for the image to the one or more candidate labels based on the feature vector and the image classification model for the n-gram matching the one or more candidate labels; and

classifying the image as relevant to the one or more candidate labels based on the classification score.

8. The computer readable medium of claim 7, wherein classifying the image to the one or more candidate labels based on the classification score comprises:

determining that the image has a classification score to a particular candidate label that meets or exceeds a relevance threshold; and

labeling the image with a label corresponding to the particular candidate label.

9. The computer readable medium of claim 8, wherein the determining that the image has a classification score to a particular candidate label and labeling the image is performed for each of at least two candidate labels.

10. The computer readable medium of claim 6, wherein obtaining text associated with the image comprises obtaining text appearing with the image on a Web page.

11. A system comprising:

a data store storing images; and

one or more computers that interact with the data store and execute instructions that cause the one more computers to perform operations comprising;

obtaining text associated with an image;

obtaining a feature vector representing visual features of the image;

parsing the text into candidate labels for the image, each candidate label being a unique n-gram of the text;

determining that an image classification model has been trained for an n-gram matching one or more of the candidate labels, wherein the image classification model has been trained to classify images based on the feature vector; and

assigning at least one of the one or more candidate labels as a label for the image based on the feature vector and the image classification model for the n-gram matching the one or more candidate labels.

12. The system of claim 11, wherein the instructions cause the one or more computers to perform operations comprising:

computing a classification score for the image to the one or more candidate labels based on the feature vector and the image classification model for the n-gram matching the one or more candidate labels; and

classifying the image as relevant to the one or more candidate labels based on the classification score.

13. The system of claim 12, wherein classifying the image to the one or more candidate labels based on the classification score comprises:

determining that the image has a classification score to a particular candidate label that meets or exceeds a relevance threshold; and

labeling the image with a label corresponding to the particular candidate label.

14. The system of claim 13, wherein the determining that the image has a classification score to a particular candidate label and labeling the image is performed for each of at least two candidate labels.

15. The system of claim 11, wherein obtaining text associated with the image comprises obtaining text appearing with the image on a Web page.