Image classification

• US 9,183,226 B2
• Filed: 07/21/2014
• Issued: 11/10/2015
• Est. Priority Date: 07/17/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Yee, Yangli Hector, Bengio, Samy, Rosenberg, Charles J., Murphy-Chutorian, Erik
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): TORRES, JOSE
• Application Number:
  US14/336,692
• Publication Number:
  US 20150161170A1
• Time in Patent Office: 477 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/55 Clustering; Classification
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/18152 Extracting features based on a plurality of salient regional features, e.g. "bag of words"
• CPC Class Codes: G06V 30/19147 Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning

Abstract

An image classification system trains an image classification model to classify images relative to text appearing with the images. Training images are iteratively selected and classified by the image classification model according to feature vectors of the training images. An independent model is trained for unique n-grams of text. The image classification system obtains text appearing with an image and parses the text into candidate labels for the image. The image classification system determines whether an image classification model has been trained for the candidate labels. When an image classification model corresponding to a candidate label has been trained, the image classification subsystem classifies the image relative to the candidate label. The image is labeled based on candidate labels for which the image is classified as a positive image.

Claims

1. A computer-implemented method, comprising:

obtaining a user query;

identifying, by one or more computers, images that have been assigned, by one or more image classification models for one or more corresponding n-grams that each match the user query, the n-gram as a text label of the image, wherein the image classification models assign n-grams as labels to images based on the images being associated with text matching the corresponding n-gram of the image classification model and a feature vector of the image;

applying, by one or more computers, a boost value to a relevance score of each of at least one of the images to obtain an adjusted relevance score, wherein the boost value applied is based, at least in part, on a strength of the match between the user query and the text label of the image;

ranking, by one or more computers, the images based, at least in part, on the adjusted relevance score; and

outputting, by one or more computers, at least a portion of the images for presentation on a search results page responsive to the user query.

2. The method of claim 1, further comprising assigning a magnitude to the boost value based on a strength of the match between the user query and the text label, wherein the magnitude of the boost value is proportional to the strength of the match.

3. The method of claim 2, wherein assigning a magnitude to the boost value comprises assigning a highest magnitude to the boost value when the user query exactly matches the text label.

4. The method of claim 1, further comprising training the classification model.

5. The method of claim 4, wherein training the classification model comprises training the classification model based on feature vectors of positive training images having relevance measures, for the n-grams, that satisfy a relevance threshold and negative training images having relevance measures, for the n-grams, that do not satisfy the relevance threshold.

6. The method of claim 5, wherein training the image classification model comprises:

for each of one or more training images from the positive training images or the negative training images;

computing a classification score for the training image based on an initial image classification model;

classifying the training image based on the classification score;

determining that the training image is incorrectly classified; and

computing a minimum kernel approximation based on the feature vector for the training image to adjust the initial image classification model.

7. The method of claim 5, further comprising assigning an image a label based on a feature vector for the image and the image classification model for an n-gram matching the label.

8. A system comprising:

a data store storing images; and

one or more computers that interact with the data store and execute instructions that cause the one more computers to perform operations comprising;

obtaining a user query;

identifying images that have been assigned, by one or more image classification models for one or more corresponding n-grams that each match the user query, the n-gram as a text label of the image, wherein the image classification models assign n-grams as labels to images based on the images being associated with text matching the corresponding n-gram of the image classification model and a feature vector of the image;

applying a boost value to a relevance score of each of at least one of the images to obtain an adjusted relevance score, wherein the boost value applied is based, at least in part, on a strength of the match between the user query and the text label of the image;

ranking the images based, at least in part, on the adjusted relevance score; and

outputting at least a portion of the images for presentation on a search results page responsive to the user query.

9. The system of claim 8, wherein the instructions cause the one more computers to perform operations further comprising assigning a magnitude to the boost value based on a strength of the match between the user query and the text label, wherein the magnitude of the boost value is proportional to the strength of the match.

10. The system of claim 9, wherein assigning a magnitude to the boost value comprises assigning a highest magnitude to the boost value when the user query exactly matches the text label.

11. The system of claim 8, wherein the instructions cause the one more computers to perform operations further comprising training the classification model.

12. The method of claim 11, wherein training the classification model comprises training the classification model based on feature vectors of positive training images having relevance measures, for the n-grams, that satisfy a relevance threshold and negative training images having relevance measures, for the n-grams, that do not satisfy the relevance threshold.

13. The system of claim 12, wherein training the image classification model comprises:

for each of one or more training images from the positive training images or the negative training images;

computing a classification score for the training image based on an initial image classification model;

classifying the training image based on the classification score;

determining that the training image is incorrectly classified; and

computing a minimum kernel approximation based on the feature vector for the training image to adjust the initial image classification model.

14. The system of claim 12, wherein the instructions cause the one more computers to perform operations further comprising assigning an image a label based on a feature vector for the image and the image classification model for an n-gram matching the label.

15. A non-transitory computer readable medium encoded with a computer program comprising instructions that when executed cause one or more computers to perform operations comprising:

obtaining a user query;

identifying images that have been assigned, by one or more image classification models for one or more corresponding n-grams that each match the user query, the n-gram as a text label of the image, wherein the image classification models assign n-grams as labels to images based on the images being associated with text matching the corresponding n-gram of the image classification model and a feature vector of the image;

applying a boost value to a relevance score of each of at least one of the images to obtain an adjusted relevance score, wherein the boost value applied is based, at least in part, on a strength of the match between the user query and the text label of the image;

ranking the images based, at least in part, on the adjusted relevance score; and

outputting at least a portion of the images for presentation on a search results page responsive to the user query.

16. The computer readable medium of claim 15, wherein the instructions cause the one more computers to perform operations further comprising assigning a magnitude to the boost value based on a strength of the match between the user query and the text label, wherein the magnitude of the boost value is proportional to the strength of the match.

17. The computer readable medium of claim 16, wherein assigning a magnitude to the boost value comprises assigning a highest magnitude to the boost value when the user query exactly matches the text label.

18. The computer readable medium of claim 15, wherein the instructions cause the one more computers to perform operations further comprising training the classification model.

19. The computer readable medium of claim 18, wherein training the classification model comprises training the classification model based on feature vectors of positive training images having relevance measures, for the n-grams, that satisfy a relevance threshold and negative training images having relevance measures, for the n-grams, that do not satisfy the relevance threshold.

20. The computer readable medium of claim 19, wherein training the image classification model comprises:

for each of one or more training images from the positive training images or the negative training images;

computing a classification score for the training image based on an initial image classification model;

classifying the training image based on the classification score;

determining that the training image is incorrectly classified; and

computing a minimum kernel approximation based on the feature vector for the training image to adjust the initial image classification model.