Refining image relevance models

• US 8,891,858 B1
• Filed: 07/10/2012
• Issued: 11/18/2014
• Est. Priority Date: 09/30/2011
• Status: Expired due to Fees

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Preetham, Arcot J., Duerig, Thomas J., Rosenberg, Charles J., Yee, Yangli Hector, Bengio, Samy
• Primary Examiner(s): Mehta, Bhavesh
• Assistant Examiner(s): Dunphy, David F
• Application Number:
  US13/545,222
• Time in Patent Office: 861 Days
• Field of Search: None
• US Class Current: 382/159
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/29 Graphical models, e.g. Bayesian networks
• CPC Class Codes: G06V 10/42 Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
• CPC Class Codes: G06V 10/751 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
• CPC Class Codes: G06V 30/19147 Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning

Abstract

Methods, systems and apparatus for refining image relevance models. In general, one aspect of the subject matter described in this specification can be implemented in methods that include re-training an image relevance model by generating a first re-trained model based on content feature values of first images of a first portion of training images in a set of training images, receiving, from the first re-trained model, image relevance scores for second images of a second portion of the set of training images, removing, from the set of training images, some of the second images identified as outlier images for which the image relevance score received from the first re-trained model is below a threshold score, and generating a second re-trained model based on content feature values of the first images of the first portion and the second images of the second portion that remain following removal of the outlier images.

Claims

1. A method comprising:

receiving a trained image relevance model that generates relevance measures of images to a query, wherein the trained image relevance model has been trained based on content feature values of a set of training images, the query being a unique set of one or more query terms received by a search system as a query input; and

re-training the image relevance model, the re-training comprising;

generating a first re-trained image relevance model based on content feature values of first images of a first portion of training images in the set of training images;

receiving, from the first re-trained image relevance model, image relevance scores for second images of a second portion of the set of training images;

removing, from the set of training images, at least some of the second images of the second portion of the set training images identified as outlier images, the outlier images being training images for which the image relevance score received from the first re-trained image relevance model is below a threshold score;

generating an aggregation of near duplicate images among the set of training images;

associating image selection data of the aggregated near duplicate images with the aggregation of the near duplicate images; and

generating a second re-trained image relevance model based on content feature values of the first images of the first portion, the second images of the second portion that remain following removal of the at least some of the second images of the second portion of the set of training images, and the image selection data.

2. The method of claim 1, wherein receiving image relevance scores comprises receiving image relevance scores for the second images of the second portion of the set of training images that were not included in the first portion of the set of training images.

3. The method of claim 1, further comprising generating multiple re-trained image relevance models based on content feature values of multiple distinct portions of the set of training images.

4. The method of claim 3, further comprising generating the multiple distinct portions of the set of training images, wherein each of the multiple distinct portions is generated to include less than 100 percent of the images in the set of training images.

5. The method of claim 3, wherein each of the multiple distinct portions includes at least one image that is not included in another distinct portion.

6. The method of claim 1, wherein the aggregation of near duplicate images comprises fewer than a threshold number of near duplicate images.

7. A system, comprising:

a data processing apparatus; and

a memory coupled to the data processing apparatus having instructions stored thereon which, when executed by the data processing apparatus cause the data processing apparatus to perform operations comprising;

receiving a trained image relevance model that generates relevance measures of images to a query, wherein the trained image relevance model has been trained based on content feature values of a set of training images, the query being a unique set of one or more query terms received by a search system as a query input; and

re-training the image relevance model, the re-training comprising;

generating a first re-trained image relevance model based on content feature values of first images of a first portion of training images in the set of training images;

receiving, from the first re-trained image relevance model, image relevance scores for second images of a second portion of the set of training images;

removing, from the set of training images, at least some of the second images of the second portion of the set training images identified as outlier images, the outlier images being training images for which the image relevance score received from the first re-trained image relevance model is below a threshold score;

generating an aggregation of near duplicate images among the set of training images;

associating image selection data of the aggregated near duplicate images with the aggregation of the near duplicate images; and

generating a second re-trained image relevance model based on content feature values of the first images of the first portion and the second images of the second portion that remain following removal of the at least some of the second images of the second portion of the set of training images, and the image selection data.

8. The system of claim 7, wherein receiving image relevance scores comprises receiving image relevance scores for the second images of the second portion of the set of training images that were not included in the first portion of the set of training images.

9. The system of claim 7, further comprising generating multiple re-trained image relevance models based on content feature values of multiple distinct portions of the set of training images.

10. The system of claim 9, further comprising generating the multiple distinct portions of the set of training images, wherein each of the multiple distinct portions is generated to include less than 100 percent of the images in the set of training images.

11. The system of claim 9, wherein each of the multiple distinct portions includes at least one image that is not included in another distinct portion.

12. The system of claim 7, wherein the aggregation of near duplicate images comprises fewer than a threshold number of near duplicate images.

13. A non-transitory computer readable media storing software comprising instructions executable by a processing device and upon such execution cause the processing device to perform operations comprising:

receiving a trained image relevance model that generates relevance measures of images to a query, wherein the trained image relevance model has been trained based on content feature values of a set of training images, the query being a unique set of one or more query terms received by a search system as a query input; and

re-training the image relevance model, the re-training comprising;

generating a first re-trained image relevance model based on content feature values of first images of a first portion of training images in the set of training images;

receiving, from the first re-trained image relevance model, image relevance scores for second images of a second portion of the set of training images;

removing, from the set of training images, at least some of the second images of the second portion of the set training images identified as outlier images, the outlier images being training images for which the image relevance score received from the first re-trained image relevance model is below a threshold score;

generating an aggregation of near duplicate images among the set of training images;

associating image selection data of the aggregated near duplicate images with the aggregation of the near duplicate images; and

generating a second re-trained image relevance model based on content feature values of the first images of the first portion and the second images of the second portion that remain following removal of the at least some of the second images of the second portion of the set of training images, and the image selection data.

14. The computer readable media of claim 13, wherein receiving image relevance scores comprises

receiving image relevance scores for the second images of the second portion of the set of training images that were not included in the first portion of the set of training images.

15. The computer readable media of claim 13, further comprising instructions to generate multiple re-trained image relevance models based on content feature values of multiple distinct portions of the set of training images.

16. The computer readable media of claim 15, further comprising instructions to generate the multiple distinct portions of the set of training images, wherein each of the multiple distinct portions is generated to include less than 100 percent of the images in the set of training images.

17. The computer readable media of claim 15, wherein each of the multiple distinct portions includes at least one image that is not included in another distinct portion.

18. The computer readable media of claim 13, wherein the aggregation of near duplicate images comprises fewer than a threshold number of near duplicate images.