Image relevance model

• US 8,515,212 B1
• Filed: 07/17/2009
• Issued: 08/20/2013
• Est. Priority Date: 07/17/2009
• Status: Expired due to Fees

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Bengio, Samy, Murphy-Chutorian, Erik, Yee, Yangli Hector, Rosenberg, Charles
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): TORRES, JOSE
• Application Number:
  US12/505,287
• Time in Patent Office: 1,495 Days
• Field of Search: 382/305, 382/155, 382/159, 707/E17.004, 707/E17.019, 707/E17.02, 707/E17.023
• US Class Current: 382/305
• CPC Class Codes: G06F 16/56 having vectorial format
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/23213 with fixed number of clusters, e.g. K-means clustering
• CPC Class Codes: G06V 10/763 Non-hierarchical techniques, e.g. based on statistics of modelling distributions
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene

Abstract

Methods, systems, and apparatus, including computer program products, for identifying images relevant to a query are disclosed. An image search subsystem selects images to reference in image search results that are responsive to a query based on an image relevance model that is trained for the query. An independent image relevance model is trained for each unique query that is identified by the image search subsystem. The image relevance models can be applied to images to order image search results obtained for the query. Each relevance model is trained based on content feature values of images that are identified as being relevant to the query (e.g., frequently selected from the image search results) and images that are identified as being relevant to another unique query. The trained model is applied to the content feature values of all known images to generate an image relevance score that can be used to order search results for the query.

Claims

1. A computer-implemented method, comprising:

identifying a plurality of queries, each of the queries being a unique set of one or more query terms received by a search system as a query input;

for each of the queries;

selecting training images for training an image relevance model, the training images comprising;

a first image having a first relevance measure, for the query, that satisfies a first relevance threshold; and

a second image having a second relevance measure, for a different query in the plurality of queries, that satisfies a second relevance threshold;

for each of the training images, identifying content feature values, each content feature value representing a characteristic of an aspect of the training image; and

training the image relevance model to generate relevance measures of content feature values of images to the query based on the content feature values of the training images, wherein the image relevance model comprises a vector of weights corresponding to the content feature values, the training comprising;

initializing the vector of weights to default values;

generating a first training score based on the image relevance model and the content feature values of the first image;

generating a second training score based on the image relevance model and the content feature values of the second image;

comparing the first training score and the second training score;

in response to a difference between the first training score and the second training score not satisfying a training score margin, adjusting values of the vector of weights;

determining whether a training condition has occurred; and

repeating the selecting training images, the generating the first training score, the generating the second training score and the comparing when the training condition has not occurred.

2. The method of claim 1, wherein the first relevance measure is based on selections of the first image when provided in response to the query.

3. The method of claim 1, wherein the second relevance measure is based on selections of the second image when provided in response to the different query.

4. The method of claim 1, wherein generating a first training score comprises:

determining a dot product of the vector of weights and the content feature values of the first image.

5. The method of claim 1, wherein generating a second training score comprises:

determining a dot product of the vector of weights and the content feature values of the second image.

6. The method of claim 1, further comprising:

for each query;

identifying a plurality of images;

for each image;

applying the image relevance model to content feature values of the image; and

assigning an image relevance score to the image based on an application of the image relevance model to the content feature values, the image relevance score being a relevancy measure of the image to the query.

7. The method of claim 6, wherein applying the image relevance model to the content feature values comprises determining a dot product of the image relevance model and the content feature values.

8. The method of claim 6, further comprising:

receiving second query data defining a query; and

obtaining the image relevance scores of the images for the query to identify a third plurality of images that are relevant to the query based on the image relevance model and content feature values.

9. The method of claim 6, further comprising for each query, selecting images for reference in image search results responsive to the query based on the image relevance score.

10. The method of claim 9, wherein selecting images comprises selecting images for reference in image search results based on a similarity to previously selected images.

11. The method of claim 1, wherein identifying content feature values comprises identifying content feature values from each of a plurality of scaled versions of the training images.

12. The method of claim 1, wherein the first relevance threshold and the second relevance threshold are equal.

13. A system, comprising:

a first data store storing query data, the query data defining a plurality of queries;

one or more processors that interact with the first data store and perform operations comprising;

identifying a query defined by the query data;

selecting training images for training an image relevance model, the training images comprising;

a first image having a first relevance measure, for the query, that satisfies a first relevance threshold; and

a second image having a second relevance measure, for a different query in the plurality of queries, that satisfies a second relevance threshold;

for each of the training images, identifying content feature values, each content feature value representing a characteristic of an aspect of the training image; and

training the image relevance model to generate relevance measures of content feature values of images to the query based on the content feature values of the training images, wherein the image relevance model comprises a vector of weights corresponding to the content feature values, the training comprising;

initializing the vector of weights to default values;

generating a first training score based on the image relevance model and the content feature values of the first image;

generating a second training score based on the image relevance model and the content feature values of the second image;

comparing the first training score and the second training score;

in response to a difference between the first training score and the second training score not satisfying a training score margin, adjusting values of the vector of weights;

determining whether a training condition has occurred; and

repeating the selecting training images, the generating the first training score, the generating the second training score and the comparing when the training condition has not occurred.

14. The system of claim 13, wherein the first relevance measure is based on selections of the first image when provided in response to the query.

15. The system of claim 13, wherein the second relevance measure is based on selections of the second image when provided in response to the different query.

16. The system of claim 13, wherein the one or more processors perform operations comprising assigning an image relevance score to each of a plurality of images based on the image relevance model and the content feature values of the images, the image relevance score being a relevancy measure of the image to the query.

17. The system of claim 16, wherein the image relevance score for each image is based on a dot product of the image relevance model and the content feature values of the image.

18. The system of claim, 16 wherein the one or more processors perform operations comprising selecting images to reference in image search results based on the image relevance scores.

19. A non-transitory computer readable medium encoded with a computer program comprising instructions that when executed operate to cause a computer to perform operations:

identifying a plurality of queries, each of the queries being a unique set of one or more query terms received by a search system as a query input;

for each of the queries;

selecting training images for training an image relevance model, the training images comprising;

a first image having a first relevance measure, for the query, that satisfies a first relevance threshold; and

a second image having a second relevance measure, for one or more different queries in the plurality of queries, that satisfies a second relevance threshold;

for each of the training images, identifying content feature values, each content feature value representing a characteristic of an aspect of the training image; and

training the image relevance model to generate relevance measures of content feature values of images, for the query, based on the content feature values of the training images, wherein the image relevance model comprises a vector of weights corresponding to the content feature values, the training comprising;

initializing the vector of weights to default values;

generating a first training score based on the image relevance model and the content feature values of the first image;

generating a second training score based on the image relevance model and the content feature values of the second image;

comparing the first training score and the second training score;

in response to a difference between the first training score and the second training score not satisfying a training score margin, adjusting values of the vector of weights;

determining whether a training condition has occurred; and

repeating the selecting training images, the generating the first training score, the generating the second training score and the comparing when the training condition has not occurred.