Image relevance model

• US 9,176,988 B2
• Filed: 08/14/2013
• Issued: 11/03/2015
• Est. Priority Date: 07/17/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Bengio, Samy, Murphy-Chutorian, Erik, Yee, Yangli Hector, Rosenberg, Charles J.
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): TORRES, JOSE
• Application Number:
  US13/966,737
• Publication Number:
  US 20150161172A1
• Time in Patent Office: 811 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/56 having vectorial format
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/23213 with fixed number of clusters, e.g. K-means clustering
• CPC Class Codes: G06V 10/763 Non-hierarchical techniques, e.g. based on statistics of modelling distributions
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene

Abstract

Methods, systems, and apparatus, including computer program products, for identifying images relevant to a query are disclosed. An image search subsystem selects images to reference in image search results that are responsive to a query based on an image relevance model that is trained for the query. An independent image relevance model is trained for each unique query that is identified by the image search subsystem. The image relevance models can be applied to images to order image search results obtained for the query. Each relevance model is trained based on content feature values of images that are identified as being relevant to the query (e.g., frequently selected from the image search results) and images that are identified as being relevant to another unique query. The trained model is applied to the content feature values of all known images to generate an image relevance score that can be used to order search results for the query.

Claims

1. A computer-implemented method, comprising:

identifying, as positive images, a first set of images that are considered relevant to a first query based, at least in part, on a first selection rate for images in the first set when presented as search results for the first query;

identifying, as negative images, a second set of images that differ from the first set of images and that are considered relevant to a different query based, at least in part, on a second selection rate for images in the second set when presented as search results for the different query;

training an image relevance model for the first query based on feature values of the positive images and feature values of the negative images;

generating a score for each of a plurality of images based on feature values of the plurality of images and the image relevance model;

receiving a search query;

identifying, from the plurality of images, one or more highest scoring images based on the score for each of one or more of the plurality of images; and

providing at least a portion of the one or more highest scoring images as results for the search query.

2. The method of claim 1, wherein generating a score for each of a plurality of images comprises:

identifying, from a corpus of images, an image that has not been scored using the trained image relevance model for the first query;

assigning the image a relevance score specifying a measure of relevance of the image as a search result for the first query;

storing the relevance score in association with a reference to the image; and

iteratively repeating the identifying, assigning, and storing for one or more additional images that have not been scored using the trained image relevance model for the first query.

3. The method of claim 2, wherein identifying one or more highest scoring images comprises:

in response to receiving the search query, identifying the stored relevance scores for images from the corpus of images; and

identifying, as search results for the search query, one or more of the images from the corpus of images having highest stored relevance scores.

4. The method of claim 1, comprising:

determining, based on image feature values of the highest scoring images, that a first image has at least a threshold level of similarity relative to a second image having a higher score than the first image; and

assigning the first image to a presentation position in the search results that is lower than a presentation position of at least a third image having a lower score than the first image.

5. The method of claim 1, comprising generating multiple different image relevance models for a plurality of different queries, the generating of the multiple different image relevance models comprising:

identifying a particular query that was submitted by a user;

determining that the particular query differs from each query for which an image relevance model is available; and

in response to the determination that the particular query differs from each query for which an image relevance model is available;

identifying positive images that have at least a first minimum specified user interaction rate when presented in search results for the particular query;

identifying negative images that have at least a second minimum specified user interaction rate when presented in search results for a query that differs from the particular query; and

generating the image relevance model for the particular query based on the positive images and the negative images.

6. The method of claim 5, wherein generating the image relevance model comprises iteratively adjusting the image relevance model for the particular query based on image feature values of the positive images and the negative images until a training condition has been met.

7. The method of claim 6, comprising determining whether the training condition has been met based on a difference between a positive training score associated with one or more of the positive images and a negative training score associated with one or more of the negative images.

8. A system, comprising:

a data store storing images;

one or more processors that interact with the data store and execute instructions that cause the one or more processors to perform operations comprising;

identifying, from the data store and as positive images, a first set of images that are considered relevant to a first query based, at least in part, on a first selection rate for images in the first set when presented as search results for the first query;

identifying, from the data store and as negative images, a second set of images that differ from the first set of images and that are considered relevant to a different query based, at least in part, on a second selection rate for images in the second set when presented as search results for the different query;

training an image relevance model for the first query based on feature values of the positive images and feature values of the negative images;

generating a score for each of a plurality of images based on feature values of the plurality of images and the image relevance model;

receiving a search query;

identifying, from the plurality of images, one or more highest scoring images based on the score for each of one or more of the plurality of images; and

providing at least a portion of the one or more highest scoring images as results for the search query.

9. The system of claim 8, wherein generating a score for each of a plurality of images comprises:

identifying, from a corpus of images, an image that has not been scored using the trained image relevance model for the first query;

assigning the image a relevance score specifying a measure of relevance of the image as a search result for the first query;

storing the relevance score in association with a reference to the image; and

iteratively repeating the identifying, assigning, and storing for one or more additional images that have not been scored using the trained image relevance model for the first query.

10. The system of claim 9, wherein identifying one or more highest scoring images comprises:

in response to receiving the search query, identifying the stored relevance scores for images from the corpus of images; and

identifying, as search results for the search query, one or more of the images from the corpus of images having highest stored relevance scores.

11. The system of claim 8, wherein the instructions cause the one or more processors to perform operations comprising:

determining, based on image feature values of the highest scoring images, that a first image has at least a threshold level of similarity relative to a second image having a higher score than the first image; and

assigning the first image to a presentation position in the search results that is lower than a presentation position of at least a third image having a lower score than the first image.

12. The system of claim 8, wherein the instructions cause the one or more processors to perform operations comprising generating multiple different image relevance models for a plurality of different queries, the generating of the multiple different image relevance models comprising:

identifying a particular query that was submitted by a user;

determining that the particular query differs from each query for which an image relevance model is available; and

in response to the determination that the particular query differs from each query for which an image relevance model is available;

identifying positive images that have at least a first minimum specified user interaction rate when presented in search results for the particular query;

identifying negative images that have at least a second minimum specified user interaction rate when presented in search results for a query that differs from the particular query; and

generating the image relevance model for the particular query based on the positive images and the negative images.

13. The system of claim 12, wherein generating the image relevance model comprises iteratively adjusting the image relevance model for the particular query based on image feature values of the positive images and the negative images until a training condition has been met.

14. The system of claim 13, wherein the instructions cause the one or more processors to perform operations comprising determining whether the training condition has been met based on a difference between a positive training score associated with one or more of the positive images and a negative training score associated with one or more of the negative images.

15. A non-transitory computer readable medium storing instructions that when executed by one or more data processing apparatus cause the one or more data processing apparatus to perform operations comprising:

identifying, as positive images, a first set of images that are considered relevant to a first query based, at least in part, on a first selection rate for images in the first set when presented as search results for the first query;

identifying, as negative images, a second set of images that differ from the first set of images and that are considered relevant to a different query based, at least in part, on a second selection rate for images in the second set when presented as search results for the different query;

training an image relevance model for the first query based on feature values of the positive images and feature values of the negative images;

generating a score for each of a plurality of images based on feature values of the plurality of images and the image relevance model;

receiving a search query;

identifying, from the plurality of images, one or more highest scoring images based on the score for each of one or more of the plurality of images; and

providing at least a portion of the one or more highest scoring images as results for the search query.

16. The computer readable medium of claim 15, wherein generating a score for each of a plurality of images comprises:

identifying, from a corpus of images, an image that has not been scored using the trained image relevance model for the first query;

assigning the image a relevance score specifying a measure of relevance of the image as a search result for the first query;

storing the relevance score in association with a reference to the image; and

iteratively repeating the identifying, assigning, and storing for one or more additional images that have not been scored using the trained image relevance model for the first query.

17. The computer readable medium of claim 16, wherein identifying one or more highest scoring images comprises:

in response to receiving the search query, identifying the stored relevance scores for images from the corpus of images; and

identifying, as search results for the search query, one or more of the images from the corpus of images having highest stored relevance scores.

18. The computer readable medium of claim 15, wherein the instructions cause the one or more data processing apparatus to perform operations comprising:

determining, based on image feature values of the highest scoring images, that a first image has at least a threshold level of similarity relative to a second image having a higher score than the first image; and

assigning the first image to a presentation position in the search results that is lower than a presentation position of at least a third image having a lower score than the first image.

19. The computer readable medium of claim 15, wherein the instructions cause the one or more data processing apparatus to perform operations comprising generating multiple different image relevance models for a plurality of different queries, the generating of the multiple different image relevance models comprising:

identifying a particular query that was submitted by a user;

determining that the particular query differs from each query for which an image relevance model is available; and

in response to the determination that the particular query differs from each query for which an image relevance model is available;

identifying positive images that have at least a first minimum specified user interaction rate when presented in search results for the particular query;

identifying negative images that have at least a second minimum specified user interaction rate when presented in search results for a query that differs from the particular query; and

generating the image relevance model for the particular query based on the positive images and the negative images.

20. The computer readable medium of claim 19, wherein generating the image relevance model comprises iteratively adjusting the image relevance model for the particular query based on image feature values of the positive images and the negative images until a training condition has been met.