Ordering image search results

• US 8,209,330 B1
• Filed: 05/29/2009
• Issued: 06/26/2012
• Est. Priority Date: 05/29/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Covell, Michele, Jing, Yushi, Wang, Jingbin, Rosenberg, Charles, Moussa, Sarah
• Primary Examiner(s): Liang, Vei-Chung
• Application Number:
  US12/475,006
• Time in Patent Office: 1,124 Days
• Field of Search: None
• US Class Current: 707/728
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Methods, systems, and apparatus, including computer program products, for ranking images are disclosed. An image search subsystem generates an adjustment factor representative of a quality measure of an image relative to a search query. The quality represents a relevance of the image to the query. The adjustment factor can be computed based on relevance data for the image to the query and image similarity data representing a relative similarity between the image and other images relevant to the query. The relevance data can be based on user actions in response to the image being included in search results for the query. The adjustment factor can be scaled based on whether the relevance data and the image similarity data both indicate that the image is relevant to the search query. A relevance score is computed based on the adjustment factor (e.g., a product of the adjustment factor and relevance score).

Claims

1. A computer-implemented method, comprising:

receiving a plurality of image search results that are responsive to a search query, each image search result referencing an image that has been identified as responsive to the search query based on a relevance score for the image search result, the relevance score being determined independent of a visual similarity of the image to other images referenced by the image search results;

for a particular image search result;

receiving, in a data processing apparatus, image similarity data for the particular image search result, the image similarity data being indicative of a visual similarity between the image referenced by the particular image search result and other images that are referenced by the plurality of image search results;

generating, in a data processing apparatus, an adjustment factor based on the relevance score for the particular image search result and the image similarity data for the particular image search result, the adjustment factor specifying a quality measure for the particular image search result relative to the search query;

determining that the particular image search result is a co-relevant result, the co-relevant result being a responsive image search result for which;

the relevance score meets a specified relevance score threshold; and

the image similarity data indicates that the image referenced by the responsive image search result has a least a threshold level of visual similarity to a threshold number of the images referenced by the image search results; and

scaling the adjustment factor for the particular image search result by an amplification factor in response to determining that the particular image search result is a co-relevant result.

2. The method of claim 1, further comprising:

receiving a request for image search results responsive to the search query;

for each image identified for inclusion in the image search results, generating image score data representing an image rank score based on the adjustment factor; and

ordering the image search results based on the image score data.

3. The method of claim 1, wherein the relevance score threshold is an upper relevance score threshold, and the threshold level of visual similarity is an upper similarity threshold.

4. The method of claim 1, wherein scaling the adjustment factor comprises determining a product of an amplification factor and the adjustment factor.

5. The method of claim 4, wherein scaling the adjustment factor comprises determining a product of an inverse-amplification factor and the adjustment factor.

6. The method of claim 1, further comprising scaling the adjustment factor by an inverse-amplification factor when the particular image search result is a co-relevant result.

7. The method of claim 6, wherein the relevance score threshold is a lower relevance score threshold and the threshold level of visual similarity is a lower similarity threshold.

8. The method of claim 1, wherein the relevance scores for image search results are retrieved from a data store that stores relevance data for a plurality of images.

9. The method of claim 1, wherein the relevance score is based on click data stored in a click log.

10. The method of claim 1, wherein the relevance score is based on user feedback specifying a relevance of the image to the search query.

11. The method of claim 1, wherein accessing image similarity data comprises accessing a visual distance matrix that specifies similarities between the image and the other images in the plurality of images, the similarities being specified by visual distance measures between image features of the image and image features of the other images in the plurality of images.

12. The method of claim 1, wherein generating an adjustment factor comprises determining a product of the relevance score and the image similarity data.

13. A system, comprising:

a first data storage device storing relevance data for an image search result relative to a search query, the image search result referencing an image that has been identified as responsive to the search query based on a relevance score for the image search result, the relevance score being determined independent of a visual similarity of the image to other images that are referenced by other image search results that are identified as responsive to the search query;

a second data storage device storing image similarity data that is indicative of a relative visual similarity between the image and the other images; and

an image search subsystem comprising one or more processors coupled to each of the first data storage device and the second data storage device to generate an adjustment factor for the image search result relative to the search query based on the relevance data and image similarity data, the adjustment factor representing a quality measure for the image relative to the search query, the image search subsystem being further operable to;

determine that the image search result is a co-relevant result, the co-relevant result being a responsive image search result for which;

the relevance score meets a specified relevance score threshold; and

the image similarity data indicates that the image referenced by the responsive image search result has a least a threshold level of visual similarity to a threshold number of the other images referenced by the image search results; and

scaling the adjustment factor for the image search result by an amplification factor in response to determining that the image search result is a co-relevant result.

14. The system of claim 13, wherein the relevance score threshold is an upper relevance score threshold and the threshold level of visual similarity is an upper similarity threshold.

15. The system of claim 13, wherein the image search subsystem is operable to scale the adjustment factor by an inverse-amplification factor when the image search result is a co-relevant result.

16. The system of claim 13, wherein the relevance data comprises user feedback specifying a relevance of the image to the search query.

17. The system of claim 13, wherein the relevance score threshold is a lower relevance score threshold and the threshold level of visual similarity is a lower similarity threshold.

18. The system of claim 13, wherein the image search subsystem is operable to organize the image similarity data in a visual distance matrix that specifies similarities between the image and the other images, the similarities being specified by visual distance measures between image features of the image and image features of the other images.

19. A non-transitory computer readable medium encoded with a computer program comprising instructions that when executed operate to cause a computer to perform operations:

receiving a plurality of image search results that are responsive to a search query, each image search result referencing an image that has been identified as responsive to the search query based on a relevance score for the image search result, the relevance score being determined independent of a visual similarity of the image to other images referenced by the image search results;

for a particular image search result;

receiving image similarity data for the particular image search result, the image similarity data being indicative of a visual similarity between the image referenced by the particular image search result and other images that are referenced by the plurality of image search results;

generating an adjustment factor based on the relevance score for the particular image search result and the image similarity data for the particular image search result, the adjustment factor specifying a quality measure for the particular image search result relative to the search query;

determining that the particular image search result is a co-relevant result, the co-relevant result being a responsive image search result for which;

the relevance score meets a specified relevance score threshold; and

the image similarity data indicates that the image referenced by the responsive image search result has a least a threshold level of visual similarity to a threshold number of the images referenced by the image search results; and

scaling the adjustment factor for the particular image search result by an amplification factor in response to determining that the particular image search result is a co-relevant result.

20. The computer readable medium of claim 19, further comprising instructions that when executed operate to cause a computer to perform operations:

receiving a request for image search results responsive to the search query;

for each image identified for inclusion in the image search results, generating image score data representing an image rank score based on the adjustment factor; and

ordering the image search results based on the image score data.