Supervised re-ranking for visual search

• US 8,543,521 B2
• Filed: 03/30/2011
• Issued: 09/24/2013
• Est. Priority Date: 03/30/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Yang, Linjun, Hua, Xian-Sheng
• Primary Examiner(s): Holmes, Michael B
• Application Number:
  US13/076,350
• Publication Number:
  US 20120254076A1
• Time in Patent Office: 909 Days
• Field of Search: None
• US Class Current: 706/12
• CPC Class Codes: G06F 16/54 Browsing; Visualisation therefor
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information

Abstract

Supervised re-ranking for visual search may include re-ordering images that are returned in response to a text-based image search by exploiting visual information included in the images. In one example, supervised re-ranking for visual search may include receiving a textual query, obtaining an initial ranking result including a plurality of images corresponding to the textual query, and representing the textual query by a visual context of the plurality of images. A query-independent re-ranking model may be trained based on visual re-ranking features of the plurality of images of the textual query in accordance with a supervised training algorithm.

Claims

1. A computer-implemented method comprising:

under control of one or more processors configured with executable instructions;

receiving a textual query;

obtaining an initial ranking result corresponding to the textual query, the initial ranking result including a plurality of images corresponding to the textual query;

representing the textual query using a visual context of the plurality of images, the visual context facilitating determining the visual re-ranking features including prior re-ranking features, contextual re-ranking features and pseudo relevance feedback features;

training, using a supervised training algorithm, a query-independent re-ranking model including a weighted linear combination of features based at least upon the visual re-ranking features of the plurality of images of the textual query to obtain a plurality of weights associated with the features of the weighted linear combination.

2. The computer-implemented method as recited in claim 1, wherein representing the textual query further comprises representing the textual query using features that are independent of any text.

3. The computer-implemented method as recited in claim 1, wherein the prior re-ranking features comprise initial rankings of the plurality of images adjusted by a factor for correcting non-linearity of confidence relevance degradation of the plurality of images in the initial ranking result.

4. The computer-implemented method as recited in claim 1, wherein the contextual re-ranking features comprise information of one or more neighbors and one or more reciprocal neighbors of the plurality of images.

5. The computer-implemented method as recited in claim 1, wherein the pseudo relevance feedback features are computed based at least upon a predetermined number of top images of the plurality of images in the initial ranking result.

6. The computer-implemented method as recited in claim 1, further comprising:

receiving a textual query from a client;

obtaining a first ranking result in response to receiving the textual query from the client, the first ranking result including multiple images;

computing multiple scores corresponding to the multiple images based at least upon the query-independent model re-ranking and the first ranking result;

re-ranking the first ranking result based at least upon the multiple scores.

7. The computer-implemented method as recited in claim 6, further comprising presenting the re-ranked ranking result to the client to be displayed on a device of the client.

8. A computer-implemented method comprising:

under control of one or more processors configured with executable instructions;

receiving a textual query from a client;

obtaining an initial ranking result related to the textual query, the initial ranking result including a plurality of images;

representing the textual query in terms of a visual context of at least a subset of the plurality of images;

re-ranking the initial ranking result based at least upon visual re-ranking features of the at least subset of the plurality of images and a query-independent re-ranking model, the visual re-ranking features being determined from the visual context, and the query-independent re-ranking model being trained according to a supervised algorithm using visual re-ranking features of a plurality of labeled images corresponding to one or more training textual queries.

9. The computer-implemented method as recited in claim 8, wherein the visual re-ranking features of the at least subset of the plurality of images comprise prior re-ranking features, contextual re-ranking features and/or pseudo relevance feedback features of the at least subset of the plurality of images.

10. The computer-implemented method as recited in claim 8, wherein the query-independent re-ranking model includes a weighted linear combination of re-ranking features of the plurality of labeled images, the re-ranking features being independent of any textual features and/or features depending on textual information associated with the plurality of labeled images.

11. The computer-implemented method as recited in claim 8, wherein the query-independent re-ranking model includes a weighted linear combination of re-ranking features of the plurality of labeled images, the re-ranking features including prior re-ranking features, contextual re-ranking features and/or pseudo relevance feedback features of the plurality of labeled images, and wherein the method further comprises computing scores for the at least subset of the plurality of images based upon the weighted linear combination of the query-independent re-ranking model.

12. The computer-implemented method as recited in claim 11, wherein re-ranking the initial ranking result is further based upon the scores for the at least subset of the plurality of images.

13. The computer-implemented method as recited in claim 8, wherein re-ranking the initial ranking result is further based on textual re-ranking features, the textual re-ranking features including textual information in the textual query, textual information associated with the at least subset of the plurality of images and/or textual features extracted from the at least subset of the plurality of images.

14. The computer-implemented method as recited in claim 8, wherein the visual re-ranking features comprises prior re-ranking features including initial rankings of the at least subset of the plurality of images.

15. The computer-implemented method as recited in claim 14, wherein the prior re-ranking features further comprises a factor to correct non-linearity of relevance confidence degradation of the at least subset of the plurality of images in the initial rankings.

16. The computer-implemented method as recited in claim 8, wherein the visual re-ranking features comprises contextual re-ranking features including respective information of a neighborhood structure of each image of the at least subset of the plurality of images.

17. The computer-implemented method as recited in claim 16, wherein respective information of the neighborhood structure of each image of the at least subset of the plurality of images comprises information of one or more neighbors and one or more reciprocal neighbors of each image.

18. The computer-implemented method as recited in claim 8, wherein the visual re-ranking features comprise pseudo relevance feedback re-ranking features computed based at least upon a predetermined number of top images in the initial ranking result.

19. One or more computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform acts comprising:

under control of one or more processors configured with executable instructions;

obtaining an initial ranking result corresponding to a textual query, the initial ranking result including a plurality of images corresponding to the textual query; and

re-ranking the initial ranking result based at least upon initial rankings of the plurality of images in the initial ranking result, a neighborhood structure of the plurality of images and pseudo relevance feedback of a predetermined number of top-ranked images in the initial ranking result.

20. The one or more computer-readable media as recited in claim 19, wherein re-ranking the initial ranking result is further based upon a query-independent re-ranking model that is trained using a plurality of textual queries and a plurality of initial ranking results corresponding to the plurality of textual queries according to a supervised training algorithm.