Visual search reranking

• US 8,489,589 B2
• Filed: 02/05/2010
• Issued: 07/16/2013
• Est. Priority Date: 02/05/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Mei, Tao, Hua, Xian-Sheng, Li, Shipeng, Liu, Yuan
• Primary Examiner(s): BURKE, JEFF A
• Application Number:
  US12/701,283
• Publication Number:
  US 20110196859A1
• Time in Patent Office: 1,257 Days
• Field of Search: 707/723, 707/728
• US Class Current: 707/728
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/248 Presentation of query results

Abstract

An initial ranked list of a first plurality of visual documents is obtained from a first source in response to a query, and a second plurality of visual documents relevant to the query is gathered from a plurality of second sources. Visual patterns identified from the second plurality of visual documents are compared with the first visual documents for reranking the first visual documents.

Claims

1. A method implemented on a computing device, the method comprising:

receiving a query for visual content;

obtaining an initial ranked list of results in response to the query, the initial ranked list corresponding to a first plurality of images;

obtaining additional results in response to the query from one or more external sources, the additional results including a second plurality of images;

extracting, by a processor of the computing device, features from the second plurality of images obtained from the one or more external sources to generate a plurality of visual words, wherein each visual word includes a plurality of similar feature patches extracted from the second plurality of images;

determining recurring visual patterns based on the visual words and connections between the visual words;

determining a visual word representation of each of the first plurality of images corresponding to the initial ranked list;

reordering, by the processor, the initial ranked list based at least in part on a comparison of the determined recurring visual patterns with the visual word representations of the first plurality of images to generate a reranked list.

2. The method according to claim 1, wherein the determining recurring visual patterns based on the visual words and connections between the visual words further comprises determining a salient pattern based on a number of extracted patches contained in each visual word.

3. The method according to claim 2, wherein the determining visual patterns based on the visual words and connections between the visual words further comprises

determining a concurrent pattern based on a number of connections between the visual words,

wherein there is a connection between first and second visual words when the first visual word contains a first patch and the second visual word contains a second patch, and the first patch and the second patch were extracted from a same image of the plurality of second images.

4. The method according to claim 3, wherein the reordering the initial ranked list based at least in part on the determined visual patterns to generate a reranked list further comprises:

applying the determined visual patterns to the visual word representation of each of the first plurality of images using one of a pointwise ranking or a pairwise ranking.

5. The method according to claim 1, wherein the features are extracted for generating the visual words using one of:

color histograms;

or

scale invariant feature transforms.

6. The method according to claim 1, wherein an initial ranking of the first plurality of images in the initial ranked list is also taken into consideration when reordering the initial ranked list, wherein the initial ranking is based at least in part upon text-based rankings.

7. A method comprising:

obtaining a ranked list of a first plurality of images from a first source in response to a query;

obtaining a second plurality of images relevant to the query from a plurality of second sources;

comparing, by a processor, visual patterns identified from the second plurality of images with the first plurality of images; and

reranking the first plurality of images based on the comparing of the visual patterns from the second plurality of images with the first plurality of images.

8. The method according to claim 7, wherein comparing the visual patterns identified from the second plurality of images further comprises:

extracting features from the second plurality of images to generate a plurality of visual words, wherein each visual word includes a plurality of similar feature patches extracted from the second plurality of images.

9. The method according to claim 8, further comprising determining recurring visual patterns based on the visual words and connections between the visual words for determining a salient pattern and a concurrent pattern among the visual words.

10. The method according to claim 9, further comprising determining a visual word representation of each of the first plurality of images.

11. The method according to claim 10, wherein reranking the first plurality of images based on the comparing of the visual patterns from the second plurality of images with the first plurality of images further comprises applying the determined salient pattern and concurrent pattern to each of the first plurality of images using one of a pointwise ranking or a pairwise ranking.

12. The method according to claim 7,

wherein the ranked list is initially ranked according to an initial ranking based on relevancy of the first plurality of images to the query,

wherein the ranking of the ranked list is also taken into consideration when reranking the first plurality of images

wherein the ranking of the ranked list is based at least in part upon text-based rankings.

13. The method according to claim 7,

wherein a first server computing device obtains the ranked list in response to a query received from a client computing device,

wherein the first server computing device obtains the second plurality of images from a plurality of second server computing devices in communication with the first server computing device over the Internet,

wherein the first server computing device and the plurality of second server computing devices each provide at least one of a visual content search engine or a visual content database.

14. One or more computer-readable storage devices comprising processor-executable instructions adapted to be executed by a processor for carrying out the method according to claim 7.

15. A computing device comprising:

a processor coupled to a computer-readable storage device including instructions executable by the processor to implement;

a query processing module for obtaining a ranked list of a plurality of first visual content documents in response to a query; and

a reranking module that submits the query to one or more external sources for obtaining a plurality of second visual content documents, wherein the reranking module examines the second visual content documents to extract features from the second visual content documents and reorders the ranked list based on a similarity with the extracted features.

16. The computing device according to claim 15,

wherein the reranking module submits the query to a plurality of server computing devices in communication with the computing device through the Internet as the one or more external sources, wherein each of the server computing devices provides access to at least one of a visual content database or visual content search engine.

17. The computing device according to claim 15, wherein the reranking module identifies features from the plurality of second visual content documents to generate a plurality of visual words, wherein each visual word includes a plurality of image patches having similar visual features extracted from the plurality of second visual content documents.

18. The computing device according to claim 17, wherein the reranking module identifies recurring visual patterns based on the visual words and connections between the visual words for determining a salient pattern and a concurrent pattern among the visual words.

19. The computing device according to claim 15, wherein the reranking module reorders the plurality of first visual content documents by applying the determined salient pattern and concurrent pattern to each of the plurality of first visual content documents using one of a pointwise ranking or a pairwise ranking.

20. The computing device according to claim 15, wherein the computing device is a client computing device having a web browser that comprises the query processing module and the reranking module.