CONTEXTUAL IMAGE SEARCH

• US 20110191336A1
• Filed: 01/29/2010
• Published: 08/04/2011
• Est. Priority Date: 01/29/2010
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Xu, Hao, Wang, Jingdong, Hua, Xian-Sheng, Li, Shipeng
• Application Number:
  US12/696,591
• Publication Number:
  US 20110191336A1
• Time in Patent Office: Days
• US Class Current: 707/728
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour

Abstract

Techniques for image search using contextual information related to a user query are described. A user query including at least one of textual data or image data from a collection of data displayed by a computing device is received from a user. At least one other subset of data selected from the collection of data is received as contextual information that is related to and different from the user query. Data files such as image files are retrieved and ranked based on the user query to provide a pre-ranked set of data files. The pre-ranked data files are then ranked based on the contextual information to provide a re-ranked set of data files to be displayed to the user.

Claims

1. A method of contextual image search, the method comprising:

receiving a user query, the user query including at least one of textual data or image data from a collection of data displayed by a computing device;

receiving at least one other subset of data selected from the collection of data as contextual information that is related to and different from the user query;

identifying a first subset of data files from a plurality of data files, the data files of the first subset ranked in a first order according to similarity between information contained in the user query and at least one attribute of individual data files of the plurality of data files;

identifying a second subset of data files from the first subset of data files, the data files of the second subset ranked in a second order according to similarity between the contextual information and at least one attribute of individual data files of the first subset; and

providing for display in the second order a number of images each of which is associated with a respective data file of the second subset.

2. The method of claim 1, wherein the user query includes text displayed by the computing device, and wherein the contextual information includes at least one of a word displayed spatially around the user query, a title of a document displayed by the computing device where the text of the use query is contained, an image in the displayed document, or a video in the displayed document.

3. The method of claim 1, wherein the user query includes an image or a frame of a video displayed by the computing device, wherein when the user query includes an image the contextual information includes at least one of a color moment of at least one displayed image other than the user query, a shape feature of at least one displayed image other than the user query, displayed text data, or a displayed video, and wherein when the user query includes the frame of the video the contextual information includes at least one visual feature of at least one frame of the video displayed by the computing device.

4. The method of claim 1, wherein the receiving at least one other subset of data selected from the collection of data as contextual information that is related to and different from the user query comprises:

identifying at least one instance of textual data displayed in a spatial vicinity of the user query, a title of a document that contains data identified as the user query, an image file name if the user query includes a displayed image, or a combination thereof as part of the contextual information.

5. The method of claim 4, wherein the contextual information is represented as a vector, wherein each of the identified at least one instance of textual data is assigned a respective weight according to a respective distance between the user query and the respective instance of textual data, wherein the identified title of the document is assigned a weight smaller than the respective weight of each of the identified at least one instance of textual data, and wherein the image file name is assigned a weight larger than the respective weight of each of the identified at least one instance of textual data if the user query includes a displayed image.

6. The method of claim 1, wherein the receiving at least one other subset of data selected from the collection of data as contextual information that is related to and different from the user query comprises:

identifying at least one displayed image other than the user query, textual data associated with one or more displayed images other than the user query including respective image file names and surrounding texts, at least one frame of a displayed video, textual data associated with the displayed video including a video file name and surrounding texts, or a combination thereof as an part of the contextual information.

7. The method of claim 6, wherein the contextual information is represented as a vector, wherein each of the at least one displayed image other than the user query, each of the identified at least one instance of textual data in a spatial vicinity of the at least one displayed image other than the user query, and each of the at least one frame of the video is assigned a respective weight according its respective spatial distance from the user query.

8. The method of claim 1, wherein the identifying a first subset of data files comprises:

when the user query is textual data, ranking the first subset of data files in the first order according to similarity between textual data of the user query and textual data of individual data files of the plurality of data files that is related to an image contained in the respective data file.

9. The method of claim 1, wherein the identifying a first subset of data files from a plurality of data files, the data files of the first subset ranked in a first order according to similarity between information contained in the user query and at least one attribute of individual data files of the plurality of data files comprises:

identifying at least one instance of textual data related to the user query when the user query includes an image;

identifying a respective subset of data files from the plurality of data files for each of the at least one instance of textual data related to the user query based on similarity between the respective instance of textual data related to the user query and textual data of each data file of the respective subset of data files that is related to an image contained in the respective data file; and

selecting data files from each respective subset of data files identified for each of the at least one instance of textual data related to the user query to form the first subset of data files, the data files in the first subset of data files arranged in the first order ranked according to similarity between the image of the user query and at least one image of each data file of the first subset of data files.

10. The method of claim 1, wherein the identifying a second subset of data files from the first subset of data files comprises:

ranking each data file of the first subset of data files by comparing one or more attributes of each data file of the first subset with at least one of (1) a textual element of the contextual information, (2) one or more visual features of an image element or one or more texts surrounding the image element of the contextual information, or (3) one or more visual features of a video element or one or more texts surrounding the video element of the contextual information.

11. The method of claim 1, wherein the identifying a second subset of data files from the first subset of data files comprises:

computing a respective first ranking score according to similarity between a textual element of the contextual information and at least one instance of textual data related to the respective image associated with each data file of the second subset of data files;

computing a respective second ranking score according to similarity between a visual feature and texts surrounding the visual feature of an image element of the contextual information and a respective visual feature of and textual data related to the respective image associated with each data file of the second subset of data files;

computing a respective third ranking score according to similarity between a visual feature and texts surrounding the visual feature of a video element of the contextual information and a respective visual feature of and textual data related to the respective image associated with each data file of the second subset of data files; and

combining a ranking score associated with the first subset of data files and the respective first, second, and third ranking scores to provide a respective final ranking score for the respective image of each data file of the second subset of data files.

12. The method of claim 1, wherein each of the plurality of data files includes a respective video, and wherein the data files are ranked according to similarity between at least one attribute of one frame of the respective video in individual data files and at least one of the user query or the contextual information.

13. A method of contextual image search, the method comprising:

ranking a plurality of image files to provide a first list of image files in a first order according to similarity between at least one attribute of individual image files and a user query, the user query including at least one of textual data or image data selected by a user from a collection of displayed data;

ranking the first list of image files to provide a second list of image files in a second order according to similarity between at least one attribute of the individual image files and contextual information that is related to and different from the textual data or image data of the user query, the contextual information including at least one of textual data or image data from the collection of displayed data; and

presenting the image files to a user in the second order.

14. The method of claim 13, wherein the ranking a plurality of image files to provide a first list of image files in a first order comprises:

when the user query includes a displayed image, identifying at least one instance of textual data displayed in a spatial vicinity of the user query;

ranking the plurality of image files using each of the at least one instance of textual data displayed in a spatial vicinity of the user query to provide at least one pre-ranked list of image files; and

ranking each of the at least one pre-ranked list of image files using the displayed image of the user query to provide the first list of image files in the first order.

15. The method of claim 13, wherein the ranking the first list of image files to provide a second list of image files in a second order comprises:

computing a respective first ranking score according to similarity between a textual element of the contextual information and at least one instance of textual data related to each image file of the first list of image files;

computing a respective second ranking score according to similarity between a visual feature and texts surrounding the visual feature of an image element of the contextual information and a respective visual feature of and textual data related to each image file of the first list of image files;

computing a respective third ranking score according to similarity between a visual feature and texts surrounding the visual feature of a video element of the contextual information and a respective visual feature of and textual data related to each image file of the first list of image files; and

combining a ranking score associated with the first list of image files and the respective first, second, and third ranking scores to provide a respective final ranking score for each image file of the first list of image files.

16. The method of claim 13 further comprising:

extracting at least one instance of textual data displayed in a spatial vicinity of the user query, a title of a document containing the user query, or a combination thereof as the contextual information when the user query includes an instance of textual data from the collection of displayed data.

17. The method of claim 16, wherein the contextual information is represented as a vector, wherein each of the extracted at least one instance of textual data is assigned a respective weight according to a respective distance between the user query and the respective instance of textual data, and wherein the extracted title of the document is assigned a weight smaller than the respective weight of each of the extracted at least one instance of textual data.

18. The method of claim 13 further comprising:

extracting at least one instance of textual data displayed in a spatial vicinity of the user query, an image file name of the user query, a title of a document containing the user query, at least one displayed image other than the user query, at least one instance of textual data in a spatial vicinity of the at least one displayed image other than the user query, at least one frame of a displayed video, or a combination thereof as the contextual information when the user query includes a displayed image from the collection of displayed data.

19. The method of claim 18, wherein the context query is represented as a vector, wherein each of the identified at least one instance of textual data, each of the at least one displayed image other than the user query, each of the identified at least one instance of textual data in a spatial vicinity of the at least one displayed image other than the user query, and each of the at least one frame of the displayed video is assigned a respective weight according its respective spatial distance from the user query, wherein the identified title of the document is assigned a weight smaller than the respective weight of each of the identified at least one instance of textual data, and wherein the identified image file name of the user query is assigned a weight larger than the respective weight of each instance of textual data and the respective weight of each of the at least one displayed image other than the user query.

20. One or more computer readable media storing computer-executable instructions that, when executed, perform acts comprising:

ranking a plurality of image files to provide a first list of image files in a first order according to similarity between at least one attribute of individual image files and a user query, the user query including at least one of textual data or image data selected by a user from a collection of displayed data; and

ranking the first list of image files to provide a second list of image files in a second order according to similarity between at least one attribute of the individual image files and contextual information that is related to and different from the textual data or image data of the user query, the contextual information including at least one of textual data or image data from the collection of displayed data.