MULTI-MODAL APPROACH TO SEARCH QUERY INPUT

• US 20120117051A1
• Filed: 11/05/2010
• Published: 05/10/2012
• Est. Priority Date: 11/05/2010
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: SUN, JIAN, KUO, YU-TING, LI, YI, KE, QIFA, LIU, JIYANG, SHUM, HEUNG-YEUNG, YANG, XIAOSONG, ZHANG, LEI, LIU, CE
• Application Number:
  US12/940,538
• Publication Number:
  US 20120117051A1
• Time in Patent Office: Days
• US Class Current: 707/707
• CPC Class Codes: G06F 16/43 Querying
• CPC Class Codes: G06F 16/9032 Query formulation
• CPC Class Codes: G06F 16/90335 Query processing

Abstract

Search queries containing multiple modes of query input are used to identify responsive results. The search queries can be composed of combinations of keyword or text input, image input, video input, audio input, or other modes of input. The multiple modes of query input can be present in an initial search request, or an initial request containing a single type of query input can be supplemented with a second type of input. In addition to providing responsive results, in some embodiments additional query refinements or suggestions can be made based on the content of the query or the initially responsive results.

Claims

1. One or more computer-storage media storing computer-useable instructions that, when executed by a computing device, perform a method for performing a multi-modal search, comprising:

acquiring an image, a video, or an audio file that includes a plurality of relevance features that can be extracted;

associating the image, video, or audio file with at least one keyword;

submitting the image, video, or audio file and the associated keyword as a query to a search engine;

receiving at least one responsive result that is responsive to both the plurality of relevance features and the associated keyword; and

displaying the at least one responsive result.

2. The computer-storage media of claim 1, wherein the image, video, or audio file further comprises metadata corresponding to the image, video, or audio file.

3. The computer-storage media of claim 2, wherein the at least one responsive result is responsive to the plurality of relevance features, the associated keyword, and one or more keywords extracted from the metadata corresponding to the image, video, or audio file.

4. The computer-storage media of claim 1, wherein acquiring the image or the video comprises acquiring an image from a camera associated with an acquiring device.

5. The computer-storage media of claim 1, wherein acquiring the image, video, or audio file comprises accessing a stored input via a network.

6. The computer-storage media of claim 1, wherein the at least one responsive result comprises a text document, an image, a video, an audio file, or a combination thereof.

7. The computer-storage media of claim 1, wherein the at least one responsive result comprises an identity of a text document, an identity of an image, an identity of a video, or an identity of an audio file.

8. The computer-storage media of claim 1, wherein the method further comprises displaying one or more query suggestions based on the submitted query and metadata corresponding to at least one responsive result.

9. A method for performing a multi-modal search, comprising:

receiving a query including at least two query modes;

extracting relevance features corresponding to the at least two query modes from the query;

selecting a plurality of responsive results based on the extracted relevance features;

ranking the plurality of responsive results based on the extracted relevance features; and

displaying one or more of the ranked responsive results.

10. The method of claim 9, wherein the query modes in the received query include two or more a keyword, an image, a video, or an audio file.

11. The method of claim 9, wherein the plurality of responsive documents are selected using an inverted index incorporating relevance features from the at least two query modes.

12. The method of claim 11, wherein relevance features extracted from the image, video, or audio file are incorporated into the inverted index as descriptor keywords.

13. The method of claim 9, wherein the extracted relevance features correspond to a keyword and an image.

14. The method of claim 9, further comprising:

extracting metadata from an image, a video, or an audio file;

identifying one or more keywords from the extracted metadata; and

forming a second query including at least the extracted relevance features from the received query and the keywords identified from the extracted metadata.

15. The method of claim 14, wherein ranking the plurality of responsive documents based on the extracted relevance features comprises ranking the plurality of responsive documents based on the second query.

16. The method of claim 14, wherein the second query is displayed in association with the displayed responsive results.

17. The method of claim 14, further comprising:

automatically selecting a second plurality of responsive documents based on the second query;

ranking the second plurality of responsive documents based on the second query; and

displaying at least one document from the second plurality of responsive documents.

18. A method for performing a multi-modal search, comprising:

receiving a query comprising at least one keyword;

displaying a plurality of responsive results based on the received query;

receiving supplemental query input comprising at least one of an image, a video, or an audio file;

modifying a ranking of the plurality of responsive results based on the supplemental query input; and

displaying one or more of the responsive results based on the modified ranking.

19. The method of claim 18, further comprising:

extracting additional keywords from metadata associated with the at least one image, video, or audio file;

incorporating the extracted additional keywords into the supplemental query.

20. The method of claim 18, further comprising:

extracting additional keywords from at least one responsive result based on metadata associated with the responsive result, the responsive result being an image, a video, or an audio file;

incorporating the extracted additional keywords into the supplemental query.