VISUAL AND MULTI-DIMENSIONAL SEARCH

• US 20110167053A1
• Filed: 03/15/2011
• Published: 07/07/2011
• Est. Priority Date: 06/28/2006
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Naam, Ramez, Flake, Gary W., Chaudhuri, Surajit, Gupta, Anoop, Goodman, Joshua T., Brill, Eric D., Lawler, Stephen, Meek, Christopher A., Hurst-Hiller, Oliver, Horvitz, Eric J.
• Application Number:
  US13/048,661
• Publication Number:
  US 20110167053A1
• Time in Patent Office: Days
• US Class Current: 707/709
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/432 Query formulation
• CPC Class Codes: G06F 16/433 using audio data
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results

Abstract

A system that can analyze a multi-dimensional input thereafter establishing a search query based upon extracted features from the input. In a particular example, an image can be used as an input to a search mechanism. Pattern recognition and image analysis can be applied to the image thereafter establishing a search query that corresponds to features extracted from the image input. The system can also facilitate indexing multi-dimensional searchable items thereby making them available to be retrieved as results to a search query. More particularly, the system can employ text analysis, pattern and/or speech recognition mechanisms to extract features from searchable items. These extracted features can be employed to index the searchable items.

Claims

1. A method comprising:

receiving, by a computing device including a processing unit, an input indicating selection of at least a portion of an image;

generating, by the computing device, a search query including text-based keywords based on one or more features of the image; and

retrieving, by the computing device, a plurality of data items in response to the search query based on features of the plurality of data items and the text-based keywords.

2. The method as recited in claim 1, wherein the input further includes text data, and the method further comprises parsing the text data to determine additional text-based keywords of the search query.

3. The method as recited in claim 1, wherein the input further includes audible data and the method further comprises parsing the audible data to establish additional text-based keywords of the search query.

4. The method as recited in claim 1, wherein one or more of the plurality of data items retrieved in response to the search query are rendered with at least one annotation.

5. The method as recited in claim 4, wherein the at least one annotation includes highlighting, underlining, font formatting, or a combination thereof.

6. The method as recited in claim 1, wherein the plurality of data items are retrieved based on data items previously selected by a user of the computing device with respect to queries similar to the search query.

7. The method as recited in claim 1, wherein the search query is generated utilizing sample figures.

8. One or more computer storage media including processor-executable instructions that, when executed by a processor, perform operations comprising:

receiving an input including a text portion, an audio portion, an image portion, or a combination thereof;

extracting one or more features of the input based, at least in part, on user preferences; and

generating a search query including text-based keywords based on the one or more features extracted from the input.

9. The one or more computer storage media as recited in claim 8, wherein the one or more features of the input are extracted based on user feedback with respect to previous searches.

10. The one or more computer storage media, as recited in claim 8, wherein particular features of the input are weighted more heavily than other features of the input in order to determine the one or more features of the input to be extracted.

11. The one or more computer storage media, as recited in claim 10, wherein images at the center of the input are weighted more heavily than other images of the input.

12. The one or more computer storage media, as recited in claim 10, wherein audio features of the input within a frequency range of human speech are weighted more heavily than sounds outside the frequency range of human speech.

13. The one or more computer storage media, as recited in claim 8 wherein the input includes a document and the one or more features are extracted from the input based on a length of the document, a structure of the document, or both.

14. The one or more computer storage media, as recited in claim 13, wherein the one or more features of the document are extracted based on whether the document includes a set of references at the end of the document, types of figures included in the document, image-based indications of geocentric content, or a combination thereof.

15. A system comprising:

a processor;

an input analyzer component executable by the processor to;

parse text data to establish text-based keywords of search queries when an input includes the text data;

establish additional text-based keywords of the search queries based on audible data when the input includes the audible data; and

analyze visible data to establish further text-based keywords of the search queries when the input includes the visible data; and

a search engine component executable by the processor to;

provide results of the search queries that are rendered as text results, image results, or both.

16. The system as recited in claim 15, wherein a particular search query of the search queries includes one or more text-based keywords and one or more images.

17. The system as recited in claim 15, wherein the search engine component is further executable by the processor to:

extract features from a plurality of searchable data items;

index each of the searchable data items based on one or more features extracted from each of the searchable data items; and

provide results of a particular search query based on the features of the plurality of searchable data items and text-based keywords of the particular search query.

18. The system as recited in claim 15, further comprising a filter component executable by the processor to infer one or more filtering criteria and automatically filter a subset of the results based on the one or more filtering criteria.

19. The system as recited in claim 18, wherein the one or more filtering criteria include time of day, location, device context, or combinations thereof.

20. The system as recited in claim 15, wherein the input analyzer component is further executable by the processor to identify one or more subjects of an image, a context of the image, or both, to generate a particular search query.