Visual and multi-dimensional search

• US 7,917,514 B2
• Filed: 06/28/2006
• Issued: 03/29/2011
• Est. Priority Date: 06/28/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Naam, Ramez, Brill, Eric D., Lawler, Stephen, Meek, Christopher A., Flake, Gary W., Chaudhuri, Surajit, Gupta, Anoop, Goodman, Joshua T., Hurst-Hiller, Oliver, Horvitz, Eric J.
• Primary Examiner(s): Pham; Khanh B
• Assistant Examiner(s): Ahluwalia; Navneet K
• Application Number:
  US11/427,305
• Publication Number:
  US 20080005105A1
• Time in Patent Office: 1,735 Days
• Field of Search: 707 1- 10, 707/711, 707/739
• US Class Current: 707/739
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/432 Query formulation
• CPC Class Codes: G06F 16/433 using audio data
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results

Abstract

A system that can analyze a multi-dimensional input thereafter establishing a search query based upon extracted features from the input. In a particular example, an image can be used as an input to a search mechanism. Pattern recognition and image analysis can be applied to the image thereafter establishing a search query that corresponds to features extracted from the image input. The system can also facilitate indexing multi-dimensional searchable items thereby making them available to be retrieved as results to a search query. More particularly, the system can employ text analysis, pattern and/or speech recognition mechanisms to extract features from searchable items. These extracted features can be employed to index the searchable items.

Claims

1. A system embodied on a computer-readable storage medium that facilitates multi-dimension computer search, comprising:

a processor;

an input analyzer component executable by the processor to;

receive an input indicating selection of at least a portion of a picture image;

identify one or more features, attributes, characteristics, or a combination thereof, of the at least a portion of the picture image;

generate a search query including text-based keywords and visual criteria based on the one or more features, attributes, characteristics, or a combination thereof, of the at least a portion of the picture image; and

a search engine comprising;

a multi-dimension index generator executable by the processor to collect and catalog a plurality of searchable data items as a function of a plurality of index criteria, the plurality of index criteria including at least visual criteria extracted from one or more image data items using at least one of pattern recognition or facial recognition analysis; and

a retrieval component executable by the processor to access and retrieve a subset of the plurality of data items in response to the search query based on the text-based keywords and by comparing the visual criteria specified in the search query with the visual criteria extracted from the one or more image data items, the subset of the plurality of data items including one or more web pages, one or more image files, one or more audio files, one or more multi-media files, or a combination thereof.

2. The system of claim 1, the plurality of index criteria further includes at least one of text or audio.

3. The system of claim 1,

wherein the input analyzer component extracts additional search criteria from audio data of the input;

and

wherein the search engine identifies a plurality of indexed data items having index criteria matching the additional search criteria.

4. The system of claim 3, the input analyzer component identifies an individual included in the portion of the image data, and the search engine employs the identified individual as search criteria.

5. The system of claim 1, the index generator comprises at least one of a text analyzer, a sound analyzer, or an image analyzer that facilitates identification of the plurality of index criteria.

6. The system of claim 5, the text analyzer includes a language parser that automatically extracts the plurality of index criteria from the plurality of searchable data items.

7. The system of claim 5, the sound analyzer includes a speech recognition component that automatically extracts the plurality of index criteria from the plurality of searchable data items.

8. The system of claim 5, the image analyzer includes a pattern recognition component that automatically extracts the plurality of index criteria from the plurality of searchable data items.

9. The system of claim 1, further comprising a results configuration component that selectively organizes the accessed subset of the plurality of searchable data items as a function of a rendering criterion.

10. The system of claim 9, the results configuration component includes a filter component that sorts the subset of the plurality of searchable data items in accordance with at least one of a user context, a user preference, a relevancy factor with respect to the input, or a device context.

11. The system of claim 9, the results configuration component includes a ranking component that arranges the subset of the plurality of searchable data items based at least in part on a user preference.

12. The system of claim 9, the results configuration component includes an ordering component that determines a display order with respect to a subset of the plurality of searchable data items.

13. A method of indexing multidimensional searchable data items having visual and audio characteristics, the method comprising:

employing a processor executing computer-executable instructions stored on a computer-readable storage medium to implement the following acts;

receiving an image file as an input, the image file including a document having at least one image;

identifying one or more individuals, a context of the one or more individuals, a location of the one or more individuals, one or more landmarks, one or more venues, one or more buildings, or a combination thereof, of the at least one image;

identifying properties of the document;

generating a search query including text-based keywords and visual criteria based on the properties of the document, the one or more individuals, the context of the one or more individuals, the location of the one or more individuals, the one or more landmarks, the one or more venues, the one or more buildings, or a combination thereof, of the at least one image;

analyzing the visual and audio characteristics of the searchable data items, the searchable data items including a plurality of individuals, contexts of the plurality of individuals, locations of the plurality of individuals, a plurality of landmarks, a plurality of venues;

a plurality of buildings, or a combination thereof;

identifying one or more patterns within the visual and audio characteristics of the searchable data items;

generating a plurality of index parameters based at least in part upon a subset of the identified one or more patterns;

indexing the searchable data items as a function of the plurality of index parameters; and

providing results to the search query utilizing the plurality of index parameters, the results to the search query including one or more web pages, one or more image files, one or more audio files, one or more multi-media files, or a combination thereof.

14. The method of claim 13, further comprising:

recognizing audio patterns in the searchable data items; and

converting the one or more patterns within the audio characteristics into a subset of the plurality of index parameters.

15. The method of claim 13, further comprising:

executing a search query that includes a visual image as input;

extracting search criteria from the visual image using at least one of pattern recognition or facial recognition; and

returning a particular searchable data item in response to the search query based on relevance to the extracted search criteria.

20. The method of claim 13, wherein the acts further comprise recognizing trends of graphical data of the at least one image and wherein the visual criteria include the trends of the graphical data.

16. A computer-executable system embodied on a computer-readable storage medium that when executed by a processor facilitates indexing multi-dimensional data items, the system comprising:

means for analyzing a plurality of multi-dimensional data items having visual and audio characteristics;

means for extracting a plurality of features relating to the visual and audio characteristics of the plurality of multi-dimensional data items;

means for converting the extracted plurality of features into a set of index criteria for the plurality of multi-dimensional data items;

means for indexing the plurality of multi-dimensional data items according to the set of index criteria;

means for receiving an input including an audio portion, a picture image portion, a text portion, or a combination thereof;

means for determining whether the input includes at least one of an audio dimension, an image dimension, or a text dimension, the audio dimension including streaming audio and the image dimension including streaming video;

means for parsing the input when the input includes a text portion to identify keywords of the text portion;

means for translating audible speech into text when the input includes an audio portion to establish keywords of the audio portion;

means for identifying one or more subjects, a context, or a combination thereof, of the picture image portion when the input includes the picture image portion;

means for generating a search query based on the plurality of features extracted from the input, wherein the search query includes search criteria and the search criteria include text, one or more images, or a combination thereof; and

means for selecting a subset of the indexed data items in response to the search query based in part on a comparison between the search criteria and the set of index criteria, the subset of the indexed data items including one or more web pages, one or more image files, one or more audio files, one or more multi-media files, or a combination thereof.

17. The computer-executable system of claim 16, the means for analyzing the plurality of multi-dimensional data items is an audio analyzer system that extracts a subset of the plurality of features from the audio characteristics of a particular multi-dimensional data item.

18. The computer-executable system of claim 16, the means for analyzing the plurality of multi-dimensional data items is an image analyzer system that extracts a subset of the plurality of features from the visual characteristics of a particular multi-dimensional data item.

19. The computer-executable system of claim 16, further comprising means for searching the plurality of multi-dimensional data items based at least in part upon two dimensions.