VOICE DIRECTED CONTEXT SENSITIVE VISUAL SEARCH

• US 20130086105A1
• Filed: 10/03/2011
• Published: 04/04/2013
• Est. Priority Date: 10/03/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Hammontree, Monty Lee, Bapat, Vikram, Athans, Emmanuel John
• Granted Patent: US 9,098,533 B2
• Time in Patent Office: Days
• US Class Current: 707/769
• CPC Class Codes: G06F 16/243 Natural language query formulation
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera

Abstract

Various technologies described herein pertain to voice directed context sensitive visual searching. Visual content can be rendered on a display, and a voice directed query related to the visual content can be received. Contextual information related to the visual content can also be identified. Moreover, a search word recognized from the voice directed query and/or the contextual information can be used to detect an object from the visual content, where the object can be a part of the visual content. Further, a search can be performed using the object detected from the visual content, and a result of the search can be rendered on the display.

Claims

1. A method of searching, comprising:

receiving a voice directed query related to visual content rendered on a display, wherein the visual content is one of a frame from a video stream, a two-dimensional image, or a three-dimensional image;

identifying contextual information pertaining to the visual content rendered on the display;

causing a processor to detect an object from the visual content based on at least one of a search word from the voice directed query or the contextual information, wherein the object is a part of the visual content;

performing a search using the object detected from the visual content; and

rendering a result of the search on the display.

2. The method of claim 1, further comprising performing the search using the object detected from the visual content and using one or more of the search word from the voice directed query or the contextual information.

3. The method of claim 1, further comprising:

disambiguating the search word from the voice directed query based on at least one of the visual content or the contextual information; and

performing the search using the object detected from the visual content and the search word as disambiguated.

4. The method of claim 1, detecting the object from the visual content based on the at least one of the search word from the voice directed query or the contextual information further comprises:

performing a disparate search using the at least one of the search word from the voice directed query or the contextual information;

obtaining a result from the disparate search, wherein the result includes a set of images; and

detecting the object from the visual content as a function of the result from the disparate search, wherein the result from the disparate search guides and refines detection of the object from the visual content and a reverse visual search is performed using the object detected from the visual content.

5. The method of claim 1, further comprising:

extracting the object from the visual content based on the at least one of the search word from the voice directed query or the contextual information; and

performing the search using the object extracted from the visual content.

6. The method of claim 5, further comprising rendering the object extracted from the visual content with the result of the search on the display.

7. The method of claim 5, further comprising:

selecting an edge detection algorithm from a set of edge detection algorithms based on at least one of the visual content, the search word from the voice directed query, or the contextual information; and

extracting the object from the visual content using the edge detection algorithm selected from the set of edge detection algorithms.

8. The method of claim 1, further comprising recognizing the search word from the voice directed query.

9. The method of claim 1, further comprising capturing the visual content in response to a voice command recognized from the voice directed query.

10. The method of claim 1, further comprising capturing the frame from the video stream in response to a command recognized from the voice directed query, wherein the visual content is the frame from the video stream.

11. The method of claim 1, further comprising rendering the result of the search and the video stream concurrently on the display, wherein the visual content is the frame selected from the video stream.

13. The method of claim 1, wherein the contextual information further pertains to the voice directed query.

12. (canceled)

14. A system that performs a visual search, comprising:

a processor; and

a memory that comprises a plurality of components that are executed by the processor, the plurality of components comprising;

a context identification component that recognizes contextual information related to at least one of visual content rendered on a display or a voice directed query related to the visual content, wherein the visual content is one of a frame from a video stream, a two-dimensional image, or a three-dimensional image;

an object detection component that detects an object from the visual content based on one or more of a search word recognized from the voice directed query or the contextual information, wherein the object is part of the visual content; and

a search component that performs a search using the object detected from the visual content to generate a result.

15. The system of claim 14, wherein the plurality of components further comprises:

an algorithm selection component that selects an edge detection algorithm from a set of edge detection algorithms based on at least one of the visual content, the search word from the voice directed query, or the contextual information; and

an extraction component that uses the edge detection algorithm selected from the set of edge detection algorithms to extract the object from the visual content, wherein the search component performs the search using the object extracted from the visual content.

16. The system of claim 14, wherein the search component performs a disparate search using the one or more of the search word recognized from the voice directed query or the contextual information, and the object detection component detects the object from the visual content based on a result from the disparate search, wherein the result from the disparate search includes a set of images used by the object detection component to guide and refine detection of the object from the visual content.

17. The system of claim 14, wherein the plurality of components further comprises a query analysis component that disambiguates the search word from the voice directed query, wherein the search component performs the search using the search word as disambiguated with the object detected from the visual content.

18. The system of claim 14, wherein the plurality of components further comprises a render component that concurrently renders the object from the visual content and the result generated by the search component on a display.

19. The system of claim 14, the search component further performs the search using one or more of the search word from the voice directed query or the contextual information.

20. (canceled)

21. A computer-readable storage medium including computer-executable instructions that, when executed by a processor, cause the processor to perform acts including:

rendering visual content on a display, wherein the visual content is one of a frame from a video stream, a two-dimensional image, or a three-dimensional image;

receiving a voice directed query related to the visual content rendered on the display, wherein the voice directed query comprises at least a search word;

recognizing contextual information related to the visual content rendered on the display;

performing a first search using at least one of the search word from the voice directed query or the contextual information;

detecting an object from the visual content as a function of a first result of the first search, wherein the object is a part of the visual content;

performing a second search using the object detected from the visual content; and

rendering a second result of the second search on the display.

22. The computer-readable storage medium of claim 21, wherein the computer-executable instructions, when executed by the processor, further cause the processor to perform acts including:

rendering the second result of the search and the video stream concurrently on the display, wherein the visual content is the frame selected from the video stream.