Voice directed context sensitive visual search

• US 9,098,533 B2
• Filed: 10/03/2011
• Issued: 08/04/2015
• Est. Priority Date: 10/03/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Hammontree, Monty Lee, Bapat, Vikram, Athans, Emmanuel John
• Primary Examiner(s): TRUONG, CAM Y T
• Application Number:
  US13/251,304
• Publication Number:
  US 20130086105A1
• Time in Patent Office: 1,401 Days
• Field of Search: 707/736, 707/769
• US Class Current: 1/1
• CPC Class Codes: G06F 16/243 Natural language query formulation
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera

Abstract

Various technologies described herein pertain to voice directed context sensitive visual searching. Visual content can be rendered on a display, and a voice directed query related to the visual content can be received. Contextual information related to the visual content can also be identified. Moreover, a search word recognized from the voice directed query and/or the contextual information can be used to detect an object from the visual content, where the object can be a part of the visual content. Further, a search can be performed using the object detected from the visual content, and a result of the search can be rendered on the display.

Claims

1. A method of searching, comprising:

receiving a voice directed query related to visual content rendered on a display, wherein the visual content is one of a frame from a video stream, a two-dimensional image, or a three-dimensional image;

causing a processor to detect an object from the visual content based on a search word from the voice directed query;

selecting an edge detection algorithm from a set of edge detection algorithms based on at least one of the visual content, the search word from the voice directed query, or contextual information;

extracting an image of the object from the visual content based on the search word from the voice directed query, the image of the object being extracted from the visual content using the edge detection algorithm selected from the set of edge detection algorithms, wherein the image of the object is a portion of the visual content and is extracted from a remainder of the visual content;

using the image of the object extracted from the visual content as an input for a reverse visual search, wherein the reverse visual search is executed based upon the image of the object extracted from the visual content, and wherein the reverse visual search returns a result; and

rendering the result of the reverse visual search on the display.

2. The method of claim 1, further comprising narrowing the reverse visual search based on one or more of the search word from the voice directed query or the contextual information.

3. The method of claim 1, further comprising:

disambiguating the search word from the voice directed query based on the visual content; and

using the search word as disambiguated as an additional search term to return the result.

4. The method of claim 1, detecting the object from the visual content based on the search word from the voice directed query further comprises:

using the search word from the voice directed query as an input for a disparate search;

obtaining a result from the disparate search, wherein the result from the disparate search includes a set of images; and

detecting the object from the visual content as a function of the result from the disparate search, wherein the result from the disparate search guides and refines detection of the object from the visual content.

5. The method of claim 1, further comprising rendering the image of the object extracted from the visual content with the result of the reverse visual search on the display.

6. The method of claim 1, further comprising capturing the visual content in response to a voice command recognized from the voice directed query.

7. The method of claim 1, further comprising capturing the frame from the video stream in response to a command recognized from the voice directed query, wherein the visual content is the frame from the video stream.

8. The method of claim 1, further comprising rendering the result of the reverse visual search and the video stream concurrently on the display, wherein the visual content is the frame selected from the video stream.

9. The method of claim 1, further comprising:

disambiguating the search word from the voice directed query based on the contextual information; and

using the search word as disambiguated as an additional search term to return the result.

10. The method of claim 1, wherein an edge of the image of the object is not delineated in the visual content prior to the detecting of the object.

11. A system that performs a visual search, comprising:

a processor; and

a memory that comprises a plurality of components that are executed by the processor, the plurality of components comprising;

an object detection component that detects an object from visual content rendered on a display based on a search word recognized from a voice directed query related to the visual content, wherein the visual content is one of a frame from a video stream, a two-dimensional image, or a three-dimensional image;

an algorithm selection component that selects an edge detection algorithm from a set of edge detection algorithms based on at least one of the visual content, the search word from the voice directed query, or contextual information;

an extraction component that extracts an image of the object from the visual content using the edge detection algorithm selected from the set of edge detection algorithms, wherein the image of the object is a portion of the visual content and is extracted from a remainder of the visual content; and

a search component that uses the image of the object extracted from the visual content as an input for a reverse visual search, wherein the reverse visual search is performed based upon the image of the object extracted from the visual content, and wherein the reverse visual search returns a result.

12. The system of claim 11, wherein the search component uses the search word recognized from the voice directed query as an input for a disparate search, and the object detection component detects the object from the visual content based on a result from the disparate search, wherein the result from the disparate search includes a set of images used by the object detection component to guide and refine detection of the object from the visual content.

13. The system of claim 11, wherein the plurality of components further comprises a query analysis component that disambiguates the search word from the voice directed query, wherein the search component uses the search word as disambiguated as an additional search term to return the result.

14. The system of claim 11, wherein the plurality of components further comprises a render component that concurrently renders the image of the object from the visual content and the result generated by the search component on a display.

15. The system of claim 11, the search component further narrows the reverse visual search based on one or more of the search word from the voice directed query or the contextual information.

16. The system of claim 11, wherein the plurality of components further comprises a context identification component that recognizes contextual information related to at least one of the visual content rendered on the display or the voice directed query related to the visual content, wherein the object detection component further detects the object from the visual content based on the contextual information.

17. A computer-readable memory including computer-executable instructions that, when executed by a processor, cause the processor to perform acts including:

rendering visual content on a display, wherein the visual content is one of a frame from a video stream, a two-dimensional image, or a three-dimensional image;

receiving a voice directed query related to the visual content rendered on the display, wherein the voice directed query comprises at least a search word;

using the search word from the voice directed query as an input for a first search;

detecting an object from the visual content as a function of a first result of the first search;

selecting an edge detection algorithm from a set of edge detection algorithms;

extracting an image of the object from the visual content using the edge detection algorithm selected from the set of edge detection algorithms, wherein the image of the object is a portion of the visual content and is extracted from a remainder of the visual content;

using the image of the object extracted from the visual content as an input for a reverse visual search, wherein the reverse visual search is executed based upon the image of the object extracted from the visual content; and

rendering a second result of the reverse visual search on the display.

18. The computer-readable memory of claim 17, wherein the computer-executable instructions, when executed by the processor, further cause the processor to perform acts including:

rendering the second result of the reverse visual search and the video stream concurrently on the display, wherein the visual content is the frame selected from the video stream.

19. The computer-readable memory of claim 17, wherein the edge detection algorithm is selected from the set of edge detection algorithms based on at least one of the visual content or the search word from the voice directed query.