GESTURE-BASED SEARCH

• US 20120197857A1
• Filed: 01/31/2011
• Published: 08/02/2012
• Est. Priority Date: 01/31/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Huang, Xuedong David, Rajvanshy, Vikas, Lu, Qing (Alex), Jiang, Zhaowei (Charlie)
• Granted Patent: US 10,409,851 B2
• Time in Patent Office: Days
• US Class Current: 707/706
• CPC Class Codes: G06F 16/436 using biological or physiological data of a human being, e.g. blood pressure, facial expression, gestures
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures

Abstract

A search of displayed content may be automatically performed in response to receipt of a search gesture that defines a scope of the search and initiates the search. The search gesture may define a region of content within the displayed content. A search query may be formulated based on the region of content defined by the search gesture. In response to completion of the search gesture, a search may be automatically initiated. In some examples, the search gesture comprises a generally circular gesture that substantially bounds the region of content.

Claims

1. A method of conducting a search, the method comprising:

under control of one or more processors configured with computer-executable instructions;

detecting a gesture defining a region of displayed content;

determining the gesture is a search gesture based at least in part on the curvature of the gesture, the speed at which the gesture is entered, or context information;

formulating a search query based on the region of content defined by the search gesture and context information relevant to the search; and

automatically causing a search to be performed using the search query in response to completion of the search gesture.

2. The method of claim 1, wherein detecting the search gesture comprises detecting a touch input on a touch screen or a touch pad.

3. The method of claim 1, wherein detecting the search gesture comprises detecting a body motion using an image capture device.

4. The method of claim 1, wherein detecting the search gesture comprises detecting a motion of a cursor controlled by a user input device, or detecting a motion of the client device itself.

6. The method of claim 1, wherein the search gesture further comprises a substantially circular motion that bounds or substantially bounds the region of content.

7. The method of claim 1, wherein the region of content defined by the search gesture comprises a region that is touched by, proximate to, or overlapped by the search gesture.

8. The method of claim 1, further comprising displaying a representation of the region of content defined by the search gesture on the displayed content.

9. The method of claim 8, wherein the representation of the region of content comprises a line enclosing the region of content, a highlighting view of the region of content, and/or a magnifying lens view of the region of content.

10. The method of claim 8, further comprising receiving a subsequent gesture in relation to the representation of the region of content, and altering a scope of the search in response to receiving the subsequent gesture.

11. The method of claim 8, further comprising receiving a resizing gesture to resize the representation of the region of content, and reformulating the search query based on the resized region of content.

12. The method of claim 8, further comprising receiving a relocation gesture to relocate the representation of the region of content to include other content, and reformulating the search query based the relocated region of content.

13. The method of claim 1, further comprising receiving a voice input and/or eye tracking data, and refining the search query based on the voice input and/or the eye tracking data.

14. The method of claim 1, further comprising presenting one or more search results of the search by overlaying the one or more search results on the displayed content.

15. The method of claim 14, further comprising expanding a size of the one or more search results in response to a receipt of an expand gesture.

16. The method of claim 15, further comprising collapsing or hiding the one or more search results in response to a collapse gesture.

17. The method of claim 15, further comprising:

presenting one or more search entities along with the one or more search results;

receiving one or more selections of the one or more of the search entities; and

refining the search query based on the one or more selections of the one or more of the search entities.

18. The method of claim 1, wherein the search query comprises content information within the region of content and context information providing signals usable to improve relevance of one or more search results.

19. The method of claim 18, wherein the context information comprises:

content proximate to the region of content;

a paragraph having a portion thereof within the region of content;

a sentence having a portion thereof within the region of content;

a title of a document having a portion of the document within the region of content;

a uniform resource locator where the displayed content is located;

an application identifier of an application used to display the displayed content; and

/or

metadata associated with the displayed content.

20. The method of claim 1, wherein detecting the search gesture comprises:

determining that a user is inputting a gesture; and

determining based on characteristics of the gesture that the gesture is a search gesture usable to initiate the search and is not a gesture usable to pan or scroll the displayed content.

22. The method of claim 1, further comprising receiving a command, prior to detecting the search gesture, the command indicating that the search gesture is imminent.

23. The method of claim 22, wherein the command comprises a touch and hold of a screen of a device at which the gesture is detected, a touch input in a specified location of the screen of the device at which the gesture is detected, a predetermined search gesture, and/or a predetermined motion of the device at which the gesture is detected.

24. The method of claim 1, wherein automatically causing the search to be performed comprises sending the search query to a remote search engine and/or performing the search locally.

5. (canceled)

21. (canceled)

25. One or more computer-readable media storing instructions that, when executed by a processor, configure the processor to perform acts comprising:

detecting a gesture defining a region of displayed content, the gesture comprising a motion that substantially bounds the region of content;

determining the gesture is a search gesture based at least in part on the curvature of the gesture, the speed at which the gesture is entered, or context information;

formulating a search query based on the region of content defined by the search gesture and context information relevant to the search; and

automatically causing a search to be performed using the search query in response to completion of the search gesture.

26. The one or more computer-readable media of claim 25, the acts further comprising displaying a representation of the region of content defined by the search gesture on the displayed content, the representation of the region of content comprising a line enclosing the region of content, a highlighting view of the region of content, and/or a magnifying lens view of the region of content.

27. The one or more computer-readable media of claim 26, the acts further comprising receiving a subsequent gesture in relation to the representation of the region of content, and altering a scope of the search in response to receiving the subsequent gesture.

28. The one or more computer-readable media of claim 25, the acts further comprising receiving a voice input and/or eye tracking data, and refining the search query based on the voice input and/or the eye tracking data.

29. The one or more computer-readable media of claim 25, wherein the search query comprises content information within the region of content and the context information provides signals usable to improve relevance of one or more search results, the context information comprising:

content proximate to the region of content;

a paragraph having a portion thereof within the region of content;

a sentence having a portion thereof within the region of content;

a title of a document having a portion of the document within the region of content;

a uniform resource locator where the displayed content is located;

an application identifier of an application used to display the displayed content; and

/or

metadata associated with the displayed content.

30. A client device comprising:

a processor;

memory communicatively coupled to the processor;

a touch screen configured to display content and to receive user input;

a gesture detection module stored in the memory and executable on the processor to detect a gesture input to the touch screen;

a gesture discrimination module configured to determine whether the gesture is a search gesture based at least in part on the curvature of the gesture, the speed at which the gesture is entered, or context information; and

a content analysis module configured to identify subject matter based on the gesture, and to cause a search to be performed based on the identified subject matter in response to detection of completion of the search gesture by the gesture detection module.

31. The client device of claim 30, the content analysis module further configured to analyze context information providing signals as to a context in which the subject matter is presented.

32. The client device of claim 30, further comprising a gesture tracing module configured to render a representation of the gesture on the touch screen, the representation of the gesture comprising a line, highlighting, and/or a magnifying lens.

33. The client device of claim 30, further comprising an audio input, and a voice recognition module stored in the memory and communicatively coupled to the processor and the audio input to detect voice input and to refine the search query based on the voice input.

34. The client device of claim 30, the gesture detection module further configured to recognize a command indicating that the search gesture is imminent, the command comprising a touch and hold of the touch screen, a touch input in a specified location of the touch screen, and/or a predetermined motion of the client device.

35. A method of conducting a search, the method comprising:

under control of one or more processors configured with computer-executable instructions;

detecting a gesture with regard to displayed content;

determining the gesture is a search gesture based at least in part on the curvature of the gesture, the speed at which the gesture is entered, or context information;

identifying subject matter of the displayed content based on the search gesture; and

automatically causing a search to be performed based at least in part on the identified subject matter in response to completion of the search gesture.

36. The method of claim 35, further comprising identifying context information providing signals as to a context in which the subject matter is presented.

37. The method of claim 35, wherein the context information comprises:

content proximate to the region of content;

a paragraph having a portion thereof within the region of content;

a sentence having a portion thereof within the region of content;

a title of a document having a portion of the document within the region of content;

a uniform resource locator where the displayed content is located;

an application identifier of an application used to display the displayed content; and

/or

metadata associated with the displayed content.

38. The method of claim 37, wherein automatically causing the search to be performed comprises sending a search query to a remote search engine and/or performing the search locally.