Gestures, interactions, and common ground in a surface computing environment

• US 9,134,798 B2
• Filed: 12/15/2008
• Issued: 09/15/2015
• Est. Priority Date: 12/15/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Morris, Meredith June, Horvitz, Eric J., Wilson, Andrew David, Jones, F. David, Hodges, Stephen E., Hinckley, Kenneth P., Butler, David Alexander, Sands, Ian M., Russ, V. Kevin, Benko, Hrvoje, LeProwse, Shawn R., Izadi, Shahram, Kunz, William Ben
• Primary Examiner(s): Okebato, Sahlu
• Application Number:
  US12/335,301
• Publication Number:
  US 20100149090A1
• Time in Patent Office: 2,465 Days
• Field of Search: 345/156, 345/158, 715/862, 715/863
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: H04L 67/01 Protocols

Abstract

Aspects relate to detecting gestures that relate to a desired action, wherein the detected gestures are common across users and/or devices within a surface computing environment. Inferred intentions and goals based on context, history, affordances, and objects are employed to interpret gestures. Where there is uncertainty in intention of the gestures for a single device or across multiple devices, independent or coordinated communication of uncertainty or engagement of users through signaling and/or information gathering can occur.

Claims

1. A method performed by one or more processors that are configured with executable instructions to perform acts comprising:

receiving, by a first computing device, a gesture input from a touch-sensitive input device;

in response to determining that the gesture input comprises;

an undefined gesture that is excluded from a set of gestures stored by the first computing device, receiving a set of user gestures from a second computing device;

in response to determining that the gesture input is included in the set of user gestures,

performing an action corresponding to the gesture input; and

in response to determining that a second gesture input received within a predetermined time period comprises a duplicate gesture, ignoring the duplicate gesture, wherein the gesture input and the second gesture input are provided to a same software application being executed by the first computing device.

2. The method of claim 1, the method further comprising:

determining a context associated with the gesture input; and

determining the action based at least partly on the context associated with the gesture input.

3. The method of claim 1, the method further comprising:

providing a visual representation of the gesture input.

4. The method of claim 1, the method further comprising:

in response to determining that the gesture input is included in a set of common gestures, determining a predetermined action corresponding to the gesture input based on the set of common gestures; and

performing the predetermined action.

5. The method of claim 1, the method further comprising:

identifying the second computing device as being associated with a user that generated the gesture input;

sending a request to the second computing device to provide the set of user gestures; and

receiving the set of user gestures from the second computing device.

6. The method of claim 1, further comprising:

in response to determining that a third gesture input is excluded from the set of user gestures, prompting a user to select an additional action to perform when the third gesture is received.

7. A device comprising:

a touch-screen device;

one or more processors;

one or more memory devices to store instructions that are executable by the one or more processors to perform operations comprising;

receiving a gesture input from the touch-screen device;

in response to determining that the gesture input is excluded from a set of gestures stored in the one or more memory devices, receiving a set of user gestures from a second computing device;

determining an action corresponding to the gesture input based at least partly on the set of user gestures;

performing the action corresponding to the gesture input; and

in response to determining that a second gesture input received from the touch-screen device within a predetermined time period after receiving the gesture input comprises a duplicate gesture of the gesture input, ignoring the duplicate gesture, wherein the gesture input and the second gesture input are provided to a same software application being executed by the device.

8. The device of claim 7, the operations further comprising:

displaying a visual representation of the gesture.

9. The device of claim 7, the operations further comprising:

determining a context associated with the gesture input; and

determining the action corresponding to the gesture input based at least partly on the context associated with the gesture input.

10. The device of claim 7, the operations further comprising;

determining that a third gesture input is included in the set of gestures; and

automatically performing a predetermined action corresponding to the third gesture input.

11. The device of claim 7, the operations further comprising:

determining that the second computing device is associated with a user that generated the gesture input; and

receiving the set of user gestures from the second computing device in response to sending, from the device to the second computing device, a request to send the set of user gestures.

12. The device of claim 7, the operations further comprising:

in response to determining that a third gesture is excluded from the set of user gestures, receiving user input selecting a third action to perform when the third gesture is received.

13. The device of claim 7, the operations further comprising:

tracking a gaze of a user that is using the device;

identifying, based on the gaze of the user, an item displayed on a display device; and

performing the action to the item.

14. A method comprising:

detecting, by a first computing device, a gesture performed on a touch-screen display of the first computing device;

determining that the gesture is excluded from a set of gestures stored on the first computing device;

requesting a set of user gestures from a second computing device;

performing an action corresponding to the gesture based at least partly on the set of user gestures;

determining that a second gesture input received from the touch-screen display within a predetermined time period comprises a duplicate gesture, wherein the gesture and the second gesture are sent to a same software application executing on the first computing device; and

ignoring the duplicate gesture.

15. The method of claim 14, further comprising:

determining a context associated with the gesture; and

determining the action corresponding to the gesture based at least partly on the context associated with the gesture.

16. The method of claim 14, further comprising:

determining that a third gesture is included in the set of gestures; and

performing a pre-determined action corresponding to the third gesture.

17. The method of claim 14, further comprising:

tracking a gaze of a user that input the gesture;

determining, based on the gaze of the user, a context associated with the gesture; and

performing a pre-determined action based at least partly on the context.

18. The method of claim 14, further comprising:

determining that the second computing device is associated with a user that input the gesture;

sending a request to the second computing device; and

receiving the set of user gestures from the second computing device.

19. The method of claim 14, further comprising:

in response to determining that a fourth gesture is excluded from the set of user gestures, receiving user input selecting a fourth action;

associating the fourth action with the fourth gesture; and

storing the fourth action and the fourth gesture in the set of user gestures.

20. The method of claim 14, further comprising:

creating the set of gestures by training a classifier implemented using a support vector machine.