Gestures, interactions, and common ground in a surface computing environment

• US 10,409,381 B2
• Filed: 08/10/2015
• Issued: 09/10/2019
• Est. Priority Date: 12/15/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Morris, Meredith June, Horvitz, Eric J., Wilson, Andrew David, Jones, F. David, Hodges, Stephen E., Hinckley, Kenneth P., Butler, David Alexander, Sands, Ian M., Russ, V. Kevin, Benko, Hrvoje, LeProwse, Shawn R., Izadi, Shahram, Kunz, William Ben
• Primary Examiner(s): Okebato, Sahlu
• Application Number:
  US14/822,456
• Publication Number:
  US 20150346837A1
• Time in Patent Office: 1,492 Days
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: H04L 67/01 Protocols

Abstract

Aspects relate to detecting gestures that relate to a desired action, wherein the detected gestures are common across users and/or devices within a surface computing environment. Inferred intentions and goals based on context, history, affordances, and objects are employed to interpret gestures. Where there is uncertainty in intention of the gestures for a single device or across multiple devices, independent or coordinated communication of uncertainty or engagement of users through signaling and/or information gathering can occur.

Claims

1. A method performed by one or more processors that are configured with executable instructions, the method comprising:

receiving, by a first computing device, a first gesture input from a touch-sensitive input device;

determining whether the first gesture input is an undefined gesture that is absent from a first set of gestures stored by the first computing device;

communicating, over a network, with a second computing device of a user to obtain a second set of gestures of the user stored by the second computing device, the second set of gestures including a plurality of gestures defined by the user, each gesture of the second set of gestures associated with a particular command;

automatically converting the second set of gestures into gestures to be used on the first computing device;

determining whether the first gesture input is a gesture included in the automatically converted second set of gestures;

determining a command associated with the first gesture input based on a gesture of the automatically converted second set of gestures when the first gesture input is determined to be a gesture included in the automatically converted second set of gestures; and

performing the determined command associated with the first gesture input.

2. The method of claim 1, wherein:

the second computing device comprises a server; and

the network comprises an Internet.

3. The method of claim 1, further comprising:

receiving a second gesture input within a predetermined time period after receiving the first gesture input;

determining whether the second gesture input is a duplicate gesture based on the first gesture input, wherein the first gesture input and the second gesture input are associated with a same software application; and

ignoring the duplicate gesture when the second gesture input is determined to be the duplicate gesture.

4. The method of claim 1, further comprising:

determining whether a second gesture input is included in the first set of gestures;

determining a second command corresponding to the second gesture input based on the first set of gestures or the automatically converted second set of gestures; and

performing the determined second command corresponding to the second gesture input.

5. The method of claim 1, further comprising:

determining whether a second gesture is absent from the first set of gestures; and

displaying a prompt to select a second command to be associated with the second gesture when the second gesture is absent from the first set of gestures;

receiving selection of the second command; and

associating the selected second command with the second gesture.

6. A first computing device comprising:

an input device;

one or more processors;

one or more memory devices to store instructions that are executable by the one or more processors to perform operations comprising;

receiving a first gesture input from the input device;

determining whether the first gesture input is absent from a first set of gestures stored in the one or more memory devices;

communicating, over a network, with a second computing device of a user to obtain a second set of gestures of the user stored by the second computing device, the second set of gestures including a plurality of gestures defined by the user, each gesture of the second set of gestures associated with a particular command;

automatically converting the second set of gestures into gestures to be used on the first computing device;

identifying a first command associated with the first gesture input based on a gesture of the automatically converted second set of gestures; and

sending the identified first command to a software application.

7. The first computing device of claim 6, wherein the second computing device comprises a server.

8. The first computing device of claim 6, wherein the network comprises an Internet.

9. The first computing device of claim 6, wherein the operations further comprise:

receiving a second gesture input;

determining whether the second gesture input is included in the first set of gestures;

determining a second command is associated with the second gesture input based on a gesture of the first set of gestures when the second gesture input is included in the first set of gestures; and

performing the determine second command associated with the second gesture input.

10. The first computing device of claim 6, wherein communicating, over the network, with the second computing device of the user to obtain the second set of gestures of the user stored by the second computing device comprises:

sending a request, over the network, from the first computing device to the second computing device to provide the second set of gestures; and

receiving, over the network, the second set of gestures from the second computing device.

11. The first computing device of claim 6, wherein the operations further comprise:

receiving a second gesture input;

determining whether the second gesture input is absent from the first set of gestures;

receiving user input selecting a second command to be associated with the second gesture input when the second gesture is absent from the first set of gestures; and

associating the selected second command with the second gesture input.

12. The first computing device of claim 6, wherein the operations further comprise:

tracking a gaze of the user that is using the first computing device; and

identifying, based on the gaze of the user, an item displayed on a display device, wherein sending the identified first command to the software application causes the software application to perform the identified first command on the item displayed on the display device.

13. A method comprising:

receiving, by a first computing device, a first gesture input from an input device associated with the first computing device;

determining whether the first gesture input is absent from a first set of gestures stored on the first computing device;

communicating, over a network, with a second computing device of a user to obtain a second set of gestures of the user stored by the second computing device, the second set of gestures including a plurality of gestures defined by the user, each gesture of the second set of gestures associated with a particular command;

automatically converting the second set of gestures into gestures to be used on the first computing device;

determining whether the first gesture input is a gesture included in the automatically converted second set of gestures;

determining a first command associated with the first gesture input based on a gesture of the automatically converted second set of gestures when the first gesture input is determined to be a gesture included in the automatically converted second set of gestures;

performing the determined first command associated with the first gesture input;

receiving a second gesture input from the input device within a predetermined time period from receiving the first gesture input;

determining whether the second gesture input is a duplicate gesture based on the first gesture input, wherein the first gesture input and the second gesture input are associated with a same software application executing on the first computing device; and

ignoring the second gesture input when the second gesture input is determined to be the duplicate gesture.

14. The method of claim 13, wherein:

the second computing device comprises a server; and

the network comprises an Internet.

15. The method of claim 13, wherein communicating, over the network, with the second computing device of the user to obtain the second set of gestures of the user stored by the second computing device comprises:

sending a request, over the network, to the second computing device to provide the second set of gestures; and

receiving the second set of gestures from the second computing device over the network.

16. The method of claim 13, further comprising:

receiving a third gesture input;

determining whether the third gesture input is included in at least one of the first set of gestures or the automatically converted second set of gestures;

determining a third command associated with the third gesture input based on a gesture of the at least one of the first set of gestures or the automatically converted second set of gestures when the third gesture input is determined to be included in the at least one of the first set of gestures or the automatically converted second set of gestures; and

performing the determined third command associated with the third gesture input.

17. The method of claim 13, further comprising:

receiving a third gesture input;

determining whether the third gesture input is absent from both the first set of gestures and the automatically converted second set of gestures;

receiving user input selecting a third command to be associated with the third gesture input when the third gesture is absent from both the first set of gestures and the automatically converted second set of gestures; and

associating the selected third command with the third gesture input.

18. The method of claim 13, further comprising:

receiving a third gesture input;

determining whether the third gesture input is absent from both the first set of gestures and the automatically converted second set of gestures;

receiving, over the network, a third set of gestures from a third computing device when the third gesture input is absent from both the first set of gestures and the automatically converted second set of gestures;

determining whether the third gesture input is included in the third set of gestures;

determining a third command associated with the third gesture input based on a gesture of the third set of gestures when the third gesture input is determined to be included in the third set of gestures; and

performing the determined third command associated with the third gesture input.

19. The method of claim 13, further comprising:

sending, over the network, the first set of gestures from the first computing device to the second computing device.