Customizable gestures for mobile devices

• US 9,600,169 B2
• Filed: 02/27/2012
• Issued: 03/21/2017
• Est. Priority Date: 02/27/2012
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: de Sa, Marco, Churchill, Elizabeth F., McAfee, R. Preston, Shamma, David Ayman
• Primary Examiner(s): IPAKCHI, MARYAM M
• Application Number:
  US13/405,708
• Publication Number:
  US 20130227418A1
• Time in Patent Office: 1,849 Days
• Field of Search: 715/728, 715/847, 715/863, 715/866, 715/735, 715/747, 715/763, 715/765, 715/764, 715/737, 715/810, 345/173
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces

Abstract

Users are enabled to define and modify mappings between (1) gestures and (2) actions performed by one or more computing devices in response to a device detecting performance of a gesture. A generalized gesture-to-action mapping framework allows users to intuitively define and modify such mappings. In response to a device detecting the performance of a particular gesture, one or more devices may cause the execution of one or more device actions based on a set of user-defined gesture-to-action mappings.

Claims

1. A computer-implemented method comprising:

receiving, from a user via a user interface of a first device, mapping input that specifies a particular gesture, one or more first device actions to associate with the particular gesture, and a first device context to associate with the particular gesture and the one or more first device actions, wherein the first device context includes one or more devices, other than the first device, are in the presence of the first device, or the one or more devices have logged in to a particular user account;

storing, in a non-transitory storage medium, a first mapping that specifies the particular gesture, the first device context, and the one or more first device actions, wherein the non-transitory storage medium further stores a second mapping that specifies the particular gesture, a second device context, and one or more second device actions;

detecting, by the first device, performance of a given gesture;

in response to detecting performance of the given gesture;

determining that the given gesture matches the particular gesture that is specified in the first mapping and the second mapping;

in response to determining that the given gesture matches the particular gesture specified in the first mapping and the second mapping, reading the first mapping to determine the first device context that is specified in the first mapping and reading the second mapping to determine the second device context that is specified in the second mapping;

determining whether one or more conditions external to the first device, detected by the first device, match the first device context;

responsive to determining that the one or more conditions external to the first device match the first device context, causing the first device to perform the one or more first device actions that are specified in the first mapping;

determining whether the one or more conditions external to the first device, detected by the first device, match the second device context; and

responsive to determining that the one or more conditions external to the first device match the second device context, causing the first device to perform the one or more second device actions that are specified in the second mapping.

2. The computer-implemented method of claim 1, wherein the particular gesture is a defined input event and includes one or more of:

a motion detected by a touch-enabled interface associated with the first device, a detected motion of the first device, a motion detected by a camera associated with the first device, a pattern of speech, or a sound.

3. The computer-implemented method of claim 1, wherein:

detecting, by the first device, performance of the given gesture includes detecting first input and the particular gesture is a first particular gesture;

the computer-implemented method further comprises;

receiving second input indicating that the first device detected a gesturing event that is one of the first particular gesture or a second particular gesture, and

determining, based on the first device context, that the gesturing event corresponds to the first particular gesture.

4. The computer-implemented method of claim 1, wherein the first mapping is associated with a gesture package comprising a stored representation of one or more user-defined mappings in a portable format.

5. The computer-implemented method of claim 1, wherein the first device context is based on one or more devices other than the first device detectable by the first device.

6. The computer-implemented method of claim 5, wherein the first device context is further based on one or more characteristics of the one or more devices other than the first device detectable by the first device.

7. The computer-implemented method of claim 1, wherein the one or more conditions external to the first device include that the one or more devices, other than the first device, are in the presence of the first device.

8. The computer-implemented method of claim 7, wherein the one or more conditions external to the first device include that the one or more devices, other than the first device, are logged in to the particular user account.

9. The computer-implemented method of claim 1, wherein the one or more conditions external to the first device includes whether a particular amount of ambient light is detected by the first device.

10. The computer-implemented method of claim 1, wherein the first device context further includes whether a particular ambient temperature is detected by the first device.

11. The computer-implemented method of claim 1, further comprising:

detecting, by a second device of the one or more devices other than the first device, performance of the given gesture at the second device;

in response to detecting performance of the given gesture by the second device;

accessing, by the second device, a device hierarchy establishing a precedence among the first device and the second device of the one or more devices other than the first device;

determining, based on the precedence of the second device relative to the first device in the device hierarchy, whether to cause the first device to perform the one or more first device actions.

12. The computer-implemented method of claim 11, wherein the device hierarchy is selected based on the first device context.

13. One or more non-transitory computer-readable storage media storing instructions which, when processed by one or more processors cause:

receiving, from a user via a user interface of a first device, mapping input that specifies a particular gesture, one or more first device actions to associate with the particular gesture, and a first device context to associate with the particular gesture and the one or more first device actions, wherein the first device context includes one or more devices, other than the first device, are in the presence of the first device, or the one or more devices have logged in to a particular user account;

storing, in a non-transitory storage medium, a first mapping that specifies the particular gesture, the first device context, and the one or more first device actions, wherein the non-transitory storage medium further stores a second mapping that specifies the particular gesture, a second device context, and one or more second device actions;

detecting, by the first device, performance of a given gesture;

in response to detecting performance of the given gesture;

determining that the given gesture matches the particular gesture that is specified in the first mapping and the second mapping;

in response to determining that the given gesture matches the particular gesture specified in the first mapping and the second mapping, reading the first mapping to determine the first device context that is specified in the first mapping and reading the second mapping to determine the second device context that is specified in the second mapping;

determining whether one or more conditions external to the first device, detected by the first device, match the first device context;

responsive to determining that the one or more conditions external to the first device match the first device context, causing the first device to perform the one or more first device actions that are specified in the first mapping;

determining whether the one or more conditions external to the first device, detected by the first device, match the second device context; and

responsive to determining that the one or more conditions external to the first device match the second device context, causing the first device to perform the one or more second device actions that are specified in the second mapping.

14. The one or more non-transitory computer-readable storage media of claim 13, wherein the particular gesture is a defined input event and includes one or more of:

a motion detected by a touch-enabled interface associated with the first device, a detected motion of the first device, a motion detected by a camera associated with the first device, or a pattern of speech, and a sound.

15. The one or more non-transitory computer-readable storage media of claim 13, wherein:

detecting, by the first device, performance of the given gesture includes detecting first input and the particular gesture is a first particular gesture;

the non-transitory computer-readable medium stores additional instructions which, when processed by the one or more processors, cause;

receiving second input indicating that the first device detected a gesturing event that is one of the first particular gesture or a second particular gesture, and

determining, based on the first device context, that the gesturing event corresponds to the first particular gesture.

16. The one or more non-transitory computer-readable storage media of claim 13, wherein the first mapping is associated with a gesture package comprising a stored representation of one or more user-defined mappings in a portable format.

17. The one or more non-transitory computer-readable storage media of claim 13, wherein the first device context is based on one or more devices other than the first device detectable by the first device.

18. The one or more non-transitory computer-readable storage media of claim 17, wherein the first device context is further based on one or more characteristics of the one or more devices other than the first device detectable by the first device.

19. The one or more non-transitory computer-readable storage media of claim 13, wherein the one or more conditions external to the first device include that the one or more devices, other than the first device, are in the presence of the first device.

20. The one or more non-transitory computer-readable storage media of claim 19, wherein the one or more conditions external to the first device include that the one or more devices, other than the first device, are logged in to the particular user account.

21. The one or more non-transitory computer-readable storage media of claim 13, wherein the one or more conditions external to the first device includes whether a particular amount of ambient light is detected by the first device.

22. The one or more non-transitory computer-readable storage media of claim 13, wherein the first device context further includes whether a particular ambient temperature is detected by the first device.

23. The one or more non-transitory computer-readable storage media of claim 13, further comprising additional instructions which, when processed by the one or more processors, cause:

detecting, by a second device of the one or more devices other than the first device, performance of the given gesture at the second device;

in response to detecting performance of the given gesture by the second device;

accessing, by the second device, a device hierarchy establishing a precedence among the first device and the second device of the one or more devices other than the first device;

determining, based on the precedence of the second device relative to the first device in the device hierarchy, whether to cause the first device to perform the one or more first device actions.

24. The one or more non-transitory computer-readable storage media of claim 23, wherein the device hierarchy is selected based on the first device context.

25. A computing apparatus comprising:

one or more processors, and

one or more non-transitory computer-readable storage media storing instructions which,

when processed by one or more processors cause;

receiving, from a user via a user interface of a first device, mapping input that specifies a particular gesture, one or more first device actions to associate with the particular gesture, and a first device context to associate with the particular gesture and the one or more first device actions, wherein the first device context includes one or more devices, other than the first device, are in the presence of the first device, or the one or more devices have logged in to a particular user account;

storing, in a non-transitory storage medium, a first mapping that specifies the particular gesture, the first device context, and the one or more first device actions, wherein the non-transitory storage medium further stores a second mapping that specifies the particular gesture, a second device context, and one or more second device actions;

detecting, by the first device, performance of a given gesture;

in response to detecting performance of the given gesture;

determining that the given gesture matches the particular gesture that is specified in the first mapping and the second mapping;

in response to determining that the given gesture matches the particular gesture specified in the first mapping and the second mapping, reading the first mapping to determine the first device context that is specified in the first mapping and reading the second mapping to determine the second device context that is specified in the second mapping;

determining whether one or more conditions external to the first device, detected by the first device, match the first device context;

responsive to determining that the one or more conditions external to the first device match the first device context, causing the first device to perform the one or more first device actions that are specified in the first mapping;

determining whether the one or more conditions external to the first device, detected by the first device, match the second device context; and

responsive to determining that the one or more conditions external to the first device match the second device context, causing the first device to perform the one or more second device actions that are specified in the second mapping.

26. The computing apparatus of claim 25, wherein the particular gesture is a defined input event and includes one or more of:

a motion detected by a touch-enabled interface associated with the first device, a detected motion of the first device, a motion detected by a camera associated with the first device, or a pattern of speech, and a sound.

27. The computing apparatus of claim 25, wherein:

detecting, by the first device, performance of the given gesture includes detecting first input and the particular gesture is a first particular gesture;

the non-transitory computer-readable medium stores additional instructions which, when processed by the one or more processors, cause;

receiving second input indicating that the first device detected a gesturing event that is one of the first particular gesture or a second particular gesture, and

determining, based on the first device context, that the gesturing event corresponds to the first particular gesture.

28. The computing apparatus of claim 25, wherein the first mapping is associated with a gesture package comprising a stored representation of one or more user-defined mappings in a portable format.

29. The computing apparatus of claim 25, wherein the first device context is based on one or more devices other than the first device detectable by the first device.

30. The computing apparatus of claim 29, wherein the first device context is further based on one or more characteristics of the one or more devices other than the first device detectable by the first device.

31. The computing apparatus of claim 25, wherein the one or more conditions external to the first device include that the one or more devices, other than the first device, are in the presence of the first device.

32. The computing apparatus of claim 31, wherein the one or more conditions external to the first device include that the one or more devices, other than the first device, are logged in to the particular user account.

33. The computing apparatus of claim 25, wherein the one or more conditions external to the first device includes whether a particular amount of ambient light is detected by the first device.

34. The computing apparatus of claim 25, wherein the first device context further includes whether a particular ambient temperature is detected by the first device.

35. The computing apparatus of claim 25, wherein the one or more non-transitory computer-readable storage media store additional instructions which, when processed by the one or more processors, cause:

detecting, by a second device of the one or more devices other than the first device, performance of the given gesture at the second device;

in response to detecting performance of the given gesture by the second device;

accessing, by the second device, a device hierarchy establishing a precedence among the first device and the second device of the one or more devices other than the first device;

determining, based on the precedence of the second device relative to the first device in the device hierarchy, whether to cause the first device to perform the one or more first device actions.

36. The computing apparatus of claim 35, wherein the device hierarchy is selected based on the first device context.