COLLABORATIVE GESTURE-BASED INPUT LANGUAGE

• US 20130074014A1
• Filed: 09/20/2011
• Published: 03/21/2013
• Est. Priority Date: 09/20/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Ouyang, Yu, Li, Yang
• Granted Patent: US 8,751,972 B2
• Time in Patent Office: Days
• US Class Current: 715/863
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

In one example, a method includes receiving, by a server, data representative of a group of gestures detected by the plurality of computing devices and data representative of one or more shortcuts associated with the group of gestures from a plurality of computing devices, wherein each shortcut corresponds to an action performed by at least one of the computing devices. The method may further include aggregating, by the server, the data representative of the gestures and the data representative of the associated shortcuts received from the plurality of computing devices based at least in part on detected similarities between at least one of 1) the group of gestures and 2) the associated shortcuts, and defining, by the server, a gesture-shortcut language based at least in part on the aggregated data.

Claims

1. A method comprising:

receiving, by a computing device, data representative of a gesture detected by a presence-sensitive screen of the computing device;

identifying, by the computing device, a shortcut associated with the gesture, wherein the shortcut corresponds to an action to be performed by the computing device, and wherein identifying the shortcut comprises accessing at least a portion of an aggregated group of gesture-shortcut associations determined based at least in part upon prior user input from at least one other user; and

providing for display, by the computing device, data representative of the shortcut.

2. The method of claim 1, wherein identifying the shortcut associated with the gesture further comprises:

identifying, by the computing device, a letter associated with at least a portion of the gesture; and

identifying the shortcut associated with the gesture based at least in part on the identified letter.

3. The method of claim 1, wherein identifying the shortcut associated with the gesture further comprises:

identifying a first shortcut associated with the gesture based at least in part on detected similarities between the gesture and images of a group of image-shortcut associations;

identifying a second shortcut associated with the gesture using handwriting recognition to identify at least one letter associated with the gesture; and

selecting one of the first shortcut and the second shortcut as the identified shortcut.

4. The method of claim 1, wherein identifying the shortcut associated with the gesture comprises:

sending, by the computing device, the data representative of the gesture to a server; and

receiving, by the computing device, an indication of the shortcut associated with the gesture from the server.

5. The method of claim 1,

wherein at least a portion of the aggregated group of gesture-shortcut associations is stored on the computing device, and

wherein identifying the shortcut associated with the detected gesture further comprises:

accessing the portion of the aggregated group of gesture-shortcut associations stored on the computing device to identify a first candidate shortcut;

sending, by the computing device, the data representative of the gesture to a server;

receiving, by the computing device, an indication of a second candidate shortcut associated with the gesture from the server; and

selecting one of the first candidate shortcut and the second candidate shortcut as the shortcut associated with the gesture.

6. The method of claim 5, further comprising:

selecting the first candidate shortcut as the shortcut associated with the gesture when the association between the gesture and the first candidate shortcut is configured by a user of the computing device.

7. The method of claim 1,

wherein the aggregated group of gesture-shortcut associations includes a plurality of gesture clusters that each include a plurality of gestures each determined to have features that are less than a threshold amount different than the features of other gestures in the respective gesture cluster, and

wherein the plurality of gesture clusters are organized in a hierarchy such that another threshold amount difference between features of the gestures included in the gesture cluster for a child gesture cluster is less than the threshold amount difference between features of the gestures included in the gesture cluster of the respective gesture cluster.

8. The method of claim 7,

wherein the gesture includes a first group of features that define a shape of the gesture, and

wherein identifying the shortcut associated with the gesture further comprises:

determining whether the first group of features of the gesture are less than a threshold amount different than a second group of features of a respective gesture cluster of the plurality of gesture clusters.

9. The method of claim 8, further comprising:

when the first group of features are less than the threshold amount different than the second group of features;

responsive to determining that the respective gesture cluster has at least one child gesture cluster, recursively selecting a child gesture cluster of the at least one child cluster of the respective gesture cluster, and determining whether the first group of features of the gesture are less than another threshold amount different than a third group of features of the child gesture cluster; and

responsive to determining that the respective gesture cluster does not have at least one child gesture cluster, identifying shortcuts associated with gestures included in the respective gesture cluster based at least in part on a mapping of each of the gestures included in the respective gesture cluster to a respective shortcut.

10. The method of claim 8, further comprising:

when the first group of features are not less than the threshold amount different than the second group of features;

responsive to determining that the respective gesture cluster has at least one sibling gesture cluster previously unanalyzed, recursively selecting a sibling gesture cluster of the at least one sibling gesture cluster, and determining whether the first group of features of the gesture are less than another threshold amount different than a group of features of the sibling gesture cluster;

responsive to determining that all of the sibling gestures of the respective gesture cluster were previously analyzed and at least one parent gesture cluster of the respective gesture has not been previously analyzed, selecting one of the parent gesture clusters that has not been previously analyzed, and analyzing the selected parent gesture cluster of the respective gesture cluster; and

responsive to determining that all of the sibling gestures of the respective gesture cluster does were previously analyzed and all of the parent gesture clusters of the respective gesture have been previously analyzed, identifying shortcuts associated with gestures included in the parent gesture cluster based at least in part on a mapping of each of the gestures included in the parent gesture cluster to a shortcut.

11. The method of claim 1, further comprising:

associating, by the computing device, a different shortcut with the gesture based at least in part on user input received by the computing device; and

storing an indication of the association between the different shortcut and the gesture in a data repository of the computing device.

12. The method of claim 11, further comprising:

sending, by the computing device, an indication of the association between the different shortcut and the gesture to a server.

13. The method of claim 1, further comprising:

receiving, by the computing device, at least one text-based search term;

generating a query based at least in part on the at least one text-based search term;

executing the query to retrieve data associated with a group of shortcuts and associated gestures from a data repository of the computing device; and

outputting, using the presence-sensitive screen of the computing device, a representation of the retrieved group of shortcuts and associated gestures.

14. The method of claim 1, further comprising:

when no shortcuts are associated with the gesture, receiving, by the computing device, a selection of the action to be performed by the computing device; and

storing an association between the gesture and a shortcut to the selected action.

15. A method comprising:

receiving, by a server and from a plurality of computing devices, data representative of a group of gestures detected by the plurality of computing devices;

receiving, by the server, data representative of one or more shortcuts associated with the group of gestures from the plurality of computing devices, wherein each of the shortcuts corresponds to an action performed by at least one of the plurality of computing devices;

aggregating, by the server, the data representative of the group of gestures and the data representative of the associated shortcuts received from the plurality of computing devices based at least in part on detected similarities between at least one of

     1) the group of gestures and

     2) the associated shortcuts; and

defining, by the server, a gesture-shortcut language based at least in part on the aggregated data, wherein the gesture-shortcut language includes at least a portion of the aggregated data representative of the group of gestures and associated shortcuts.

16. The method of claim 15, further comprising:

sending a portion of the gesture-shortcut language based at least in part on the aggregated data to at least one computing device of the plurality of computing devices.

17. The method of claim 15, further comprising:

storing data representative of the group of gestures, the shortcuts, and a set of gesture-shortcut associations in a data repository of the server; and

transmitting, by the server, at least a portion of the data representative of the gesture-shortcut associations to a computing device of the plurality of computing devices.

18. The method of claim 15, further comprising:

receiving, by the server, data representative of a gesture detected on a presence-sensitive screen of a computing device from one of the plurality of computing devices;

identifying, by the server, a shortcut associated with the gesture based at least in part on the received data; and

transmitting data representative of the shortcut associated with the identified gesture to the computing device.

19. The method of claim 18, wherein identifying the shortcut associated with the gesture comprises:

comparing the data representative of the detected gesture to data representative of a gesture cluster, wherein the gesture cluster comprises a generational hierarchy of parent and child gesture clusters formed by grouping gestures and gesture clusters with similar features;

selecting a predefined number of child gesture clusters similar to the detected gesture;

repeating the selection of child gesture clusters for each generation of gesture clusters until only gestures matching the detected gesture within a predefined tolerance are selected; and

identifying shortcuts associated with the selected gestures via the gesture-shortcut associations of the gesture-based language.

20. The method of claim 15, further comprising:

receiving a text-based search term representative of a shortcut from a computing device;

identifying one or more gestures associated with the shortcut based at least in part on a gesture-shortcut association of the gesture-shortcut language; and

transmitting, to the computing device, data representative of the one or more identified gestures associated with the shortcut.

21. The method of claim 15, wherein defining the gesture-shortcut language based at least in part on the aggregated data comprises:

determining a frequency at which a particular gesture is associated with a particular shortcut; and

deleting associations of the particular gesture with the particular shortcut when a number of associations between the particular gesture and the particular shortcut are below a threshold number of associations.

22. The method of claim 15, wherein defining the gesture-shortcut language based at least in part on the aggregated data comprises:

determining an amount of time that has elapsed between receiving an indication of an association of a particular gesture with a particular shortcut; and

deleting the associations of the particular gesture with the particular shortcut upon determining that the amount of time exceeds a threshold amount of time.

23. A computer-readable storage medium encoded with instructions that, when executed, cause one or more processors of a computing device to perform operations comprising:

receiving data representative of a gesture detected by a presence-sensitive screen of the computing device;

identifying a shortcut associated with the gesture, wherein the shortcut corresponds to an action to be performed by the computing device, and wherein identifying the shortcut comprises accessing at least a portion of an aggregated group of gesture-shortcut associations determined based upon prior user input from at least one other user; and

providing for display data representative of the shortcut.

24. A computer-readable storage medium encoded with instructions that, when executed, cause one or more processors of a computing device to perform operations comprising:

receiving data representative of group of gestures detected by a plurality of computing devices;

receiving data representative of one or more shortcuts associated with the group of gestures, wherein each of the shortcuts corresponds to an action performed by at least one of the plurality of computing devices;

aggregating the data representative of the group of gestures and associated shortcuts received from the plurality of computing devices based on detected similarities between at least one of

     1) the group of gestures and

     2) the associated shortcuts; and

defining a gesture-shortcut language based on the aggregated data, wherein the gesture-shortcut language includes at least a portion of the aggregated data representative of the group of gestures and associated shortcuts.

25. A device comprising:

at least one processor;

a network interface configured to receive data representative of a group of gestures detected by a plurality of computing devices and receive data representative of one or more shortcuts associated with the group of gestures, wherein each of the shortcuts corresponds to an action performed by at least one of the plurality of computing devices; and

a language development module operable by the at least one processor to aggregate the data representative of the group of gestures and the data representative of the associated shortcuts received from the plurality of computing devices based at least in part on detected similarities between at least one of

     1) the group of gestures and

     2) the associated shortcuts, and define a gesture-shortcut language based at least in part on the aggregated data, wherein the gesture-shortcut language includes at least a portion of the aggregated data representative of the group of gestures and associated shortcuts.