Collaborative gesture-based input language

• US 8,196,066 B1
• Filed: 09/30/2011
• Issued: 06/05/2012
• Est. Priority Date: 09/20/2011
• Status: Expired due to Fees

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Li, Yang, Ouyang, Yu
• Primary Examiner(s): Roswell, Michael
• Application Number:
  US13/250,848
• Time in Patent Office: 249 Days
• Field of Search: 715/863, 715/811, 715/745, 715/812, 715/708
• US Class Current: 715/863
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

In one example, a method includes receiving, by a server and from a plurality of computing devices, data representative of a group of gesture-shortcut pairs that each include a gesture having been detected by at least one computing device from the plurality of computing devices and a shortcut associated with the detected gesture, and wherein the shortcut corresponds to an operation to be executed by the at least one from the plurality of computing devices. The method includes sorting the data representative of each respective gesture-shortcut pair from the group of gesture-shortcut pairs into at least two subgroups, a first subgroup including the first gesture-shortcut pair and a second including the second gesture-shortcut pair, the sorting being based on detected similarities between at least one of each respective gesture from the group of gesture-shortcut pairs and each respective shortcut from the group of gesture-shortcut pairs.

Claims

1. A method comprising:

receiving, by a first computing device, data representative of a gesture detected by a presence-sensitive screen of the computing device;

receiving, by the first computing device and from a server, an aggregated group of gesture-shortcut associations,

wherein the aggregated group of gesture-shortcut associations includes a plurality of gesture-shortcut associations received by the server from at least a second computing device, and

wherein the aggregated group of gesture-shortcut associations is aggregated based at least in part on similarities between at least one of

     1) each respective gesture from the aggregated group of gesture-shortcut associations, and

     2) each respective shortcut from the aggregated group of gesture-shortcut associations;

identifying, by the first computing device, a shortcut associated with the detected gesture,

wherein the shortcut corresponds to an action to be performed by the first computing device, and

wherein identifying the shortcut comprises accessing at least a portion of the aggregated group of gesture-shortcut associations, the portion of the aggregated group of gesture-shortcut associations having been determined based at least in part upon prior user input received from the at least one other computing device; and

providing for display, by the computing device, data representative of the shortcut.

2. The method of claim 1, wherein identifying the shortcut associated with the gesture further comprises:

identifying, by the computing device, a letter associated with at least a portion of the gesture; and

identifying the shortcut associated with the gesture based at least in part on the identified letter.

3. The method of claim 1, wherein identifying the shortcut associated with the gesture further comprises:

identifying a first shortcut associated with the gesture based at least in part on detected similarities between the gesture and images of a group of image-shortcut associations;

identifying a second shortcut associated with the gesture using handwriting recognition to identify at least one letter associated with the gesture; and

selecting one of the first shortcut and the second shortcut as the identified shortcut.

4. The method of claim 1, wherein identifying the shortcut associated with the gesture comprises:

sending, by the first computing device, the data representative of the gesture to the server; and

receiving, by the first computing device, an indication of the shortcut associated with the gesture from the server.

5. The method of claim 1,

wherein the portion of the aggregated group of gesture-shortcut associations determined based at least in part upon prior user input from the at least one other user comprises a first portion of the aggregated group of gesture-shortcut associations,

wherein at least a second portion of the aggregated group of gesture-shortcut associations is stored on the computing device, and

wherein identifying the shortcut associated with the detected gesture further comprises:

accessing the second portion of the aggregated group of gesture-shortcut associations stored on the computing device to identify a first candidate shortcut;

sending, by the computing device, the data representative of the gesture to the server;

receiving, by the computing device, an indication of a second candidate shortcut associated with the gesture from the server; and

selecting one of the first candidate shortcut and the second candidate shortcut as the shortcut associated with the gesture.

6. The method of claim 5, further comprising:

selecting the first candidate shortcut as the shortcut associated with the gesture when the association between the gesture and the first candidate shortcut is configured by a user of the computing device.

7. The method of claim 1,

wherein the aggregated group of gesture-shortcut associations includes a plurality of gesture clusters, wherein each gesture cluster from the plurality of gesture clusters includes a first plurality of gestures and a second plurality of gestures, each gesture from the first plurality of gestures determined to have a first set of features that is less than a first threshold amount different than a second set of features of the second plurality of and

wherein the first plurality of gestures are organized in a hierarchy such that a second threshold amount difference between a third set of features of a third set of gestures included a child gesture cluster of a respective gesture cluster from the plurality of gesture clusters is less than the first threshold amount difference between the first set of features of gestures included in the respective gesture cluster.

8. The method of claim 7,

wherein the gesture includes a third set of features that define a shape of the gesture, and

wherein identifying the shortcut associated with the gesture further comprises:

determining whether the third set of features is less than a third threshold amount different than a fourth set of features of the respective gesture cluster of the plurality of gesture clusters.

9. The method of claim 8, further comprising:

when the third set of features are less than the threshold amount different than the fourth set of features;

responsive to determining that the respective gesture cluster has at least one child gesture cluster, recursively selecting a child gesture cluster of the at least one child cluster of the respective gesture cluster, and determining whether the third set of features of the gesture are less than a third threshold amount different than a fifth set first group of features of the child gesture cluster of the at least one child cluster of the respective gesture cluster;

and

responsive to determining that the respective gesture cluster does not have at least one child gesture cluster, identifying shortcuts associated with gestures included in the respective gesture cluster based at least in part on a mapping of each of the gestures included in the respective gesture cluster to a respective shortcut.

10. The method of claim 8, further comprising:

when the third set of features are not less than the first threshold amount different than the fourth set of features;

responsive to determining that the respective gesture cluster has at least one sibling gesture cluster previously unanalyzed, recursively selecting a sibling gesture cluster from the at least one sibling gesture cluster, and determining whether the third set first group of features of the gesture is less than a third threshold amount different than a first set of features of the sibling gesture cluster;

responsive to determining that all of the sibling gestures of the respective gesture cluster were previously analyzed and at least one parent gesture cluster of the respective gesture has not been previously analyzed, selecting one of the parent gesture clusters that has not been previously analyzed, and analyzing the selected parent gesture cluster of the respective gesture cluster; and

responsive to determining that all of the sibling gestures of the respective gesture cluster does were previously analyzed and all of the parent gesture clusters of the respective gesture have been previously analyzed, identifying shortcuts associated with gestures included in the parent gesture cluster based at least in part on a mapping of each of the gestures included in the parent gesture cluster to a shortcut.

11. The method of claim 1, further comprising:

associating, by the computing device, a different shortcut with the gesture based at least in part on user input received by the first computing device; and

storing an indication of the association between the different shortcut and the gesture in a data repository of the first computing device.

12. The method of claim 11, further comprising:

sending, by the computing device, an indication of the association between the different shortcut and the gesture to the server.

13. The method of claim 1, further comprising:

receiving, by the first computing device, at least one text-based search term;

generating a query based at least in part on the at least one text-based search term;

executing the query to retrieve data associated with a group of shortcuts and associated gestures from a data repository of the first computing device; and

outputting, at the presence-sensitive screen of the first computing device, a representation of the retrieved group of shortcuts and associated gestures.

14. The method of claim 1, further comprising:

when no shortcuts are associated with the gesture, receiving, by the first computing device, a selection of the action to be performed by the computing device; and

storing an association between the gesture and a shortcut to the selected action.

15. A method comprising:

receiving, by a server and from a plurality of computing devices, data representative of a group of gesture-shortcut associations, a first gesture-shortcut association from the plurality of gesture-shortcut associations being different from a second gesture-shortcut association from the plurality of gesture-shortcut associations, wherein each gesture-shortcut association from the group of gesture-shortcut associations pairs includes a respective gesture having been detected by at least one computing device from the plurality of computing devices and a respective shortcut associated with the respective gesture, and wherein the respective shortcut corresponds to an operation to be executed by the at least one computing device from the plurality of computing devices;

sorting, by the server, the data representative of each respective gesture-shortcut association from the group of gesture-shortcut associations into at least two subgroups, a first subgroup from the at least two subgroups including the first gesture-shortcut association and a second subgroup from the at least two subgroups including the second gesture-shortcut association, the sorting being based at least in part on detected similarities between at least one of

     1) each respective gesture from the group of gesture-shortcut associations and

     2) each respective shortcut from the group of gesture-shortcut associations;

receiving, by the server, data representative of a gesture detected at a presence-sensitive screen of a computing device from one of the plurality of computing devices;

comparing the data representative of the detected gesture to data representative of a gesture grouping, wherein each respective sorted subgroup of gesture-shortcut associations from the at least two sorted subgroups of gesture-shortcut associations includes a respective generational hierarchy of parent gesture groups and child gesture groups formed by sorting gestures from the respective subgroup of gesture-shortcut associations based at least in part on detected similarities between the gestures from the respective subgroup of gesture-shortcut associations;

identifying, by the server, a shortcut associated with the gesture based at least in part on the comparing;

transmitting data representative of the identified shortcut to the computing device.

16. The method of claim 15, further comprising:

sending a portion of the data representative of the group of gesture-shortcut associations to the at least one computing device.

17. The method of claim 15, wherein identifying the shortcut associated with the gesture comprises:

selecting a predefined number of child gesture groups based at least in part on detected similarities between gestures included in each respective child gesture group of the selected child gesture groups and the detected gesture;

repeating the selecting of child gesture groups for each generation of gesture groups in the gesture grouping until only gestures matching the detected gesture within a predefined threshold are selected; and

identifying, by the server, a shortcut associated with a selected gesture that matches the detected gesture within the predefined threshold.

18. The method of claim 15, further comprising:

receiving, by the server, a text-based search term representative of a first shortcut from a computing device of the plurality of computing devices;

identifying, by the server, one or more gestures associated with the first shortcut by at least;

accessing the group of gesture-shortcut associations;

identifying a second shortcut of the group of gesture-shortcut associations that matches the first shortcut represented by the text-based search term; and

identifying the one or more gestures associated with the second shortcut; and

transmitting, to the computing device, data representative of the one or more identified gestures associated with the first shortcut.

19. The method of claim 15, further comprising:

determining, by the server, a frequency at which a particular gesture is grouped with a particular shortcut in the group of gesture-shortcut associations; and

removing a gesture-shortcut association from the group of gesture-shortcut associations when the frequency of the groupings between the particular gesture and the particular shortcut is below a threshold value.

20. The method of claim 15, further comprising:

determining, by the server, an amount of time that has elapsed between receiving an indication of a particular gesture-shortcut association from the group of gesture-shortcut associations; and

removing the particular gesture-shortcut association from the group of gesture-shortcut associations in response to determining that the amount of elapsed time exceeds a threshold amount of time.

21. A non-transitory computer-readable storage medium encoded with instructions that, when executed, cause one or more processors of a computing device to perform operations comprising:

receiving, based on input of a first user, data representative of a gesture detected by a presence-sensitive screen of the computing device;

receiving an aggregated group of graphical representation-shortcut associations from a server,

wherein the aggregated group of graphical representation-shortcut associations includes a plurality of graphical representation-shortcut associations received by the server from at least a second user, and

wherein the aggregated group of graphical representation-shortcut associations is aggregated based at least in part on similarities between at least one of

     1) each respective graphical representation from the aggregated group of graphical representation-shortcut associations, and

     2) each respective shortcut from the aggregated group of graphical representation-shortcut associations;

identifying a first shortcut associated with the gesture based at least in part on detected similarities between the gesture and graphical representations of at least a portion of an aggregated group of graphical representation-shortcut associations, the portion of the aggregated group of graphical representation-shortcut associations having been determined based at least in part on prior user input from at least the second user;

identifying a second shortcut associated with the gesture, wherein identifying the second shortcut comprises;

using at least one handwriting recognition operation to identify at least one letter associated with the gesture; and

identifying the second shortcut by at least accessing a portion of an aggregated group of letter-shortcut associations determined based upon prior user input from at least a third user;

selecting one of the first shortcut and the second shortcut as an identified shortcut, wherein the identified shortcut corresponds to an action to be performed by the computing device; and

providing for display data representative of the identified shortcut.

22. The computer-readable storage medium of claim 21, wherein at least two of a) the first user, b) the second user, and c) the third user are the same user.

23. A device comprising:

at least one processor;

a network interface; and

a language development module,

wherein the network interface is configured to

receive data representative of a group of gestures, each gesture from the group of gestures having been detected by at least one computing device from a plurality of computing devices, and receive data representative of one or more shortcuts, each shortcut from the one or more shortcuts being associated with at least one gesture from the group of gestures, wherein each of the one or more shortcuts corresponds to an operation to be executed by at least one of the plurality of computing devices,

receive data representative of a gesture detected at a presence-sensitive screen of a computing device from one of the plurality of computing devices, wherein the language development module is operable by the at least one processor to;

sort the data representative of the group of gestures and the data representative of the associated shortcuts received from the plurality of computing devices into a plurality gesture-shortcut groups, the sorting being based at least in part on detected similarities between at least one of

     1) each gesture from the group of gestures and

     2) each shortcut from the one or more shortcuts, wherein each gesture-shortcut group from the plurality of gesture-shortcut groups includes either a) data representative of a group of gesture-shortcut associations sorted based at least in part on detected similarities between each gesture included in the respective gesture-shortcut group or b) data representative of a group of gesture-shortcut associations sorted based at least in part on detected similarities between each shortcut included in the respective gesture-shortcut group,

receive data representative of a gesture detected at a presence-sensitive screen of a computing device from one of the plurality of computing devices,

compare the data representative of the detected gesture to data representative of a gesture grouping, wherein each respective sorted subgroup of gesture-shortcut associations from the at least two sorted subgroups of gesture-shortcut associations includes a respective generational hierarchy of parent gesture groups and child gesture groups formed by sorting gestures from the respective subgroup of gesture-shortcut associations based at least in part on detected similarities between the gestures from the respective subgroup of gesture-shortcut associations,

identify a shortcut associated with the gesture based at least in part on the comparing, and

wherein the network interface is further configured to transmit data representative of the identified shortcut to the computing device.