Collaborative gesture-based input language

• US 8,751,972 B2
• Filed: 09/20/2011
• Issued: 06/10/2014
• Est. Priority Date: 09/20/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Ouyang, Yu, Li, Yang
• Primary Examiner(s): ALVESTEFFER, STEPHEN D
• Application Number:
  US13/237,862
• Publication Number:
  US 20130074014A1
• Time in Patent Office: 994 Days
• Field of Search: None
• US Class Current: 715/863
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

In one example, a method includes receiving, by a server, data representative of a group of gestures detected by the plurality of computing devices and data representative of one or more shortcuts associated with the group of gestures from a plurality of computing devices, wherein each shortcut corresponds to an action performed by at least one of the computing devices. The method may further include aggregating, by the server, the data representative of the gestures and the data representative of the associated shortcuts received from the plurality of computing devices based at least in part on detected similarities between at least one of 1) the group of gestures and 2) the associated shortcuts, and defining, by the server, a gesture-shortcut language based at least in part on the aggregated data.

Claims

1. A method comprising:

storing, by the computing device and in a data repository of the computing device, at least a portion of an aggregated group of gesture-shortcut associations;

receiving, by a computing device, data representative of a gesture detected by a presence-sensitive screen of the computing device;

identifying, by the computing device and based at least in part on the data representative of the gesture, a first candidate shortcut by at least accessing the portion of the aggregated group of gesture-shortcut associations stored in the data repository;

responsive to sending, by the computing device, the data representative of the gesture to a server, receiving, by the computing device and from the server, an indication of a second candidate shortcut associated with the gesture;

selecting one of the first candidate shortcut and the second candidate shortcut as a shortcut associated with the gesture, wherein the shortcut associated with the gesture corresponds to an action to be performed by the computing device; and

outputting, by the computing device, for display, data representative of the shortcut associated with the gesture.

2. The method of claim 1, wherein identifying the first candidate shortcut comprises:

identifying, by the computing device, a letter associated with at least a portion of the gesture; and

identifying, based at least in part on the letter, the first candidate shortcut.

3. The method of claim 1, wherein identifying the first candidate shortcut further comprises:

identifying a first shortcut associated with the gesture based at least in part on detected similarities between the gesture and images of a group of image-shortcut associations;

identifying a second shortcut associated with the gesture using handwriting recognition to identify at least one letter associated with the gesture; and

selecting one of the first shortcut and the second shortcut as the first candidate shortcut.

4. The method of claim 1, further comprising:

selecting the first candidate shortcut as the shortcut associated with the gesture when the association between the gesture and the first candidate shortcut is configured by a user of the computing device.

5. The method of claim 1,

wherein the aggregated group of gesture-shortcut associations includes a plurality of gesture clusters that each include a plurality of gestures each determined to have features that are less than a threshold amount different than the features of other gestures in the respective gesture cluster, and

wherein the plurality of gesture clusters are organized in a hierarchy such that another threshold amount difference between features of the gestures included in the gesture cluster for a child gesture cluster is less than the threshold amount difference between features of the gestures included in the gesture cluster of the respective gesture cluster.

6. The method of claim 5,

wherein the gesture includes a first group of features that define a shape of the gesture, and

wherein identifying the first candidate shortcut further comprises:

determining whether the first group of features of the gesture is less than a threshold amount different than a second group of features of a respective gesture cluster of the plurality of gesture clusters.

7. The method of claim 6, further comprising:

responsive to determining that the first group of features is less than the threshold amount different than the second group of features;

responsive to determining that the respective gesture cluster has at least one child gesture cluster, recursively selecting a child gesture cluster of the at least one child cluster of the respective gesture cluster, and determining whether the first group of features of the gesture are less than another threshold amount different than a third group of features of the child gesture cluster; and

responsive to determining that the respective gesture cluster does not have at least one child gesture cluster, identifying shortcuts associated with gestures included in the respective gesture cluster based at least in part on a mapping of each of the gestures included in the respective gesture cluster to a respective shortcut.

8. The method of claim 6, further comprising:

responsive to determining that the first group of features is not less than the threshold amount different than the second group of features;

responsive to determining that the respective gesture cluster has at least one sibling gesture cluster previously unanalyzed, recursively selecting a sibling gesture cluster of the at least one sibling gesture cluster, and determining whether the first group of features of the gesture are less than another threshold amount different than a group of features of the sibling gesture cluster;

responsive to determining that all of the sibling gestures of the respective gesture cluster were previously analyzed and at least one parent gesture cluster of the respective gesture has not been previously analyzed, selecting one of the parent gesture clusters that has not been previously analyzed, and analyzing the selected parent gesture cluster of the respective gesture cluster; and

responsive to determining that all of the sibling gestures of the respective gesture cluster does were previously analyzed and all of the parent gesture clusters of the respective gesture have been previously analyzed, identifying shortcuts associated with gestures included in the parent gesture cluster based at least in part on a mapping of each of the gestures included in the parent gesture cluster to a shortcut.

9. The method of claim 1, further comprising:

associating, by the computing device and based at least in part on user input received by the computing device, a different shortcut with the gesture; and

storing an indication of the association between the different shortcut and the gesture in the data repository of the computing device.

10. The method of claim 9, further comprising:

sending, by the computing device, an indication of the association between the different shortcut and the gesture to the server.

11. The method of claim 1, further comprising:

receiving, by the computing device, at least one text-based search term;

generating a query based at least in part on the at least one text-based search term;

executing the query to retrieve data associated with a group of shortcuts and associated gestures from the data repository of the computing device; and

outputting, by the computing device, for display, a representation of the retrieved group of shortcuts and associated gestures.

12. The method of claim 1, further comprising:

when no shortcuts are associated with the gesture, receiving, by the computing device, a selection of the action to be performed by the computing device; and

storing an association between the gesture and a shortcut to the selected action.

13. A method comprising:

receiving, by a server and from a plurality of computing devices, data representative of a group of gestures detected by the plurality of computing devices;

receiving, by the server, data representative of one or more shortcuts associated with the group of gestures from the plurality of computing devices, wherein each of the shortcuts corresponds to an action performed by at least one of the plurality of computing devices;

aggregating, by the server and based at least in part on detected similarities between at least one of

     1) the group of gestures and

     2) the one or more shortcuts associated with the group of gestures, the data representative of the group of gestures and the data representative of the associated shortcuts received from the plurality of computing devices into a plurality of gesture clusters organized into a generational hierarchy of parent and child gesture clusters such that each child gesture cluster of the generational hierarchy includes a plurality of gestures each determined to have features that are less than a first threshold amount different than corresponding features of gestures in a corresponding parent gesture cluster of the generational hierarchy;

defining, by the server, a gesture-shortcut language based at least in part on the aggregated data, wherein the gesture-shortcut language includes at least a portion of the aggregated data representative of the group of gestures and associated shortcuts;

receiving, by the server, data representative of a gesture detected at a presence-sensitive screen of a computing device from one of the plurality of computing devices;

comparing the data representative of the detected gesture to data representative of a parent gesture cluster from the plurality of gesture clusters;

repeating a selection of a child gesture cluster for each generation of gesture clusters until features of gestures included in the respective child gesture cluster are less than a second threshold amount different from corresponding features of the detected gesture;

identifying at least one shortcut associated with the selected child gesture cluster; and

transmitting data representative of the at least one shortcut associated with the selected child gesture cluster to the computing device.

14. The method of claim 13, further comprising:

sending a portion of the gesture-shortcut language based at least in part on the aggregated data to at least one computing device of the plurality of computing devices.

15. The method of claim 13, further comprising:

storing data representative of the group of gestures, the shortcuts, and a set of gesture-shortcut associations in a data repository of the server; and

transmitting, by the server, at least a portion of the data representative of the gesture-shortcut associations to a computing device of the plurality of computing devices.

16. The method of claim 13, further comprising:

receiving a text-based search term representative of a shortcut from a computing device;

identifying one or more gestures associated with the shortcut based at least in part on a gesture-shortcut association of the gesture-shortcut language; and

transmitting, to the computing device, data representative of the one or more identified gestures associated with the shortcut.

17. The method of claim 13, wherein defining the gesture-shortcut language based at least in part on the aggregated data comprises:

determining a frequency at which a particular gesture is associated with a particular shortcut; and

deleting associations of the particular gesture with the particular shortcut when a number of associations between the particular gesture and the particular shortcut are below a threshold number of associations.

18. The method of claim 13, wherein defining the gesture-shortcut language based at least in part on the aggregated data comprises:

determining an amount of time that has elapsed between receiving an indication of an association of a particular gesture with a particular shortcut; and

deleting the associations of the particular gesture with the particular shortcut upon determining that the amount of time exceeds a threshold amount of time.

19. A non-transitory computer-readable storage medium encoded with instructions that, when executed, cause one or more processors of a computing device to perform operations comprising:

storing, by the computing device and in a data repository of the computing device, at least a portion of an aggregated group of gesture-shortcut associations;

receiving data representative of a gesture detected by a presence-sensitive screen of the computing device;

identifying, by the computing device and based at least in part on the data representative of the gesture, a first candidate shortcut by at least accessing the portion of the aggregated group of gesture-shortcut associations stored in the data repository;

sending, by the computing device, the data representative of the gesture to a server;

receiving, by the computing device and from the server, an indication of a second candidate shortcut associated with the gesture;

selecting one of the first candidate shortcut and the second candidate shortcut as a shortcut associated with the gesture, wherein the shortcut associated with the gesture corresponds to an action to be performed by the computing device; and

outputting, by the computing device, for display, data representative of the shortcut associated with the gesture.

20. A non-transitory computer-readable storage medium encoded with instructions that, when executed, cause one or more processors of a computing device to perform operations comprising:

receiving data representative of group of gestures detected by a plurality of computing devices;

receiving data representative of one or more shortcuts associated with the group of gestures, wherein each of the shortcuts corresponds to an action performed by at least one of the plurality of computing devices;

aggregating, based at least in part on detected similarities between at least one of

     1) the group of gestures and

     2) the one or more shortcuts associated with the group of gestures, the data representative of the group of gestures and the data representative of the associated shortcuts received from the plurality of computing devices into a plurality of gesture clusters organized into a generational hierarchy of parent and child gesture clusters such that each child gesture cluster of the generational hierarchy includes a plurality of gestures each determined to have features that are less than a first threshold amount different than corresponding features of gestures in a corresponding parent gesture cluster of the generational hierarchy;

receiving data representative of a gesture detected at a presence-sensitive screen of a computing device from one of the plurality of computing devices;

comparing the data representative of the detected gesture to data representative of a parent gesture cluster from the plurality of gesture clusters;

repeating a selection of a child gesture cluster for each generation of gesture clusters until features of gestures included in the respective child gesture cluster are less than a second threshold amount different from corresponding features of the detected gesture;

identifying at least one shortcut associated with the selected child gesture cluster; and

transmitting data representative of the at least one shortcut associated with the selected child gesture cluster to the computing device.

21. A device comprising:

at least one processor;

a network interface configured to receive data representative of a group of gestures detected by a plurality of computing devices and receive data representative of one or more shortcuts associated with the group of gestures, wherein each of the shortcuts corresponds to an action performed by at least one of the plurality of computing devices;

a language development module operable by the at least one processor to aggregate, based at least in part on detected similarities between at least one of

     1) the group of gestures and

     2) the one or more shortcuts associated with the group of gestures, the data representative of the group of gestures and the data representative of the associated shortcuts received from the plurality of computing devices into a plurality of gesture clusters organized into a generational hierarchy of parent and child gesture clusters such that each child gesture cluster of the generational hierarchy includes a plurality of gestures each determined to have features that are less than a first threshold amount different than corresponding features of gestures in a corresponding parent gesture cluster of the generational hierarchy; and

a gesture recognition module;

wherein the network interface is further configure to receive data representative of a gesture detected at a presence-sensitive screen of a computing device from one of the plurality of computing devices,

wherein the gesture recognition module is operable by the at least one processor to compare the data representative of the detected gesture to data representative of a parent gesture cluster from the plurality of gesture clusters, repeat a selection of a child gesture cluster for each generation of gesture clusters until features of gestures included in the respective child gesture cluster are less than a second threshold amount different from corresponding features of the detected gesture, and identify at least one shortcut associated with the selected child gesture cluster, and

wherein the network interface is further configure to transmit data representative of at least one shortcut associated with a selected child gesture cluster to the computing device.