Incremental multi-touch gesture recognition

• US 9,021,380 B2
• Filed: 10/05/2012
• Issued: 04/28/2015
• Est. Priority Date: 10/05/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Zhai, Shumin, Bi, Xiaojun, Ouyang, Yu
• Primary Examiner(s): BHARGAVA, ANIL K
• Application Number:
  US13/646,521
• Publication Number:
  US 20140098023A1
• Time in Patent Office: 935 Days
• Field of Search: 715/773, 715/780, 715/812, 715/863, 345/168
• US Class Current: 715/773
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0237 using prediction or retrieval techniques
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

In one example, a method comprises outputting, by a computing device and for display at an output device, a graphical keyboard comprising a plurality of keys, and receiving, by the computing device, an indication of a multi-touch gesture detected at a presence-sensitive display, the multi-touch gesture comprising a first sub-gesture that traverses a first group of keys of the plurality of keys and a second sub-gesture that traverses a second group of keys of the plurality of keys. This example method further comprises determining, in response to detecting the first sub-gesture and the second sub-gesture, a candidate word based at least in part on the first and second groups of keys, and outputting, by the computing device and for display at the output device, the candidate word.

Claims

1. A method comprising:

outputting, by a computing device and for display at a presence-sensitive display, a graphical keyboard comprising a plurality of keys;

receiving, by the computing device, an indication of a multi-touch gesture detected at the presence-sensitive display, the multi-touch gesture performed by a user having a first finger and a second finger, the multi-touch gesture comprising a first sub-gesture of the first finger that traverses a first group of keys of the plurality of keys and a second sub-gesture of the second finger that traverses a second group of keys of the plurality of keys, the first sub-gesture being disjoint from the second sub-gesture, wherein at least a portion of the first sub-gesture is performed simultaneously with at least a portion the second sub-gesture;

determining, by the computing device and in response to receiving the indication of the first sub-gesture of the first finger and the second sub-gesture of the second finger, a candidate word based at least in part on the first and second groups of keys, wherein the determining comprises;

determining, by the computing device and based at least in part on the first sub-gesture of the first finger, a first group of points on the presence-sensitive display traversed by the first sub-gesture;

determining, by the computing device and based at least in part on the second sub-gesture of the second finger, a second group of points on the presence-sensitive display traversed by the second sub-gesture;

determining, by the computing device and based at least in part on the first and second groups of points on the presence-sensitive display, at least one probability that at least one key included in at least one of the first and second groups of keys is associated with at least one of the first sub-gesture of the first finger and the second sub-gesture of the second finger; and

determining, by the computing device, the candidate word based at least in part on the at least one probability that the at least one key is associated with at least one of the first sub-gesture of the first finger and the second sub-gesture of the second finger; and

outputting, by the computing device and for display at the presence-sensitive display, the candidate word.

2. The method of claim 1, wherein outputting the candidate word comprises:

outputting the candidate word at a location of the presence-sensitive display that corresponds to a point of the first group of points on the presence-sensitive display traversed by the first sub-gesture or a point of the second group of alignment points on the presence-sensitive display traversed by the second sub-gesture.

3. The method of claim 1,

wherein receiving the indication of the gesture at the presence-sensitive display comprises receiving an indication of at least a portion of the first group of points on the presence-sensitive display traversed by the first sub-gesture before receiving an indication of at least a portion of the second group of points on the presence-sensitive display traversed by the second sub-gesture, and

wherein outputting the candidate word comprises outputting the candidate word at a location of the presence-sensitive display that corresponds to a point of the first group of points on the presence-sensitive display traversed by the first sub-gesture.

4. The method of claim 3, wherein receiving the indication of at least the portion of the first group of points on the presence-sensitive display traversed by the first sub-gesture before receiving the indication of at least the portion of the second group of points on the presence-sensitive display traversed by the second sub-gesture comprises receiving an indication of multiple points of the first group of points before receiving the indication of at least the portion of the second group of points.

5. The method of claim 4, wherein receiving the indication of the multiple points of the first group of points on the presence-sensitive display traversed by the first sub-gesture before receiving the indication of at least the portion of the second group of points on the presence-sensitive display traversed by the second sub-gesture comprises receiving the indication of the multiple points of the first group of points before receiving an indication of multiple points of the second group of points.

6. The method of claim 1, further comprising:

receiving, by the computing device, an indication of additional points on the presence-sensitive display traversed by at least one of the first sub-gesture and the second sub-gesture across the graphical keyboard; and

determining, by the computing device, whether the additional points are associated with at least one of the first sub-gesture and the second sub-gesture,

wherein outputting the candidate word comprises outputting the candidate word at a location of the presence-sensitive display that corresponds to one of the additional points.

7. The method of claim 6, wherein at least one of the first sub-gesture and the second sub-gesture comprises one or more segments.

8. The method of claim 6, wherein determining the candidate word further comprises determining the candidate word based at least in part on the association of the additional points with at least one of the first sub-gesture and the second sub-gesture.

9. The method of claim 1, wherein the first group of points on the presence-sensitive display traversed by the first sub-gesture comprises a first sequence of points, wherein the second group of points on the presence-sensitive display traversed by the second sub-gesture comprises a second sequence of points, and wherein the method further comprises:

determining, by the computing device, a first distance between a final point of the first sequence of points on the presence-sensitive display traversed by the first sub-gesture and a boundary of the graphical keyboard; and

determining, by the computing device, a second distance between a final point of the second sequence of points on the presence-sensitive display traversed by the second sub-gesture and the boundary of the graphical keyboard,

wherein outputting the candidate word comprises outputting the candidate word at a location of the presence-sensitive display that corresponds to either the final point of the first sequence of points or the final point of the second sequence of points based at least in part on a comparison the first distance to the second distance.

10. The method of claim 9, wherein outputting the candidate word comprises outputting the candidate word at a location of the presence-sensitive display that corresponds to the final point of the first sequence of points on the presence-sensitive display traversed by the first sub-gesture when the first distance is less than the second distance.

11. The method of claim 9, wherein outputting the candidate word comprises outputting the candidate word at a location of the presence-sensitive display that corresponds to the final point of the second sequence of points on the presence-sensitive display traversed by the second sub-gesture when the second distance is less than the first distance.

12. The method of claim 9, wherein the boundary of the graphical keyboard comprises a topmost boundary of the graphical keyboard.

13. The method of claim 1, wherein the first group of points on the presence-sensitive display traversed by the first sub-gesture comprises a first sequence of points, wherein the second group of points on the presence-sensitive display traversed by the second sub-gesture comprises a second sequence of points, and wherein the method further comprises:

determining, by the computing device, whether a final point of the first sequence of points or a final point of the second sequence of points was received at the presence-sensitive display more recently,

wherein outputting the candidate word comprises outputting the candidate word at a location of the presence-sensitive display corresponding to the one of the final point of the first sequence of points and the final point of the second sequence of points based at least in part on whether the final point of the first sequence of points or the final point of the second sequence of points was received more recently.

14. The method of claim 1, further comprising:

receiving, by the computing device, an indication of a termination of the first sub-gesture at the presence-sensitive display,

wherein outputting the candidate word comprises outputting the candidate word at a location of the presence-sensitive display that corresponds to a point of the second group of points on the presence-sensitive display traversed by the second sub-gesture responsive to receiving the indication of the termination of the first sub-gesture.

15. The method of claim 1, further comprising:

receiving, by the computing device, a first indication of a termination of the first sub-gesture and a second indication of a termination of the second sub-gesture at the presence-sensitive display,

wherein outputting the candidate word comprises outputting the candidate word responsive to receiving the first indication of the termination of the first sub-gesture and the second indication of the termination of the second sub-gesture.

16. The method of claim 1, wherein the first group of keys are associated with a first group of characters, wherein the second group of keys are associated with a second group of characters, and wherein the candidate word comprises characters that are not included in either the first group of characters or the second group of characters, such that the computing device performs auto-correct functionality to output the candidate word.

17. The method of claim 1, wherein the first group of keys are associated with a first group of characters, wherein the second group of keys are associated with a second group of characters, and wherein the candidate word comprises a greater number of characters than included in a combination of the first and second group of characters, such that the computing device performs auto-complete functionality to output the candidate word.

18. The method of claim 1, wherein determining the first group of points on the presence-sensitive display traversed by the first sub-gesture is based at least in part on one or more of:

a length of a segment associated with the first sub-gesture, wherein the segment comprises a path traversed by the first sub-gesture at the presence-sensitive display;

a direction of movement associated with the segment of the first sub-gesture;

a curvature of the segment of the first sub-gesture;

a local speed that represents a rate at which the segment of the first sub-gesture is detected; and

a global speed that represents a rate at which the first sub-gesture is detected.

19. The method of claim 18, wherein determining the at least one probability that the at least one key included in the at least one of the first and second groups of keys is associated with at least one of the first sub-gesture and the second sub-gesture comprises:

determining a first probability that the at least one key is included in the first group of keys associated with the first sub-gesture;

determining a second probability that the at least one key is included in the second group of keys associated with the second sub-gesture;

if the first probability is greater than the second probability, determining that the at least one key is included in the first group of keys; and

if the second probability is greater than the first probability, determining that the at least one key is included in the second group of keys.

20. The method of claim 19, wherein determining the first probability is based at least in part on one or more of:

a length of a segment associated with the first sub-gesture, wherein the segment comprises a path traversed by the first sub-gesture at the presence-sensitive display;

a direction of movement associated with the segment of the first sub-gesture;

a curvature of the segment of the first sub-gesture;

a local speed that represents a rate at which the segment of the first sub-gesture is detected;

a global speed that represents a rate at which the first sub-gesture is detected; and

a distance between one point of the first group of points and a region that is associated with at least a portion of the at least one key.

21. The method of claim 1, wherein the candidate word is determined contemporaneously with receiving at least one of the first sub-gesture and the second sub-gesture at the presence-sensitive display.

22. The method of claim 1, wherein determining the candidate word further comprises:

determining, by the computing device, a respective cost value for each of at least two keys included in at least one of the first and second groups of keys, wherein each of the respective cost values represents a probability that a point of the first group of points or the second group of points indicates the corresponding key of the at least two keys;

comparing, by the computing device, the respective cost values for the at least two keys to determine a combination of keys having a combined cost value that satisfies a threshold.

23. The method of claim 22, wherein determining the respective cost value for each of the at least two keys comprises determining a respective lexical cost value for each of the at least two keys, wherein each of the respective lexical cost values represents a probability that a letter represented by the corresponding key of the at least two keys is included in the candidate word, and wherein determining the respective lexical cost value for each of the at least two keys comprises comparing each of the at least two keys with a language model.

24. The method of claim 1,

wherein determining the candidate word further comprises determining the candidate word and one or more other suggested words based at least in part on the at least one probability that the at least one key is associated with at least one of the first sub-gesture and the second sub-gesture, and

wherein outputting the candidate word comprises outputting the candidate word along with the one or more other suggested words, such that the candidate word is visually distinguished from the one or more other suggested words.

25. The method of claim 1, wherein the first sub-gesture and the second sub-gesture comprise swipe gestures, tap gestures, or a combination thereof.

26. The method of claim 1, wherein determining the at least one probability that the at least one key included in at least one of the first and second groups of keys is associated with at least one of the first sub-gesture and the second sub-gesture comprises determining the at least one probability further based at least in part of one or more words included in a language model.

27. A computing device, comprising:

at least one processor,

wherein the at least one processor is configured to;

output, for display at a presence-sensitive display, a graphical keyboard comprising a plurality of keys;

receive an indication of a multi-touch gesture detected at the presence-sensitive display, the multi-touch gesture performed by a user having a first finger and a second finger, the multi-touch gesture comprising a first sub-gesture of the first finger that traverses a first group of keys of the plurality of keys and a second sub-gesture of the second finger that traverses a second group of keys of the plurality of keys, the first sub-gesture being disjoint from the second sub-gesture, wherein at least a portion of the first sub-gesture is performed simultaneously with at least a portion the second sub-gesture;

determine, in response to receiving the indication of the first sub-gesture of the first finger and the second sub-gesture of the second finger, a candidate word based at least in part on the first and second groups of keys, wherein the determining comprises;

determining, based at least in part on the first sub-gesture of the first finger, a first group of points on the presence-sensitive display traversed by the first sub-gesture;

determining, based at least in part on the second sub-gesture, a second group of points on the presence-sensitive display traversed by the second sub-gesture of the second finger;

determining, based at least in part on the first and second groups of points on the presence-sensitive display, at least one probability that at least one key included in at least one of the first and second groups of keys is associated with at least one of the first sub-gesture of the first finger and the second sub-gesture of the second finger; and

determining the candidate word based at least in part on the at least one probability that the at least one key is associated with at least one of the first sub-gesture of the first finger and the second sub-gesture of the second finger; and

output, for display at the presence-sensitive display, the candidate word.

28. A non-transitory computer-readable storage device medium encoded with instructions that, when executed, cause at least one processor to:

output, for display at a presence-sensitive display, a graphical keyboard comprising a plurality of keys;

receive an indication of a multi-touch gesture detected at the presence-sensitive display, the multi-touch gesture performed by a user having a first finger and a second finger, the multi-touch gesture comprising a first sub-gesture of the first finger that traverses a first group of keys of the plurality of keys and a second sub-gesture of the second finger that traverses a second group of keys of the plurality of keys, the first sub-gesture being disjoint from the second sub-gesture, wherein at least a portion of the first sub-gesture is performed simultaneously with at least a portion the second sub-gesture;

determine, in response to receiving the indication of the first sub-gesture of the first finger and the second sub-gesture of the second finger, a candidate word based at least in part on the first and second groups of keys, wherein the determining comprises;

determining, based at least in part on the first sub-gesture, a first group of points on the presence-sensitive display traversed by the first sub-gesture of the first finger;

determining, based at least in part on the second sub-gesture, a second group of points on the presence-sensitive display traversed by the second sub-gesture of the second finger;

determining, based at least in part on the first and second groups of points on the presence-sensitive display, at least one probability that at least one key included in at least one of the first and second groups of keys is associated with at least one of the first sub-gesture of the first finger and the second sub-gesture of the second finger; and

determining the candidate word based at least in part on the at least one probability that the at least one key is associated with at least one of the first sub-gesture of the first finger and the second sub-gesture of the second finger; and

output, for display at the presence-sensitive display, the candidate word.