Neural network for keyboard input decoding

• US 10,248,313 B2
• Filed: 03/29/2017
• Issued: 04/02/2019
• Est. Priority Date: 04/10/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Zhai, Shumin, Breuel, Thomas, Alsharif, Ouais, Ouyang, Yu, Beaufays, Francoise, Schalkwyk, Johan
• Primary Examiner(s): Cheng, Joe H
• Application Number:
  US15/473,010
• Publication Number:
  US 20170199665A1
• Time in Patent Office: 734 Days
• CPC Class Codes: G06F 3/0219 Special purpose keyboards
• CPC Class Codes: G06F 3/0233 Character input methods
• CPC Class Codes: G06F 3/0237 using prediction or retrieval techniques
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06F 3/04895 Guidance during keyboard input operation, e.g. prompting
• CPC Class Codes: G06F 40/232 Orthographic correction, e.g. spell checking or vowelisation
• CPC Class Codes: G06F 40/274 Converting codes to words; Guess-ahead of partial word inputs
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

In some examples, a computing device includes at least one processor; and at least one module, operable by the at least one processor to: output, for display at an output device, a graphical keyboard; receive an indication of a gesture detected at a location of a presence-sensitive input device, wherein the location of the presence-sensitive input device corresponds to a location of the output device that outputs the graphical keyboard; determine, based on at least one spatial feature of the gesture that is processed by the computing device using a neural network, at least one character string, wherein the at least one spatial feature indicates at least one physical property of the gesture; and output, for display at the output device, based at least in part on the processing of the at least one spatial feature of the gesture using the neural network, the at least one character string.

Claims

1. A method comprising:

outputting, by a computing device, for display, a graphical keyboard that includes a plurality of keys;

receiving, by the computing device, a gesture that selects one or more keys from the plurality of keys;

determining, by the computing device, based on the gesture and a Long Short Term Memory network, at least one character string; and

outputting, by the computing device, for display, the at least one character string.

2. The method of claim 1, wherein determining the at least one character string comprises:

determining, based on the gesture, one or more input values to the Long Short Term Memory network, the one or more input values corresponding to one or more features; and

determining, based on the one or more input values, one or more output values from the Long Short Term Memory network, the one or more output values corresponding to the one or more features; and

determining, based on the one or more output values, the at least one character string.

3. The method of claim 2, wherein the one or more features comprise at least one of spatial features, temporal features, lexical features, and contextual features.

4. The method of claim 3, wherein the spatial features include one or more of:

a location of the gesture relative to a presence-sensitive input device, a speed of the gesture, a direction of the gesture, a curvature of the gesture, a particular key from the plurality of keys that is traversed by the gesture, or a type of the gesture.

5. The method of claim 3, wherein the temporal features include information about a time at which the gesture is received.

6. The method of claim 3, wherein the lexical features include one or more of one or more characters, character strings, or multi-character string phrases associated with other gestures received prior to the gesture.

7. The method of claim 3, wherein the contextual features include one or more of an identity of a user of the computing device, a geolocation of the computing device, an environment or climate of the computing device, audio or visual information determined by the computing device, sensor information detected by the computing device, a type of input field associated with text determined from the gesture, an application associated with the text determined from the gesture, or a recipient associated with the text determined from the gesture.

8. The method of claim 2, wherein the one or more output values that correspond to one or more features comprise a plurality of one or more of characters, character strings, or multi-string phrases and a respective probability for each of the plurality of one or more of characters, character strings, or multi-string phrases.

9. The method of claim 1, wherein outputting the at least one character string for display comprises outputting the at least one character string for display in a word suggestion region of the graphical keyboard.

10. The method of claim 1, wherein the Long Short Term Memory network comprises at least one of:

one or more memory block or a plurality of layers of memory blocks.

11. The method of claim 1, wherein the Long Short Term Memory network is configured to execute at one or more local processors of the computing device.

12. The method of claim 1, wherein the computing device does not include an n-gram language model.

13. The method of claim 1, wherein the computing device does not include an n-gram spatial model.

14. The method of claim 1, wherein the gesture is at least one of a tap gesture, continuous gesture, or combination of tap gesture and continuous gesture.

15. A non-transitory computer-readable storage medium encoded with instructions that, when executed, cause at least one processor to:

output, for display, a graphical keyboard that includes a plurality of keys;

receive a gesture that selects one or more keys from the plurality of keys;

determine, based on the gesture and a Long Short Term Memory network, at least one character string; and

output, for display, the at least one character string.

16. The non-transitory computer-readable storage medium of claim 15, wherein the instructions, when executed, further cause the at least one processor determine the at least one character string by at least:

determining, based on the gesture, one or more input values to the Long Short Term Memory network, the one or more input values corresponding to one or more features; and

determining, based on the one or more input values, one or more output values from the Long Short Term Memory network, the one or more output values corresponding to the one or more features; and

determining, based on the one or more output values, the at least one character string.

17. The non-transitory computer-readable storage medium of claim 15, wherein the instructions, when executed, further cause the at least one processor to output the at least one character string for display by at least outputting the at least one character string for display in a word suggestion region of the graphical keyboard.

18. A computing device comprising:

at least one processor configured to;

output, for display, a graphical keyboard that includes a plurality of keys;

receive a gesture that selects one or more keys from the plurality of keys;

determine, based on the gesture and a Long Short Term Memory network, at least one character string; and

output, for display, the at least one character string.

19. The computing device of claim 18, wherein the at least one processor is configured to determine the at least one character string by at least:

determining, based on the gesture, one or more input values to the Long Short Term Memory network, the one or more input values corresponding to one or more features; and

determining, based on the one or more input values, one or more output values from the Long Short Term Memory network, the one or more output values corresponding to the one or more features; and

determining, based on the one or more output values, the at least one character string.

20. The computing device of claim 18, wherein the at least one processor is configured to output the at least one character string for display by at least outputting the at least one character string for display in a word suggestion region of the graphical keyboard.