Neural network for keyboard input decoding

• US 10,671,281 B2
• Filed: 01/30/2019
• Issued: 06/02/2020
• Est. Priority Date: 04/10/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Zhai, Shumin, Breuel, Thomas, Alsharif, Ouais, Ouyang, Yu, Beaufays, Francoise, Schalkwyk, Johan
• Primary Examiner(s): Cheng, Joe H
• Application Number:
  US16/261,640
• Publication Number:
  US 20190155504A1
• Time in Patent Office: 489 Days
• CPC Class Codes: G06F 3/0219 Special purpose keyboards
• CPC Class Codes: G06F 3/0233 Character input methods
• CPC Class Codes: G06F 3/0237 using prediction or retrieval techniques
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06F 3/04895 Guidance during keyboard input operation, e.g. prompting
• CPC Class Codes: G06F 40/232 Orthographic correction, e.g. spell checking or vowelisation
• CPC Class Codes: G06F 40/274 Converting codes to words; Guess-ahead of partial word inputs
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

In some examples, a computing device includes at least one processor; and at least one module, operable by the at least one processor to: output, for display at an output device, a graphical keyboard; receive an indication of a gesture detected at a location of a presence-sensitive input device, wherein the location of the presence-sensitive input device corresponds to a location of the output device that outputs the graphical keyboard; determine, based on at least one spatial feature of the gesture that is processed by the computing device using a neural network, at least one character string, wherein the at least one spatial feature indicates at least one physical property of the gesture; and output, for display at the output device, based at least in part on the processing of the at least one spatial feature of the gesture using the neural network, the at least one character string.

Claims

1. A method comprising:

outputting, by a computing device, for display, a graphical keyboard that includes a plurality of keys;

receiving, by the computing device, at least one gesture that selects one or more keys from the plurality of keys;

determining, by the computing device, based on the at least one gesture and a deep neural network, at least one character string; and

outputting, by the computing device, for display, the at least one character string.

2. The method of claim 1, wherein the deep neural network is trained based on input sequences corresponding to different selections of the plurality of keys for entering word phrases and sentences.

3. The method of claim 2, wherein the deep neural network is further trained based on a frequency that at least one of:

a) the at least one character string or b) the word phrases and sentences, is entered with the graphical keyboard for different contexts associated with the computing device.

4. The method of claim 2, wherein the deep neural network is further trained based on a frequency that the at least one character string occurs in a vocabulary associated with an identity of a user of the computing device.

5. The method of claim 2, wherein the deep neural network is further trained based on a frequency that the at least one character string occurs in a vocabulary associated with an identity of an intended recipient of text including the at least one character string.

6. The method of claim 2, wherein the deep neural network is further trained based on a frequency that the at least one character string occurs in a language context including the word phrases and sentences.

7. The method of claim 1, further comprising:

determining, by the computing device, based on the deep neural network and at least one other gesture entered at the graphical keyboard prior to receiving the at least one gesture, at least one character string previously entered with the graphical keyboard, wherein determining the at least one character string comprises determining the at least one character string based on;

the at least one gesture,

the deep neural network, and

the at least one character string previously entered with the graphical keyboard.

8. The method of claim 7, wherein the at least one character string and the at least one character string previously entered with the graphical keyboard comprise word phrases or sentences.

9. The method of claim 1, wherein the deep neural network functions as a spatial model of the graphical keyboard or both a language model of the graphical keyboard and the spatial model of the graphical keyboard.

10. The method of claim 1, wherein the deep neural network is configured to execute at one or more local processors of the computing device.

11. The method of claim 1, wherein the at least one gesture comprises one or more tap gestures, one or more continuous gestures, or a combination of one or more tap gestures and one or more continuous gestures.

12. A non-transitory computer-readable storage medium encoded with instructions that, when executed, cause at least one processor to:

output, for display, a graphical keyboard that includes a plurality of keys;

receive at least one gesture that selects one or more keys from the plurality of keys;

determine, based on the at least one gesture and a deep neural network, at least one character string; and

output, for display, the at least one character string.

13. The non-transitory computer-readable storage medium of claim 12, wherein the instructions, when executed, further cause the at least one processor to train the deep neural network based on input sequences corresponding to different selections of the plurality of keys for entering word phrases and sentences.

14. The non-transitory computer-readable storage medium of claim 12, wherein the instructions, when executed, further cause the at least one processor to:

determine, based on the deep neural network and at least one other gesture entered at the graphical keyboard prior to receiving the at least one gesture, at least one character string previously entered with the graphical keyboard; and

determine the at least one character string based on;

the at least one gesture,

the deep neural network, and

the at least one character string previously entered with the graphical keyboard.

15. The non-transitory computer-readable storage medium of claim 12, wherein the at least one processor comprises one or more local processors of a mobile computing device.

16. The non-transitory computer-readable storage medium of claim 12, wherein the at least one gesture comprises one or more tap gestures, one or more continuous gestures, or a combination of one or more tap gestures and one or more continuous gestures.

17. A computing device comprising at least one processor configured to:

output, for display, a graphical keyboard that includes a plurality of keys;

receive at least one gesture that selects one or more keys from the plurality of keys;

determine, based on the at least one gesture and a deep neural network, at least one character string; and

output, for display, the at least one character string.

18. The computing device of claim 17, wherein the deep neural network is trained based on input sequences corresponding to different selections of the plurality of keys for entering word phrases and sentences.

19. The computing device of claim 17, wherein the at least one processor is further configured to:

determine, based on the deep neural network and at least one other gesture entered at the graphical keyboard prior to receiving the at least one gesture, at least one character string previously entered with the graphical keyboard; and

determine the at least one character string based on;

the at least one gesture,

the deep neural network, and

the at least one character string previously entered with the graphical keyboard.

20. The computing device of claim 17, wherein the at least one gesture comprises one or more tap gestures, one or more continuous gestures, or a combination of one or more tap gestures and one or more continuous gestures.