User state-adaptive text input

• US 10,282,089 B2
• Filed: 07/07/2015
• Issued: 05/07/2019
• Est. Priority Date: 02/03/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Foerster, Jakob, Hansson, Othar
• Primary Examiner(s): Repsher, III, John T
• Application Number:
  US14/793,070
• Publication Number:
  US 20160224240A1
• Time in Patent Office: 1,400 Days
• CPC Class Codes: G06F 3/0237 using prediction or retrieval techniques
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

In one example, a method includes outputting, by a computing device (102) and for display at a display device (104) operatively coupled to the computing device, a graphical keyboard (124) comprising a plurality of keys, and receiving, by the computing device, an indication of a gesture detected at a presence-sensitive input device operatively coupled to the computing device. In this example, the method also includes detecting, by one or more sensors (154) of the computing device, data indicative of one or more user states determined to affect gesture accuracy; selecting, by the computing device and based at least in part on the indication of the gesture and the data, a particular candidate string; and outputting, by the computing device and for display at the display device, the particular candidate string.

Claims

1. A method comprising:

outputting, by a computing device and for display at a display device operatively coupled to the computing device, a graphical keyboard comprising a plurality of keys;

receiving, by the computing device, an indication of a gesture detected at a presence-sensitive input device operatively coupled to the computing device;

detecting, by the computing device, data indicative of one or more user states determined to affect gesture accuracy;

determining, for respective user states of the one or more user states, respective models that provide at least one value indicating a degree, in a range of degrees, to which the respective user state affects gesture accuracy;

selecting, by the computing device and based at least in part on the indication of the gesture and the data indicative of the one or more user states, a particular candidate string, wherein selecting the particular candidate string comprises;

determining, based on the gesture, a plurality of candidate character strings, wherein each of the plurality of candidate character strings is associated with at least one of a respective physical cost value from a spatial model and a respective lexical cost value from a language model;

adjusting, for a particular candidate character string of the plurality of candidate character strings and based at least in part on the detected one or more user states, at least one of a physical cost value associated with the particular candidate character string and a lexical cost value associated with the particular candidate character string, wherein the adjusting comprises adjusting a relative weight based on the respective models for each of the one or more user states; and

selecting, based at least in part on the adjusting, the particular candidate character string from the plurality of candidate character strings; and

outputting, by the computing device and for display at the display device, the particular candidate string.

2. The method of claim 1, wherein the data indicative of the one or more user states further indicates:

one or more activities engaged in by a user of the computing device while the user performed the gesture, or

one or more hand-eye coordination impairment states affecting the user while the user performed the gesture.

3. The method of claim 2, wherein the one or more activities engaged in by the user include one or more of walking, running, cycling, playing a sport, exercising, and being present aboard a moving apparatus.

4. The method of claim 1, wherein the one or more hand-eye coordination impairment states affecting the user further include a user temperature condition, an intoxication state of the user, or attire worn by the user.

5. The method of claim 1, wherein adjusting at least one of the physical cost value associated with the particular candidate character string and the lexical cost value associated with the particular candidate character string further comprises:

determining, based at least in part on at least one user state, a weight; and

applying the weight to the at least one of the physical cost value associated with the particular candidate character string and the lexical cost value associated with the particular candidate character string.

6. The method of claim 1 wherein the gesture is a series of taps.

7. The method of claim 1, wherein the gesture is a swipe with two end points.

8. The method of claim 7, wherein the gesture further comprises angle-of-direction changes between the two endpoints.

9. The method of claim 1, wherein the graphical keyboard is a virtual keyboard.

10. A computing device comprising:

one or more processors;

one or more sensors, coupled to the one or more processors, configured to detect data indicative of one or more user states determined to affect gesture accuracy;

a presence-sensitive display, coupled to the one or more processors, comprising a display component and a presence-sensitive input component; and

a storage device storing at least one module, wherein the at least one module is executable by the one or more processors to;

output, for display at the display component, a graphical keyboard comprising a plurality of keys;

receive an indication of a gesture detected at the presence-sensitive input component;

determine, for respective user states of the one or more user states, respective models that provide at least one value indicating a degree, in a range of degrees, to which the respective user state affects gesture accuracy;

select, based at least in part on the indication of the gesture and the data indicative of the one or more user states, a particular candidate string, wherein, to select the particular candidate string, the at least one module is executable by the one or more processors to;

determine, based on the gesture, a plurality of candidate character strings, wherein each of the plurality of candidate character strings is associated with at least one of a respective physical cost value from a spatial model and a respective lexical cost value from a language model;

adjust, for a particular candidate character string of the plurality of candidate character strings and based at least in part on the one or more user states, at least one of a physical cost value associated with the particular candidate character string and a lexical cost value associated with the particular candidate character string by at least adjusting a relative weight based on the respective models for each of the one or more user states; and

select, based at least in part on the adjusting, the particular candidate character string from the plurality of candidate character strings; and

output, for display at the display component, the particular candidate string.

11. The computing device of claim 10, wherein the one or more sensors configured to detect the data indicative of the one or more user states further comprise:

one or more sensors configured to detect data indicative of one or more activities engaged in by a user of the computing device while the user performed the gesture, or

one or more hand-eye coordination impairment states affecting the user while the user performed the gesture.

12. The computing device of claim 11, wherein the one or more sensors configured to detect the data indicative of the one or more activities engaged in by the user of the computing device comprise a motion sensor for determining the one or more activities.

13. The computing device of claim 10, wherein the one or more hand-eye coordination impairment states affecting the user further include a user temperature condition, wherein the one or more sensors comprise a thermometer for determining an ambient temperature, and wherein the module is further executable the by the one or more processors to:

determine, based on the ambient temperature, the user temperature condition.

14. The computing device of claim 10, wherein, to adjust at least one of the physical cost value associated with the particular candidate character string and the lexical cost value associated with the particular candidate character string, the at least one module is further executable by the one or more processors to:

determine, based at least in part on at least one user state, a weight; and

apply the weight to the at least one of the physical cost value associated with the particular candidate character string and the lexical cost value associated with the particular candidate character string.

15. A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors of a computing device to:

output, for display at a display device that is operatively coupled to the computing device, a graphical keyboard comprising a plurality of keys;

receive an indication of a gesture detected at a presence-sensitive input device that is operatively coupled to the computing device;

detect data indicative of one or more user states determined to affect gesture accuracy;

determine, for respective user states of the one or more user states, respective models that provide at least one value indicating a degree, in a range of degrees, to which the respective user state affects gesture accuracy;

select, based at least in part on the indication of the gesture and the data indicative of the one or more user states, a particular candidate string, wherein the instructions that cause the one or more processors to select the particular candidate string comprise instructions that cause the one or more processors to;

determine, based on the gesture, a plurality of candidate character strings, wherein each of the plurality of candidate character strings is associated with at least one of a respective physical cost value from a spatial model and a respective lexical cost value from a language model;

adjust, for a particular candidate character string of the plurality of candidate character strings and based at least in part on the one or more user states, at least one of a physical cost value associated with the particular candidate character string and a lexical cost value associated with the particular candidate character string by at least adjusting a relative weight based on the respective models for each of the one or more user states; and

select, based at least in part on the adjusting, the particular candidate character string from the plurality of candidate character strings; and

output, for display at the display device, the particular candidate string.

16. The computer-readable storage medium of claim 15, wherein the data indicative of the one or more user states further indicates at least one of one or more activities engaged in by a user of the computing device while the user performed the gesture wherein the one or more activities engaged in by the user include one or more of walking, running, cycling, driving, playing a sport, exercising, and being present aboard a moving apparatus;

and wherein the one or more hand-eye coordination impairment states affecting the user further include one or more of a temperature condition, attire worn by the user, and a physiological state.