Deep machine learning to perform touch motion prediction

• US 10,514,799 B2
• Filed: 09/08/2016
• Issued: 12/24/2019
• Est. Priority Date: 09/08/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Lin, Pin-chih, Lin, Tai-hsu
• Primary Examiner(s): Hicks, Austin
• Application Number:
  US15/259,917
• Publication Number:
  US 20180067605A1
• Time in Patent Office: 1,202 Days
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/0442 characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent

Abstract

The present disclosure provides systems and methods that leverage machine learning to perform user input motion prediction. In particular, the systems and methods of the present disclosure can include and use a machine-learned motion prediction model that is trained to receive motion data indicative of motion of a user input object and, in response to receipt of the motion data, output predicted future locations of the user input object. The user input object can be a finger of a user or a stylus operated by the user. The motion prediction model can include a deep recurrent neural network.

Claims

1. A computer-implemented method to predict motion of user input objects, the method comprising:

obtaining, by one or more computing devices, a first set of motion data associated with a user input object, the first set of motion data descriptive of a location of the user input object over time;

inputting, by the one or more computing devices, the first set of motion data into a recurrent neural network of a machine-learned motion prediction model;

receiving, by the one or more computing devices as an output of the motion prediction model, a motion prediction vector that describes one or more predicted future locations of the user input object respectively for one or more future times; and

clearing, by the one or more computing devices, a cache associated with the recurrent neural network upon a determination that the user input object has ceased providing user input.

2. The computer-implemented method of claim 1, wherein obtaining, by the one or more computing devices, the first set of motion data associated with the user input object comprises

obtaining, by the one or more computing devices, the first set of motion data associated with a finger of a user or a stylus operated by the user, the first set of motion data descriptive of motion of the finger or the stylus relative to a touch-sensitive screen.

3. The computer-implemented method of claim 1, wherein inputting, by the one or more computing devices, the first set of motion data into the machine-learned recurrent neural network of the motion prediction model comprises

inputting, by the one or more computing devices, the first set of motion data into a machine-learned long short-term memory (LSTM) neural network of the motion prediction model.

4. The computer-implemented method of claim 1, further comprising:

receiving, by the one or more computing devices as an output of the recurrent neural network, a high-dimensional context vector; and

inputting, by the one or more computing devices, the high-dimensional context vector and a time vector descriptive of the one or more future times into a feed forward neural network of the motion prediction model;

wherein receiving, by the one or more computing devices as the output of the motion prediction model, the motion prediction vector comprises

receiving, by the one or more computing devices, the motion prediction vector as an output of the feed forward neural network.

5. The computer-implemented method of claim 1, further comprising:

receiving, by the one or more computing devices as an output of the recurrent neural network, a high-dimensional context vector; and

inputting, by the one or more computing devices, the high-dimensional context vector and a time vector descriptive of the one or more future times into a recurrent decoder neural network of the motion prediction model;

wherein receiving, by the one or more computing devices as the output of the motion prediction model, the motion prediction vector comprises

receiving, by the one or more computing devices, the motion prediction vector as an output of the recurrent decoder neural network.

6. The computer-implemented method of claim 1, wherein:

inputting, by the one or more computing devices, the first set of motion data into the machine-learned recurrent neural network comprises

inputting, by the one or more computing devices, the first set of motion data and a time vector descriptive of the one or more future times into the recurrent neural network; and

receiving, by the one or more computing devices as the output of the motion prediction model, the motion prediction vector comprises

receiving, by the one or more computing devices, the motion prediction vector as an output of the recurrent neural network.

7. The computer-implemented method of claim 1, wherein obtaining, by the one or more computing devices, the first set of motion data associated with the user input object comprises:

obtaining, by the one or more computing devices, the first set of motion data that provides at least one value describing a change in the location of the user input object in an x dimension, at least one value describing a change in the location of the user input object in a y dimension, and at least one value describing a change in time;

or

obtaining, by the one or more computing devices, the first set of motion data that provides at least two values describing at least two locations of the user input object in the x dimension, at least two values describing at least two locations of the user input object in the y dimension, and at least two values describing at least two times.

8. The computer-implemented method of claim 1, further comprising:

training, by the one or more computing devices, the motion prediction model based on a set of training data;

wherein training, by the one or more computing devices, the motion prediction model comprises

backpropagating, by the one or more computing devices, a loss function through the motion prediction model; and

wherein the loss function comprises an error ratio that describes a first sum of one or more lengths respectively of one or more error vectors divided by a second sum of one or more lengths respectively of one or more ground truth vectors.

9. The computer-implemented method of claim 1, wherein:

obtaining, by the one or more computing devices, the first set of motion data comprises

iteratively obtaining, by the one or more computing device, a plurality of motion updates that respectively update the location of the user input object over time;

inputting, by the one or more computing devices, the first set of motion data into the recurrent neural network comprises

iteratively inputting, by the one or more computing device, each of the plurality of motion updates into the recurrent neural network as they are iteratively obtained; and

receiving, by the one or more computing devices as the output of the motion prediction model, the motion prediction vector comprises

iteratively receiving, by the one or more computing devices, a plurality of motion prediction vectors as outputs of the motion prediction model.

10. A computing device that predicts motion of user input objects, the mobile computing device comprising:

at least one processor;

a machine-learned motion prediction model that comprises a recurrent neural network, wherein the motion prediction model is trained to receive motion data indicative of motion of a user input object and, in response to receipt of the motion data, output predicted future locations of the user input object; and

at least one tangible, non-transitory computer-readable medium that stores instructions that, when executed by the at least one processor, cause the at least one processor to;

obtain a first set of motion data associated with the user input object, the first set of motion data descriptive of a location of the user input object over time;

input the first set of motion data into the recurrent neural network of the motion prediction model;

receive, as an output of the motion prediction model, a motion prediction vector that describes the one or more predicted future locations of the user input object respectively for one or more future times; and

perform one or more actions associated with the one or more predicted future locations described by the motion prediction vector;

wherein the motion prediction model was trained by backpropagation of a loss function through the motion prediction model, and the loss function comprises an error ratio that describes a first sum of one or more lengths respectively of one or more error vectors divided by a second sum of one or more lengths respectively of one or more ground truth vectors.

11. The computing device of claim 10, wherein the first set of motion data is associated with a finger of a user or a stylus operated by the user, and wherein the first set of motion data is descriptive of motion of the finger or the stylus relative to a touch-sensitive screen.

12. The computing device of claim 10, wherein the recurrent neural network comprises a long short-term memory (LSTM) neural network.

13. The computing device of claim 10, wherein:

the motion prediction model further comprises a feed forward neural network configured to receive an output of the recurrent neural network, wherein the output of the recurrent neural network comprises a high-dimensional context vector;

execution of the instructions further causes the at least one processor to input a time vector descriptive of the one or more future times into the feed forward neural network of the motion prediction model alongside the high-dimensional context vector;

the at least one processor receives the motion prediction vector as an output of the feed forward neural network.

14. The computing device of claim 10, wherein:

the motion prediction model further comprises a recurrent decoder neural network configured to receive an output of the recurrent neural network, wherein the output of the recurrent neural network comprises a high-dimensional context vector;

execution of the instructions further causes the at least one processor to input a time vector descriptive of the one or more future times into the recurrent decoder neural network of the motion prediction model alongside the high-dimensional context vector;

the at least one processor receives the motion prediction vector as an output of the recurrent decoder neural network.

15. The computing device of claim 10, wherein:

execution of the instructions further causes the at least one processor to input a time vector descriptive of the one or more future times into the recurrent neural network alongside the first set of motion data;

the at least one processor receives the motion prediction vector as an output of the recurrent neural network.

16. A computing system, comprising:

a user computing device, the user computing device comprising;

at least one processor; and

at least one non-transitory computer-readable medium that stores;

a machine-learned motion prediction model configured to receive motion data indicative of motion of a user input object and, in response to receipt of the motion data, output predicted future locations of the user input object, the motion prediction model trained by backpropagation of a loss function through the motion prediction model, the loss function comprising an error ratio that describes a first sum of one or more lengths respectively of one or more error vectors divided by a second sum of one or more lengths respectively of one or more ground truth vectors; and

instructions that, when executed by the at least one processor, cause the user computing device to use the machine-learned motion prediction model to obtain the predicted future locations of the user input object.

17. The computing system of claim 16, wherein the motion prediction model comprises a recurrent neural network.

18. The computing system of claim 16, further comprising:

a training computing system that backpropagates the loss function through the motion prediction model to train the motion prediction model.

19. A computing device that predicts motion of user input objects, the computing device comprising:

at least one processor;

a machine-learned motion prediction model, wherein the motion prediction model is trained to receive motion data indicative of motion of a user input object and, in response to receipt of the motion data, output predicted future locations of the user input object, wherein the machine-learned motion prediction model comprises a recurrent neural network and a recurrent decoder neural network configured to receive an output of the recurrent neural network, wherein the output of the recurrent neural network comprises a high-dimensional context vector; and

at least one tangible, non-transitory computer-readable medium that stores instructions that, when executed by the at least one processor, cause the at least one processor to;

obtain a first set of motion data associated with the user input object, the first set of motion data descriptive of a location of the user input object over time;

input the first set of motion data into the recurrent neural network of the motion prediction model;

receive, as an output of the recurrent neural network, the high-dimensional context vector;

input a time vector descriptive of the one or more future times into the recurrent decoder neural network of the motion prediction model alongside the high-dimensional context vector;

receive, as an output of the recurrent decoder neural network, a motion prediction vector that describes the one or more predicted future locations of the user input object respectively for one or more future times; and

perform one or more actions associated with the one or more predicted future locations described by the motion prediction vector.