RECOGNIZING INPUT GESTURES

• US 20090006292A1
• Filed: 06/27/2007
• Published: 01/01/2009
• Est. Priority Date: 06/27/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Block, Matthew
• Granted Patent: US 7,835,999 B2
• Time in Patent Office: Days
• US Class Current: 706/20
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 30/333 Preprocessing; Feature extraction

Abstract

The present invention extends to methods, systems, and computer program products for recognizing input gestures. A neural network is trained using example inputs and backpropagation to recognize specified input patterns. Input gesture data is representative of movements in contact on a multi-touch input display surface relative to one or more axes over time. Example inputs used for training the neural network to recognize a specified input pattern can be created from sampling input gesture data for example input gestures known to represent the specified input pattern. Trained neural networks can subsequently be used to recognize input gestures that are similar to known input gestures as the specified input pattern corresponding to the known input gestures.

Claims

1. At a computer system including a multi-touch input display surface, a method for training the computer system to recognize input gesture data entered at the multi-touch input display surface as a corresponding specified input pattern, the method comprising:

an act of accessing input gesture data for at least one example of a specified input pattern, the input gesture data representing detected contact on the multi-touch input display surface over a period of time, the input gesture data including for at least the one example of the specified input pattern;

first direction movement data, the first direction movement data indicating the position of detected contact on the multi-touch input display surface relative to a first axis over the time period; and

second direction movement data, the second direction movement data indicating the position of detected contact on the multi-touch input display surface relative to a second different axis over the time period;

an act of encoding the input gesture data into an input vector of real values, each real value corresponding to a sample on the first or second axis;

an act of saving an indication that the real values contained in the input vector represent an example of the specified input pattern;

an act of accessing a fully connected neural network, the neural network including input nodes, hidden nodes, and output nodes, the input nodes being fully connected to the hidden nodes and the hidden nodes being fully connected to the output nodes;

an act of creating a training set to supply to the neural network to train the neural network to recognize the specified input pattern, the training set including at least the input vector and a corresponding output vector, the corresponding output vector indicating the desired output of the neural network that is to correspond to the specified input pattern; and

an act of using a back propagation algorithm to train the neural network to recognize the specified pattern of input by training the neural network to generate output node values approximating the corresponding output vector in response to receiving the input vector.

2. The method as recited in claim 1, wherein the act of accessing input gesture data for at least one example of a specified input pattern comprises an act of access input gesture data for a plurality of examples of the specified input pattern.

3. The method as recited in claim 1, wherein the act of accessing input gesture data for at least one example of a specified input pattern comprises an act of accessing input gesture data for one example of each of a plurality of different specified input patterns.

4. The method as recited in claim 1, wherein the act of accessing input gesture data for at least one example of a specified input pattern comprises an act of accessing input gesture data for a plurality of examples of each of a plurality of different specified input patterns.

5. The method as recited in claim 1, wherein the act of accessing input gesture data comprises an act of accessing input gesture data for a multi-stroke input pattern.

6. The method as recited in claim 1, further comprising an act of compensating for the rotation of an input gesture from which the input gesture data was created prior to encoding the input gesture data in an input vector of real values.

7. The method as recited in claim 1, wherein the input gesture data also includes third direction movement data, the third direction movement data representing movement along an arbitrary additional third axis over the time period.

8. The method as recited in claim 1, wherein the act of encoding the input gesture data into an input vector of real values comprises taking a plurality of samples of each of the first direction movement data and the second direction movement data at a plurality of designated intervals between the beginning of the time period and the end of the time period.

9. The method as recited in claim 8, wherein using a back propagation algorithm to train the neural network to recognize the specified pattern of input comprises:

an act of submitting the plurality of samples to a corresponding plurality of input nodes of the neural network, the neural network having random link weights assigned from the input nodes to a plurality of hidden nodes and the neural network having random link weights assigned from the plurality of hidden nodes to a plurality of output nodes;

an act of indicating to the neural network that the input gesture data is to cause the neural network to activate a specified output node to at least a specified threshold value, the specified output node assigned, from among the plurality of output nodes, for the specified symbol;

an act of the neural network processing the plurality of samples based on the random link weights to activate values at each of the plurality of output nodes;

an act of determining that the activated value at the specified output node is not at least the specified threshold value;

an act of calculating an error value indicating the difference between the activated value at the specified output node and the specified threshold value; and

an act of propagating the error value backwards from the plurality of output nodes into the neural network to adjust link weights within the neural network, the adjusted link weights causing the neural network to activate the specified output node closer to the specified threshold value when subsequently processing the plurality of samples.

10. The method as recited in claim 9, further comprising subsequent to propagating the error value backwards from the plurality of output nodes into the neural network:

an act of re-submitting the plurality of samples to the corresponding plurality of input nodes of the neural network; and

an act of the neural network processing the plurality of samples based on the adjusted link weights to activate second values at each of the plurality of output nodes.

12. The method as recited in claim 1, wherein the act of saving an indication that the real values contained in the input vector represent an example of the specified input pattern comprise an act of saving an indication that the real values contained in the input vector represent an example of a user-defined input pattern.

13. The method as recited in claim 1, wherein the act of saving an indication that the real values contained in the input vector represent an example of the specified input pattern comprise an act of saving an indication that the real values contained in the input vector represent an example of a input pattern from among a plurality of input patterns in an alphabet.

11. The method as recited in 10, further comprising:

an act of continuing to calculate further error values and propagate the further error values backwards from the plurality of output nodes into the neural network to further adjust link weights within the neural network until adjusted link weights result in a specified output node value that is least the specified threshold value; and

an act of, upon detecting that the specified output node value is at least the specified threshold value, indicating that the neural network is trained to recognize the specified symbol.

14. At a computer system including a multi-touch input display surface, a method for recognizing input gesture data entered at the multi-touch input display surface as a specified symbol, the method comprising:

an act of accessing input gesture data representing detected contact on the multi-touch input display surface over a period of time, the input gesture data including at least;

first direction movement data, the first direction movement data indicating the position of detected contact on the multi-touch input display surface relative to a first axis over the time period; and

second direction movement data, the second direction movement data indicating the position of detected contact on the multi-touch input display surface relative to a second different axis over the time period;

an act of taking a plurality of samples of each of the first direction movement data and the second direction movement data at a plurality of designated intervals between the beginning of the time period and the end of the time period;

an act of submitting the plurality of samples to a corresponding plurality of input nodes of a neural network, the neural network having previously trained link weights from the input nodes to a plurality of hidden nodes and the neural network having previously trained link weights from the plurality of hidden nodes to a plurality of output nodes, each output node assigned to a specified symbol such that an output node being activated to a specified threshold value is indicative of the neural network recognizing input gesture data as the specified symbol;

an act of the neural network processing the plurality of samples based on the previously trained link weights to activate values at each of the plurality of output nodes;

an act of determining that the activated value at the specified output node assigned to the specified symbol is at least the specified threshold value; and

an act of indicating that the specified symbol has been recognized from the input gesture data.

15. The method as recited in claim 14, further comprising compensating for the rotation of the input gesture from which the input gesture data was created prior to taking the plurality of samples.

16. The method as recited in claim 14, wherein the act of submitting the plurality of samples to a corresponding plurality of input nodes of a neural network comprises an act of submitting the plurality of samples to a neural network that was trained to recognize characters of a specified alphabet.

17. The method as recited in claim 14, wherein the input gesture data also includes third direction movement data, the third direction movement data representing movement along a third axis over the time period.

18. The method as recited in claim 14, wherein the act of accessing input gesture data comprises an act of accessing input gesture data for a multi-stroke input gesture.

19. The method as recited in claim 14, further comprising an act of presenting the specified symbol on the multi-touch input display surface.

20. A computer program product for use at a computer system including a multi-touch input display surface, the computer program product form implementing a method for training the computer system to recognize input gesture data entered at the multi-touch input display surface as a corresponding specified input pattern, the computer program product comprising one or more computer-readable media having stored thereon computer-executable instructions that, when executed at a processor, cause the computer system to perform the method, including performing the following:

access input gesture data for at least one example of a specified input pattern, the input gesture data representing detected contact on the multi-touch input display surface over a period of time, the input gesture data including for at least the one example of the specified input pattern;

first direction movement data, the first direction movement data indicating the position of detected contact on the multi-touch input display surface relative to a first axis over the time period; and

second direction movement data, the second direction movement data indicating the position of detected contact on the multi-touch input display surface relative to a second different axis over the time period;

encode the input gesture data into an input vector of real values, each real value corresponding to a sample on the first or second axis;

save an indication that the real values contained in the input vector represent an example of the specified input pattern;

access a fully connected neural network, the neural network including input nodes, hidden nodes, and output nodes, the input nodes being fully connected to the hidden nodes and the hidden nodes being fully connected to the output nodes;

create a training set to supply to the neural network to train the neural network to recognize the specified input pattern, the training set including at least the input vector and a corresponding output vector, the corresponding output vector indicating the desired output of the neural network that is to correspond to the specified input pattern; and

use a back propagation algorithm to train the neural network to recognize the specified pattern of input by training the neural network to generate output node values approximating the corresponding output vector in response to receiving the input vector.