Gesture Detection and Recognition

• US 20120225719A1
• Filed: 03/04/2011
• Published: 09/06/2012
• Est. Priority Date: 03/04/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Kohli, Pushmeet, Shotton, Jamie Daniel Joseph, Nowozin, Sebastian
• Granted Patent: US 9,619,035 B2
• Time in Patent Office: Days
• US Class Current: 463/36
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

A gesture detection and recognition technique is described. In one example, a sequence of data items relating to the motion of a gesturing user is received. A selected set of data items from the sequence are tested against pre-learned threshold values, to determine a probability of the sequence representing a certain gesture. If the probability is greater than a predetermined value, then the gesture is detected, and an action taken. In examples, the tests are performed by a trained decision tree classifier. In another example, the sequence of data items can be compared to pre-learned templates, and the similarity between them determined. If the similarity for a template exceeds a threshold, a likelihood value associated with a future time for a gesture associated with that template is updated. Then, when the future time is reached, the gesture is detected if the likelihood value is greater than a predefined value.

Claims

1. A computer-implemented method of gesture detection, comprising:

receiving, at a processor, a sequence of data items relating to motion of a gesturing user;

testing a plurality of selected data items from the sequence against predefined threshold values to determine a probability of the sequence representing a gesture; and

detecting the gesture if the probability is greater than a predetermined value.

2. A method as claimed in claim 1, further comprising the step of executing a command responsive to detecting the gesture.

3. A method as claimed in claim 1, wherein the data items in the sequence represent at least one of:

i) an angle between at least two body parts of the user;

ii) a rate of change of angle between at least two body parts of the user;

iii) a motion velocity for at least one body part of the user;

iv) a feature derived from a depth image of the user;

v) a feature derived from an RGB image of the user; and

vi) a speech signal from the user.

4. A method as claimed in claim 3, further comprising the step of generating data items i), ii) or iii) using a body-pose estimator taking an input from a depth camera capturing a sequence of depth images of the user.

5. A method as claimed in claim 1, wherein the data items in the sequence represent at least one of the inertia, velocity, orientation and location of a mobile device.

6. A method as claimed in claim 5, further comprising the step of generating the data items from data provided by at least one of:

an accelerometer;

an inertial measurement unit;

a compass; and

a gyroscope located in the mobile device.

7. A method as claimed in claim 1, wherein the step of testing comprises applying the sequence to a trained decision tree.

8. A method as claimed in claim 7, wherein the step of applying the sequence to the trained decision tree comprises passing the sequence through a plurality of nodes in the tree until a leaf node is reached in the tree, and wherein the probability of the sequence representing a gesture is determined in dependence on the leaf node reached in the decision tree.

9. A method as claimed in claim 8, wherein each node of the decision tree is associated with an index value and at least one of the predefined threshold values, and the step of applying the sequence to the trained decision tree comprises comparing, at each node, the data item in the sequence located at the index value to the at least one of the predefined threshold values to determine a subsequent node to which to send the sequence.

10. A method as claimed in claim 8, wherein the step of testing further comprises applying the sequence to at least one further, distinct trained decision tree, and determining the probability of the sequence representing the gesture by averaging the probabilities from each of the trained decision trees.

11. A method as claimed in claim 7, further comprising the step of training the decision tree prior to receiving the sequence, wherein the step of training the decision tree comprises:

selecting a node of the decision tree;

selecting at least one annotated training sequence;

generating a plurality of random index values and random threshold values;

comparing data items from the annotated sequence located at each of the random index values to each of the random threshold values to obtain a plurality of results;

selecting a chosen index value and at least one chosen threshold value for the node in dependence on the plurality of results; and

storing the chosen index value and the at least one chosen threshold value in association with the node at a storage device.

12. A method according to claim 11, wherein the step of selecting a chosen index value and at least one chosen threshold value comprises determining an information gain for each of the plurality of results, and selecting the chosen index value and at least one chosen threshold value associated with the result having the maximum information gain.

13. A method according to claim 12, further comprising repeating the steps of the method until the maximum information gain is less than a predefined minimum value or the node of the decision tree has a maximum predefined depth.

20. A game application at least partially stored in memory and at least partially executed on a processor to perform the method of claim 1.

14. A computer-implemented method of gesture detection, comprising:

receiving, at a processor, a sequence of data items relating to motion of a gesturing user;

comparing the sequence of data items to a plurality of templates, each associated with a gesture, and generating a measure of similarity for each;

in the case that the measure of similarity for a template is greater than a threshold, updating a likelihood value associated with a future time for the gesture associated with that template; and

when the future time is reached, detecting the gesture if the likelihood value is greater than a predefined value.

15. A method as claimed in claim 14, further comprising the step of, prior to receiving the sequence, learning the plurality of templates and associated thresholds and future times for each of the templates using a logistic regression algorithm.

16. A method as claimed in claim 14, wherein the step of generating a measure of similarity comprises determining a Euclidean distance value between the sequence of data items and the template.

17. A method as claimed in claim 14, wherein the step of updating the likelihood value comprises aggregating a weight value associated with the template with additional weight values previously stored at the future time for the gesture.

18. A gesture recognition system, comprising

an input interface arranged to receive a sequence of depth images of a user;

a memory arranged to store a random decision forest comprising a plurality of distinct trained decision trees; and

a processor arranged to;

generate a sequence of data items relating to motion of the user from the depth images;

apply the sequence of data items to each of the trained decision trees to obtain a plurality of probabilities of the sequence representing one of a plurality of predefined gestures;

aggregate the probabilities from each of the trained decision trees; and

execute a command associated with a detected gesture if the aggregated probability for that gesture is greater than a predetermined value.

19. A computer game system comprising the gesture recognition system as claimed in claim 18.