Gesture detection and recognition

• US 9,619,035 B2
• Filed: 03/04/2011
• Issued: 04/11/2017
• Est. Priority Date: 03/04/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Nowozin, Sebastian, Kohli, Pushmeet, Shotton, Jamie Daniel Joseph
• Primary Examiner(s): Laneau, Ronald
• Assistant Examiner(s): WILLIAMS, ROSS A
• Application Number:
  US13/040,487
• Publication Number:
  US 20120225719A1
• Time in Patent Office: 2,230 Days
• Field of Search: 463 35- 37, 463 40- 42, 382103, 382181, 382220, 382226, 715863
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

A gesture detection and recognition technique is described. In one example, a sequence of data items relating to the motion of a gesturing user is received. A selected set of data items from the sequence are tested against pre-learned threshold values, to determine a probability of the sequence representing a certain gesture. If the probability is greater than a predetermined value, then the gesture is detected, and an action taken. In examples, the tests are performed by a trained decision tree classifier. In another example, the sequence of data items can be compared to pre-learned templates, and the similarity between them determined. If the similarity for a template exceeds a threshold, a likelihood value associated with a future time for a gesture associated with that template is updated. Then, when the future time is reached, the gesture is detected if the likelihood value is greater than a predefined value.

Claims

1. A computer-implemented method of gesture detection, comprising:

receiving a sequence of data items relating to motion of a gesturing user as captured by one or more image capture devices;

testing, by a hardware processor of a computing device, a plurality of selected data items from the sequence against predefined threshold values to determine a probability of the sequence representing a gesture, testing comprising;

applying the sequence to a trained decision tree, applying the sequence to the trained decision tree comprising passing the sequence through a plurality of nodes in the tree until a leaf node is reached in the tree, and wherein the probability of the sequence representing a gesture is determined in dependence on the leaf node reached in the decision tree;

applying the sequence to at least one further, distinct trained decision tree; and

determining the probability of the sequence representing the gesture by averaging the probabilities from each of the trained decision trees;

detecting the gesture if the probability is greater than a predetermined value; and

controlling the computing device at least partly in response to detecting the gesture.

2. A method as claimed in claim 1, further comprising executing a command responsive to detecting the gesture.

3. A method as claimed in claim 1, wherein the data items in the sequence represent at least one of:

i) an angle between at least two body parts of the user;

ii) a rate of change of angle between at least two body parts of the user;

iii) a motion velocity for at least one body part of the user;

iv) a feature derived from a depth image of the user;

v) a feature derived from an RGB image of the user; and

vi) a speech signal from the user.

4. A method as claimed in claim 3, further comprising generating data items i), ii) or iii) using a body-pose estimator taking an input from a depth camera capturing a sequence of depth images of the user.

5. A method as claimed in claim 1, wherein the data items in the sequence represent at least one of inertia, velocity, orientation and location of a mobile device.

6. A method as claimed in claim 5, further comprising generating the data items from data provided by at least one of:

an accelerometer;

an inertial measurement unit;

a compass; and

a gyroscope located in the mobile device.

7. A method as claimed in claim 1, wherein each node of the decision tree is associated with an index value and at least one of the predefined threshold values, and applying the sequence to the trained decision tree comprises comparing, at each node, the data item in the sequence located at the index value to the at least one of the predefined threshold values to determine a subsequent node to which to send the sequence.

8. A method as claimed in claim 1, further comprising training the decision tree prior to receiving the sequence, wherein training the decision tree comprises:

selecting a node of the decision tree;

selecting at least one annotated training sequence;

generating a plurality of random index values and random threshold values;

comparing data items from the annotated training sequence located at each of the random index values to each of the random threshold values to obtain a plurality of results;

selecting a chosen index value and at least one chosen threshold value for the node in dependence on the plurality of results; and

storing the chosen index value and the at least one chosen threshold value in association with the node at a storage device.

9. A method according to claim 8, wherein selecting a chosen index value and at least one chosen threshold value comprises determining an information gain for each of the plurality of results, and selecting the chosen index value and at least one chosen threshold value associated with the result having maximum information gain.

10. A method according to claim 9, further comprising repeating the method until the maximum information gain is less than a predefined minimum value or the node of the decision tree has a maximum predefined depth.

11. A computer-implemented method of gesture detection, comprising:

receiving a sequence of data items relating to motion of a gesturing user as captured by one or more image capture devices;

comparing, by a hardware processor of a computing device, the sequence of data items to a plurality of templates, each associated with a gesture, and generating a measure of similarity for each;

in the case that the measure of similarity for a template is greater than a threshold, updating a likelihood value associated with a future time for the gesture associated with that template, updating the likelihood value comprising aggregating a weight value associated with the template with additional weight values previously stored at the future time for the gesture; and

when the future time is reached, detecting the gesture if the likelihood value is greater than a predefined value and controlling the computing device based on the gesture.

12. A method as claimed in claim 11, further comprising, prior to receiving the sequence, learning the plurality of templates and associated thresholds and future times for each of the templates using a logistic regression algorithm.

13. A method as claimed in claim 11, wherein generating a measure of similarity comprises determining a Euclidean distance value between the sequence of data items and the template.

14. A computer-implemented method of gesture detection, comprising:

receiving a sequence of data items relating to motion of a gesturing user as captured by one or more image capture devices;

testing, by a hardware processor of a computing device, a plurality of selected data items from the sequence against predefined threshold values to determine a probability of the sequence representing a gesture, testing comprising applying the sequence to a trained decision tree;

detecting the gesture if the probability is greater than a predetermined value and in response to detecting the gesture, controlling the computing device; and

training the decision tree prior to receiving the sequence, training the decision tree comprising;

selecting a node of the decision tree;

selecting at least one annotated training sequence;

generating a plurality of random index values and random threshold values;

comparing data items from the annotated training sequence located at each of the random index values to each of the random threshold values to obtain a plurality of results;

selecting a chosen index value and at least one chosen threshold value for the node in dependence on the plurality of results; and

storing the chosen index value and the at least one chosen threshold value in association with the node at a storage device.

15. A method according to claim 14, wherein selecting a chosen index value and at least one chosen threshold value comprises determining an information gain for each of the plurality of results, and selecting the chosen index value and at least one chosen threshold value associated with the result having maximum information gain.

16. A method according to claim 15, further comprising repeating the method until the maximum information gain is less than a predefined minimum value or the node of the decision tree has a maximum predefined depth.

17. A method as claimed in claim 14, further comprising executing a command responsive to detecting the gesture.

18. A method as claimed in claim 14, wherein the data items in the sequence of data items represent at least one of:

i) an angle between at least two body parts of the user;

ii) a rate of change of angle between at least two body parts of the user;

iii) a motion velocity for at least one body part of the user;

iv) a feature derived from a depth image of the user;

v) a feature derived from an RGB image of the user; and

vi) a speech signal from the user.

19. A method as claimed in claim 18, further comprising generating data items i), ii) or iii) using a body-pose estimator taking an input from a depth camera capturing a sequence of depth images of the user.

20. A method as claimed in claim 14, wherein the data items in the sequence of data items represent at least one of inertia, velocity, orientation and location of a mobile device.

21. A system, comprising:

at least one memory; and

at least one processor configured to operate with the at least one memory to;

receive a sequence of data items relating to motion of a gesturing user as captured by one or more image capture devices; and

at least one of;

test a plurality of selected data items from the sequence against predefined threshold values to determine a probability of the sequence representing a gesture at least by applying the sequence to a trained decision tree, applying the sequence to the trained decision tree comprising passing the sequence through a plurality of nodes in the tree until a leaf node is reached in the tree, and wherein the probability of the sequence representing a gesture is determined in dependence on the leaf node reached in the decision tree, applying the sequence to at least one further, distinct trained decision tree, and determining the probability of the sequence representing the gesture by averaging the probabilities from each of the trained decision trees;

detect the gesture if the probability is greater than a predetermined value; and

control a computing device at least partly in response to detecting the gesture;

or

compare the sequence of data items to a plurality of templates, each associated with the gesture, and generating a measure of similarity for each;

in the case that the measure of similarity for a template is greater than a threshold, updating a likelihood value associated with a future time for the gesture associated with that template, updating the likelihood value comprising aggregating a weight value associated with the template with additional weight values previously stored at the future time for the gesture; and

when the future time is reached, detecting the gesture if the likelihood value is greater than a predefined value and controlling the computing device based on the gesture;

or

test the plurality of selected data items from the sequence against the predefined threshold values to determine the probability of the sequence representing the gesture at least by applying the sequence to a trained decision tree;

detect the gesture if the probability is greater than the predetermined value and in response to detecting the gesture, controlling the computing device; and

train the decision tree prior to receiving the sequence by at least selecting a node of the decision tree, selecting at least one annotated training sequence, generating a plurality of random index values and random threshold values, comparing data items from the annotated training sequence located at each of the random index values to each of the random threshold values to obtain a plurality of results, selecting a chosen index value and at least one chosen threshold value for the node in dependence on the plurality of results, and storing the chosen index value and the at least one chosen threshold value in association with the node at a storage device.

22. The system of claim 21, wherein the at least one processor is configured to:

test the plurality of selected data items from the sequence of data items against predefined threshold values to determine the probability of the sequence of data items representing the gesture at least by applying the sequence of data items to the trained decision tree, applying the sequence of data items to the trained decision tree comprising passing the sequence of data items through a plurality of nodes in the tree until a leaf node is reached in the tree, and wherein the probability of the sequence of data items representing a gesture is determined in dependence on the leaf node reached in the decision tree, applying the sequence of data items to at least one further, distinct trained decision tree, and determining the probability of the sequence of data items representing the gesture by averaging the probabilities from each of the trained decision trees;

detect the gesture if the probability is greater than a predetermined value; and

control the computing device at least partly in response to detecting the gesture.

23. The system of claim 21, wherein the at least one processor is configured to:

compare the sequence of data items to a plurality of templates, each associated with the gesture, and generating a measure of similarity for each;

in the case that the measure of similarity for a template is greater than a threshold, updating a likelihood value associated with a future time for the gesture associated with that template, updating the likelihood value comprising aggregating a weight value associated with the template with additional weight values previously stored at the future time for the gesture; and

when the future time is reached, detecting the gesture if the likelihood value is greater than a predefined value and controlling the computing device based on the gesture.

24. The system of claim 21, wherein the at least one processor is configured to:

test the plurality of selected data items from the sequence of data items against the predefined threshold values to determine the probability of the sequence of data items representing the gesture at least by applying the sequence of data items to a trained decision tree;

detect the gesture if the probability is greater than the predetermined value and in response to detecting the gesture, controlling the computing device; and

train the decision tree prior to receiving the sequence of data items by at least selecting a node of the decision tree, selecting at least one annotated training sequence, generating a plurality of random index values and random threshold values, comparing data items from the annotated training sequence located at each of the random index values to each of the random threshold values to obtain a plurality of results, selecting a chosen index value and at least one chosen threshold value for the node in dependence on the plurality of results, and storing the chosen index value and the at least one chosen threshold value in association with the node at a storage device.

25. The system of claim 21, where in the at least one processor is further configured to execute a command responsive to detecting the gesture.

26. The system of claim 21, wherein the data items in the sequence of data items represent at least one of:

i) an angle between at least two body parts of the user;

ii) a rate of change of angle between at least two body parts of the user;

iii) a motion velocity for at least one body part of the user;

iv) a feature derived from a depth image of the user;

v) a feature derived from an RGB image of the user; and

vi) a speech signal from the user.

27. The system of claim 26, wherein the at least one processor is further configured to generate data items i), ii) or iii) using a body-pose estimator taking an input from a depth camera capturing a sequence of depth images of the user.

28. The system of claim 21, wherein the data items in the sequence of data items represent at least one of inertia, velocity, orientation and location of a mobile device.

29. The system of claim 28, wherein the at least one processor is further configured to generate the data items from data provided by at least one of:

an accelerometer;

an inertial measurement unit;

a compass; and

a gyroscope located in the mobile device.

30. The system of claim 21, wherein each node of the decision tree is associated with an index value and at least one of the predefined threshold values, and applying the sequence of data items to the trained decision tree includes comparing, at each node, the data item in the sequence of data items located at the index value to the at least one of the predefined threshold values to determine a subsequent node to which to send the sequence of data items.

31. The system of claim 21, wherein the at least one processor is further configured to train the decision tree prior to receiving the sequence of data items, at least by:

selecting a node of the decision tree;

selecting at least one annotated training sequence;

generating a plurality of random index values and random threshold values;

comparing data items from the annotated training sequence located at each of the random index values to each of the random threshold values to obtain a plurality of results;

selecting a chosen index value and at least one chosen threshold value for the node in dependence on the plurality of results; and

storing the chosen index value and the at least one chosen threshold value in association with the node at a storage device.

32. The system of claim 31, wherein selecting a chosen index value and at least one chosen threshold value comprises determining an information gain for each of the plurality of results, and selecting the chosen index value and at least one chosen threshold value associated with the result having maximum information gain.

33. The system of claim 32, wherein the at least one processor is further configured to determine whether the maximum information gain is less than a predefined minimum value or the node of the decision tree has a maximum predefined depth.

34. The system of claim 21, wherein the at least one processor is further configured to, prior to receiving the sequence of data items, learn the plurality of templates and associated thresholds and future times for each of the templates using a logistic regression algorithm.

35. The system of claim 21, wherein generating the measure of similarity includes determining a Euclidean distance value between the sequence of data items and the template.