Dynamic Hand Gesture Recognition Using Depth Data

• US 20130336524A1
• Filed: 06/18/2012
• Published: 12/19/2013
• Est. Priority Date: 06/18/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Zhengyou, Kurakin, Alexey Vladimirovich
• Granted Patent: US 9,536,135 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06F 18/295 Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/85 Markov-related models; Markov random fields
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

The subject disclosure is directed towards a technology by which dynamic hand gestures are recognized by processing depth data, including in real-time. In an offline stage, a classifier is trained from feature values extracted from frames of depth data that are associated with intended hand gestures. In an online stage, a feature extractor extracts feature values from sensed depth data that corresponds to an unknown hand gesture. These feature values are input to the classifier as a feature vector to receive a recognition result of the unknown hand gesture. The technology may be used in real time, and may be robust to variations in lighting, hand orientation, and the user'"'"'s gesturing speed and style.

Claims

1. In a computing environment, a method performed at least in part on at least one processor comprising:

sensing depth data for a plurality of frames that include hand movement;

for each frame of at least some of the frames, processing the depth data for that frame, including detecting a hand represented in the depth data, and extracting feature values corresponding to the hand; and

recognizing the hand movement as a hand gesture based upon the feature values.

2. The method of claim 1 wherein processing the depth data comprises compensating for orientation of the hand.

3. The method of claim 1 wherein processing the depth data comprises compensating for a relative size of the hand.

4. The method of claim 1 wherein detecting the hand comprises segmenting the depth data into a human shape, and detecting the hand based upon depth data of the hand relative to depth data of the human shape.

5. The method of claim 4 wherein detecting the hand comprises refining an object that include and arm portion and a hand portion to remove the arm portion.

6. The method of claim 1 wherein detecting the hand comprises determining a plurality of hypothesized hand regions, and determining a hand region from among the hypothesized hand regions based upon processing one or more previous frames of depth data.

7. The method of claim 1 wherein extracting the feature values corresponding to the hand comprises extracting feature values based upon hand velocity data, one or more hand rotation parameters, or at least one shape descriptor, or any combination of hand velocity data, one or more hand rotation parameters, or at least one shape descriptor.

8. The method of claim 1 wherein extracting the feature values corresponding to the hand comprises extracting shape descriptor feature values based upon one or more occupancy features.

9. The method of claim 1 wherein extracting the feature values corresponding to the hand comprises extracting shape descriptor feature values based upon one or more silhouette features.

10. A system comprising, a classifier trained from feature values extracted from frames of depth data that are associated with intended hand gestures, and a feature extractor configured to extract feature values from sensed depth data that corresponds to an unknown hand gesture, at least some of the feature values extracted from the sensed depth data represented as a feature data input to the classifier to receive a recognition result of the unknown hand gesture.

11. The system of claim 10 wherein the feature data input to the classifier comprises a feature vector.

12. The system of claim 10 further comprising a hand segmenter configured to isolate a hand represented in the depth data from other depth data, wherein the feature extractor extracts the features based upon the depth data corresponding to the hand.

13. The system of claim 12 further comprising a data normalization component configured to orient the hand to a desired orientation or to scale the hand to a desired size, or both to orient the hand to a desired orientation and to scale the hand to a desired size.

14. The system of claim 10 wherein the feature extractor extracts at least one of:

a hand velocity feature value set, a hand rotation feature value set, or a hand shape descriptor feature set.

15. The system of claim 10 wherein the feature extractor extracts hand shape descriptor feature set comprising one or more occupancy features, or one or more silhouette features, or both one or more occupancy features and one or more silhouette features.

16. The system of claim 10 wherein the classifier comprises an action graph classifier.

17. One or more computer-readable media having computer-executable instructions, which when executed perform steps, comprising:

processing sensed depth data for a plurality of frames that include hand movement, including segmenting the depth data to isolate a hand represented in the frames of depth data, and extracting feature values corresponding to the hand; and

recognizing the hand movement as a hand gesture based upon the feature values.

18. The one or more computer-readable media of claim 17 wherein processing the sensed depth data further comprises compensating for orientation of the hand or a relative size of the hand, or compensating for both orientation of the hand and a relative size of the hand.

19. The one or more computer-readable media of claim 17 wherein extracting the feature values corresponding to the hand comprises extracting a hand velocity feature value set, a hand rotation feature value set, or a hand shape descriptor feature set.

20. The one or more computer-readable media of claim 17 having further computer-executable instructions, comprising, training a classifier with feature values extracted from frames of depth data that are associated with intended hand gestures, and wherein recognizing the hand movement as a hand gesture based upon the feature values comprises inputting a feature vector representative of the feature values to the classifier.