Apparatus and method of hand gesture recognition based on depth image

• US 10,078,796 B2
• Filed: 02/24/2016
• Issued: 09/18/2018
• Est. Priority Date: 09/03/2015
• Status: Active Grant

• Current Assignee: Korea Advanced Institute of Science and Technology
• Original Assignee: Korea Advanced Institute of Science and Technology
• Inventors: Lim, Hwasup, Chun, Sungkuk, Ahn, Sang Chul
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): Shin, Brian
• Application Number:
  US15/052,063
• Publication Number:
  US 20170068849A1
• Time in Patent Office: 937 Days
• Field of Search: None
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 10/751 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
• CPC Class Codes: G06V 10/762 using clustering, e.g. of similar faces in social networks
• CPC Class Codes: G06V 40/113 Recognition of static hand signs

Abstract

Disclosed is an apparatus for hand gesture recognition based on a depth image, which includes a depth image acquiring unit configured to acquire a depth image including a hand region, a depth point classifying unit configured to classify depth points of a hand region in the depth image according to a corresponding hand portion by means of a machine studying method, and a hand model matching unit configured to match a three-dimensional hand model with the classified depth points by using distances between the classified depth points and a hand portion respectively corresponding to the depth points. A recognition method using the apparatus is also disclosed.

Claims

1. An apparatus for hand gesture recognition, comprising:

a processor configured to

acquire a depth image comprising a region of a hand performing a gesture,

classify depth points of the region in the depth image according to a corresponding hand portion,

match a three-dimensional hand model with the classified depth points, by using distances between the classified depth points and the corresponding hand portion, and by minimizing a sum of distances between a plurality of depth points and the three-dimensional hand model, and

track the gesture in an animation including the matched three-dimensional hand model.

2. The apparatus of claim 1, further comprising a memory configured to store instructions,

wherein the processor is further configured to execute the instructions to

acquire the depth image comprising the region of the hand performing the gesture,

classify the depth points of the region in the depth image according to the corresponding hand portion,

match the three-dimensional hand model with the classified depth points, by using the distances between the classified depth points and the corresponding hand portion, and by minimizing the sum of the distances between the plurality of depth points and the three-dimensional hand model, and

track the gesture in an animation including the matched three-dimensional hand model.

3. The apparatus of claim 1, wherein the processor comprises

a depth image acquiring unit configured to acquire the depth image comprising the region of the hand performing the gesture,

a depth point classifying unit configured to classify the depth points of the region in the depth image according to the corresponding hand portion, and

a hand model matching unit configured to

match the three-dimensional hand model with the classified depth points, by using the distances between the classified depth points and the corresponding hand portion, and by minimizing the sum of the distances between the plurality of depth points and the three-dimensional hand model, and

track the gesture in the animation including the matched three-dimensional hand model.

4. The apparatus of claim 3, wherein the depth points of the region in the depth image are classified according to the corresponding hand portion by a machine studying method.

5. The apparatus according to claim 4, wherein the depth point classifying unit is further configured to classify the depth points of the region into any one of a thumb, a forefinger, a middle finger, a ring finger, a little finger, and a palm, and label each of the depth points of the region according to the classification.

6. The apparatus according to claim 5, wherein the hand model matching unit is further configured to match the three-dimensional hand model with an unclassified hand portion, in response to the unclassified hand portion being present among the thumb, the forefinger, the middle finger, the ring finger, and the little finger.

7. The apparatus according to claim 6, wherein the three-dimensional hand model comprises a spring model at a finger joint.

8. The apparatus according to claim 4, wherein the machine studying method comprises any one or any combination of any two or more of a support vector machine, a random forest, and a neural network theory.

9. The apparatus according to claim 4, wherein the three-dimensional hand model comprises a phalanx portion.

10. The apparatus according to claim 4, wherein the hand model matching unit is further configured to match the three-dimensional hand model by using angle information between phalanges of a finger.

11. The apparatus according to claim 10, wherein the hand model matching unit is further configured to match the three-dimensional hand model so that angles between the phalanges of the finger are proportional to each other.

12. The apparatus according to claim 4, wherein the hand model matching unit is further configured to match the three-dimensional hand model by constraining the three-dimensional hand model to have a finger posture which is expressible by a human or constraining a hand space comprising points among the depth points of the region.

13. The apparatus according to claim 4, wherein

the depth point classifying unit is further configured to generate hand skeleton depth points representing a hand skeleton by using the depth points of the region in the depth image, and classify the hand skeleton depth points according to another corresponding hand portion, and

the hand model matching unit is further configured to match the three-dimensional hand model with the classified hand skeleton depth points.

14. The apparatus according to claim 4, wherein

the depth point classifying unit is further configured to move the depth points of the region in a normal direction of a surface of a hand to be located in a hand skeleton region, cluster the depth points located in the hand skeleton region into clusters, and determine a representative depth point of each cluster, and

the hand model matching unit is further configured to match the hand model with the representative depth points.

15. The apparatus of claim 1, further comprising:

a depth camera configured to capture the depth image; and

a display configured to display the matched three-dimensional hand model in a frame of the animation.

16. A method for hand gesture recognition, comprising:

acquiring a depth image comprising a region of a hand performing a gesture;

classifying depth points of the region in the depth image according to a corresponding hand portion;

matching a three-dimensional hand model with the classified depth points by using distances between the classified depth points and the corresponding hand portion; and

tracking the gesture in an animation including the matched three-dimensional hand model,

wherein the matching comprises matching the three-dimensional hand model so that angles between phalanges of a finger are proportional to each other.

17. The method according to claim 16, wherein the classifying of depth points comprises

classifying the depth points of the region into any one of a thumb, a forefinger, a middle finger, a ring finger, a little finger, and a palm, and

labeling each of the depth points of the region according to a result of the classifying of the depth points of the region.

18. The method according to claim 17, wherein the matching of the three-dimensional hand model comprises matching the three-dimensional hand model so that a sum of distances between the depth points and the corresponding three-dimensional hand model is minimized.

19. The method according to claim 18, wherein the three-dimensional hand model comprises a phalanx portion.

20. The method according to claim 17, wherein the matching of the three-dimensional hand model comprises matching the three-dimensional hand model with an unclassified hand portion, in response to the unclassified hand portion being present among the thumb, the forefinger, the middle finger, the ring finger, and the little finger.

21. The method according to claim 16, wherein the matching of a the three-dimensional hand model comprises constraining the three-dimensional hand model to have a finger posture which is expressible by a human, or constraining a hand space comprising points among the depth points of the region.

22. The method according to claim 21, wherein the corresponding three-dimensional hand model comprises a spring model at a finger joint.

23. The method according to claim 16, wherein

the classifying of the depth points comprises generating hand skeleton depth points representing a hand skeleton by using the depth points of the region in the depth image, and classifying the hand skeleton depth points according to another corresponding hand portion, and

the matching of the three-dimensional hand model comprises matching the three-dimensional hand model with the classified hand skeleton depth points.

24. The method according to claim 16, wherein

the classifying of depth points comprises moving the depth points of the region in a normal direction of a surface of the hand to be located in a hand skeleton region, clustering the depth points located in the hand skeleton region into clusters, and determining a representative depth point of each cluster, and

the matching of the three-dimensional hand model comprises matching the three-dimensional hand model with the representative depth points.

25. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 16.

26. The method of claim 16, wherein the classifying of the depth points comprises classifying the depth points of the region in the depth image according to the corresponding hand portion by a machine studying method.

27. The method according to claim 26, wherein the machine studying method comprises any one or any combination of any two or more of a support vector machine, a random forest, and a neural network theory.

28. The method of claim 16, further comprising displaying the matched three-dimensional hand model in a frame of the animation.

29. The method of claim 28, further comprising repeating the acquiring, the classifying, and the matching, for additional depth images, so that additional three-dimensional hand models are matched, wherein the animation includes additional frames in which the matched additional three-dimensional hand models are respectively displayed.

30. The method of claim 29, wherein gesture is recognized through the matching of the three-dimensional hand model and the matching of the additional three-dimensional hand models.

31. A method for hand gesture recognition based on a depth image, comprising:

acquiring a depth image comprising a region of a hand performing a gesture;

classifying depth points of the region in the depth image according to a corresponding hand portion by means of a machine studying method;

matching a three-dimensional hand model with the classified depth points by using distances between the classified depth points and the hand portion, which respectively corresponds to the depth points; and

tracking the gesture in an animation including the matched three-dimensional hand model,

wherein the classifying of the depth points of the region comprises classifying the depth points of the region into any one of a thumb, a forefinger, a middle finger, a ring finger, a little finger and a palm, and labeling each of the depth points according to the classification, and

wherein the matching of the three-dimensional hand model comprises matching the three-dimensional hand model so that a sum of distances between a plurality of depth points and the three-dimensional hand model, which respectively corresponds to the plurality of depth points, is minimized.