APPARATUS AND METHOD OF HAND GESTURE RECOGNITION BASED ON DEPTH IMAGE

• US 20170068849A1
• Filed: 02/24/2016
• Published: 03/09/2017
• Est. Priority Date: 09/03/2015
• Status: Active Grant

• Current Assignee: Korea Advanced Institute of Science and Technology
• Original Assignee: Korea Advanced Institute of Science and Technology
• Inventors: LIM, Hwasup, CHUN, Sungkuk, AHN, Sang Chul
• Granted Patent: US 10,078,796 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 10/751 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
• CPC Class Codes: G06V 10/762 using clustering, e.g. of similar faces in social networks
• CPC Class Codes: G06V 40/113 Recognition of static hand signs

Abstract

Disclosed is an apparatus for hand gesture recognition based on a depth image, which includes a depth image acquiring unit configured to acquire a depth image including a hand region, a depth point classifying unit configured to classify depth points of a hand region in the depth image according to a corresponding hand portion by means of a machine studying method, and a hand model matching unit configured to match a three-dimensional hand model with the classified depth points by using distances between the classified depth points and a hand portion respectively corresponding to the depth points. A recognition method using the apparatus is also disclosed.

Claims

1. An apparatus for hand gesture recognition based on a depth image, comprising:

a depth image acquiring unit configured to acquire a depth image including a hand region;

a depth point classifying unit configured to classify depth points of the hand region in the depth image according to a corresponding hand portion by means of a machine studying method; and

a hand model matching unit configured to match a three-dimensional hand model with the classified depth points by using distances between the classified depth points and a hand portion respectively corresponding to the depth points.

2. The apparatus for hand gesture recognition based on a depth image according to claim 1,

wherein the depth point classifying unit classifies depth points of the hand region into any one hand portion among a thumb, a forefinger, a middle finger, a ring finger, a little finger and a palm and labels each of the depth points according to the classification.

3. The apparatus for hand gesture recognition based on a depth image according to claim 1,

wherein the machine studying method includes at least one of a support vector machine, a random forest and a neural network theory.

4. The apparatus for hand gesture recognition based on a depth image according to claim 2,

wherein the hand model matching unit matches the three-dimensional hand model so that a sum of distances between a plurality of depth points and a three-dimensional hand model respectively corresponding to the plurality of depth points is minimized.

5. The apparatus for hand gesture recognition based on a depth image according to claim 4,

wherein the corresponding three-dimensional hand model is a phalanx portion.

6. The apparatus for hand gesture recognition based on a depth image according to claim 1,

wherein the hand model matching unit matches the three-dimensional hand model by additionally using angle information between phalanges included in a finger.

7. The apparatus for hand gesture recognition based on a depth image according to claim 6,

wherein the hand model matching unit matches the three-dimensional hand model so that angles between phalanges of a single finger are proportional to each other.

8. The apparatus for hand gesture recognition based on a depth image according to claim 1,

wherein the hand model matching unit matches the three-dimensional hand model by additionally using a constraint condition for constraining the three-dimensional hand model to have a finger posture which is expressible by a human or a constraint condition for constraining a hand space including depth points of the hand region.

9. The apparatus for hand gesture recognition based on a depth image according to claim 2,

wherein when an unclassified hand portion is present among the thumb, the forefinger, the middle finger, the ring finger and the little finger, the hand model matching unit matches the corresponding three-dimensional hand model having any posture with the unclassified hand portion.

10. The apparatus for hand gesture recognition based on a depth image according to claim 9,

wherein the corresponding three-dimensional hand model having any posture includes a spring model at a finger joint.

11. The apparatus for hand gesture recognition based on a depth image according to claim 1,

wherein the depth point classifying unit generates hand skeleton depth points representing a hand skeleton by using the depth points of the hand region in the depth image and classifies the hand skeleton depth points according to a corresponding hand portion, and

wherein the hand model matching unit matches the three-dimensional hand model with the classified hand skeleton depth points.

12. The apparatus for hand gesture recognition based on a depth image according to claim 1,

wherein the depth point classifying unit moves the depth points of the hand region in a normal direction of a surface of the hand to be located in a hand skeleton region, clusters the depth points located in the hand skeleton region, and determines a representative depth point of each cluster, and

wherein the hand model matching unit matches the hand model with the representative depth points.

13. A method for hand gesture recognition based on a depth image, comprising:

acquiring a depth image including a hand region;

classifying depth points of the hand region in the depth image according to a corresponding hand portion by means of a machine studying method; and

matching a three-dimensional hand model with the classified depth points by using distances between the classified depth points and a hand portion respectively corresponding to the depth points.

14. The method for hand gesture recognition based on a depth image according to claim 13,

wherein the classifying of depth points according to a corresponding hand portion includes:

classifying depth points of the hand region into any one hand portion among a thumb, a forefinger, a middle finger, a ring finger, a little finger and a palm; and

labeling each of the depth points according to the classification.

15. The method for hand gesture recognition based on a depth image according to claim 13,

wherein the machine studying method includes at least one of a support vector machine, a random forest and a neural network theory.

16. The method for hand gesture recognition based on a depth image according to claim 14,

wherein the matching of a three-dimensional hand model includes:

matching the three-dimensional hand model so that a sum of distances between a plurality of depth points and a three-dimensional hand model respectively corresponding to the plurality of depth points is minimized.

17. The method for hand gesture recognition based on a depth image according to claim 16,

wherein the corresponding three-dimensional hand model is a phalanx portion.

18. The method for hand gesture recognition based on a depth image according to claim 13,

wherein the matching of a three-dimensional hand model includes:

matching the three-dimensional hand model by additionally using angle information between phalanges included in a finger.

19. The method for hand gesture recognition based on a depth image according to claim 18,

wherein the matching of a three-dimensional hand model includes:

matching the three-dimensional hand model so that angles between phalanges of a single finger are proportional to each other.

20. The method for hand gesture recognition based on a depth image according to claim 13,

wherein the matching of a three-dimensional hand model further includes:

applying a constraint condition for constraining the three-dimensional hand model to have a finger posture which is expressible by a human, or

applying a constraint condition for constraining a hand space including depth points of the hand region.

21. The method for hand gesture recognition based on a depth image according to claim 14,

wherein the matching of a three-dimensional hand model further includes:

matching the corresponding three-dimensional hand model having any posture with an unclassified hand portion, when the unclassified hand portion is present among the thumb, the forefinger, the middle finger, the ring finger and the little finger.

22. The method for hand gesture recognition based on a depth image according to claim 20,

wherein the corresponding three-dimensional hand model having any posture includes a spring model at a finger joint.

23. The method for hand gesture recognition based on a depth image according to claim 13,

wherein the classifying of depth points according to a corresponding hand portion includes:

generating hand skeleton depth points representing a hand skeleton by using the depth points of the hand region in the depth image and classifying the hand skeleton depth points according to a corresponding hand portion, and

wherein the matching of a three-dimensional hand model includes;

matching the three-dimensional hand model with the classified hand skeleton depth points.

24. The method for hand gesture recognition based on a depth image according to claim 13,

wherein the classifying of depth points according to a corresponding hand portion includes:

moving the depth points of the hand region in a normal direction of a surface of the hand to be located in a hand skeleton region, clustering the depth points located in the hand skeleton region, and determining a representative depth point of each cluster, and

wherein the matching of a three-dimensional hand model includes;

matching the three-dimensional hand model with the representative depth points.

25. A computer-readable recording medium, on which a program including instructions for executing the method according to claim 13 is recorded.