Gesture recognition system and method thereof

• US 8,269,722 B2
• Filed: 08/21/2009
• Issued: 09/18/2012
• Est. Priority Date: 02/02/2009
• Status: Active Grant

• Current Assignee: Industrial Technology Research Institute
• Original Assignee: Industrial Technology Research Institute
• Inventors: Wu, Jun-Mein, Luo, Wen-Shiou, Ho, Wei-Yih, Chen, Chia-Chen, Tang, Cheng-Yuan
• Primary Examiner(s): Agustin, Peter Vincent
• Application Number:
  US12/545,340
• Publication Number:
  US 20100194679A1
• Time in Patent Office: 1,124 Days
• Field of Search: None
• US Class Current: 345/158
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

A gesture recognition system includes an image pick-up device, a processor, an operation engine, an optimal template selection means, and a display terminal. The image pick-up device is for capturing an image containing a natural gesture. The processor is for finding out a skin edge of a skin part from the image, and then classifying the skin edge into multiple edge parts at different angles. The operation engine has multiple parallel operation units and multiple gesture template libraries of different angle classes. These parallel operation units respectively find out gesture templates most resembling the edge parts in the gesture template libraries of different angle classes. The optimal template selection means selects an optimal gesture template from the resembling gesture templates found out by the parallel operation units. The display terminal is for displaying an image of the optimal gesture template. Thereby, marker-less and real-time gesture recognition is achieved.

Claims

1. A gesture recognition system, comprising:

an image pick-up device, for capturing an image data containing a hand image;

a template database, for recording multiple gesture templates representing different gestures, wherein the gesture templates are classified by angles and are respectively stored in gesture template libraries of different angle classes;

a processor, for communicating with the image pick-up device and obtaining the image data, finding out a skin part from the image data, producing a skin edge by using an edge detection means, and then classifying the skin edge into multiple edge parts of different angle classes according to angles of the skin edge;

an operation engine, having multiple parallel operation units respectively for performing template matching at different angles, wherein the edge parts of different angle classes are respectively sent to different parallel operation units for template matching, so as to find out the gesture templates most resembling the edge parts in the corresponding gesture template libraries of different angle classes;

an optimal template selection means, for further selecting an optimal gesture template from the resembling gesture templates found out by the parallel operation units; and

a display terminal, for displaying a gesture image represented by the optimal gesture template.

2. The gesture recognition system according to claim 1, wherein the image data is a color image.

3. The gesture recognition system according to claim 1, wherein the gesture template records a gesture edge of a gesture.

4. The gesture recognition system according to claim 3, wherein edge pixels in the gesture edge are classified by angles, the edge pixels in the same angle range are gathered to form the gesture template, and the gesture templates of different angles are respectively stored in the template database with different angle classes.

5. The gesture recognition system according to claim 1, wherein the skin part is filtered from the image by using a skin processing method, and the skin edge of the skin part is found out with an edge detector.

6. The gesture recognition system according to claim 2, wherein the skin part is filtered from the image by using a skin processing method, the skin part is then converted into a gray-scale image, and the skin edge of the skin part is found out with an edge detector.

7. The gesture recognition system according to claim 5, wherein the edge detector is a Sobel filter.

8. The gesture recognition system according to claim 1, wherein sets of the edge pixels within the same angle ranges are respectively found out from the edge pixels constituting the skin edge, then stored as the edge parts, and further classified into the edge parts of different angle classes according to the angles of the edge pixels contained in the edge parts.

9. The gesture recognition system according to claim 8, wherein gradients of the edge pixels of the skin edge in horizontal and vertical directions are calculated by using the edge detector to obtain the angles, and then the edge pixels are classified into the edge parts at different angles according to the angles.

10. The gesture recognition system according to claim 1, wherein the skin edge is classified into eight angle ranges equally dividing a 360°

round angle.

11. The gesture recognition system according to claim 1, wherein the processor is a personal computer (PC);

the parallel operation units of the operation engine are multi-core processors;

the optimal template selection means is a computer;

the display terminal is a computer capable of drawing, and is used for displaying the gesture image represented by the gesture template;

the processor sends the edge parts simultaneously to the multi-core processors through User Datagram Protocol broadcast (UDP broadcast), and the multi-core processors only perform template matching on the edge parts consistent with the pre-allocated angle ranges;

the communication between the multi-core processors in the operation engine and the optimal template selection means, as well as the communication between the optimal template selection means and the display terminal is realized through Transmission Control Protocol/Internet Protocol (TCP/IP) for data transmission.

12. The gesture recognition system according to claim 1, wherein in template matching, a similarity between the gesture template and the edge part is obtained by using a distance transform (DT) technology.

13. The gesture recognition system according to claim 12, wherein the DT technology performs a DT operation on the edge part to produce a DT image, and searches each position in the DT image by using the gesture template, then performs a convolution operation on the gesture template and the DT image at each of the search positions to generate an approximate value representing a similarity between the gesture template and the DT image, and finds out the gesture template with the smallest approximate value, and finally sends the gesture template with the smallest approximate value to the optimal template selection means.

14. The gesture recognition system according to claim 1, wherein in template matching, several predetermined equally divided positions of the gesture template in the edge part are first searched to find out a scope where the hand is most likely to appear, and then a fine template matching is performed on the scope.

15. The gesture recognition system according to claim 14, wherein the scope where the hand is most likely to appear refers to a scope with a number of the appearing skin pixels greater than a threshold.

16. A gesture recognition method, comprising:

a. establishing a template database, storing gesture edges representing different gestures as gesture templates, and classifying the gesture templates by angles and storing the classified gesture templates into gesture template libraries of different angle classes;

b. capturing an image data containing a hand image;

c. finding out a skin part from the image, and performing an edge detection on the skin part to obtain a skin edge of the skin part;

d. classifying by angles the skin edge into multiple edge parts of different angle classes;

e. matching the edge parts of different angle classes with the gesture templates in the gesture template libraries of different angle classes, so as to find out the gesture templates most resembling the edge parts in the gesture template libraries of different angle classes, respectively;

f. finding out an optimal gesture template from the resembling gesture templates at different angles found out in the previous step; and

g. displaying a gesture image represented by the optimal gesture template found out in the previous step.

17. The gesture recognition method according to claim 16, wherein the image data is a color image.

18. The gesture recognition method according to claim 16, wherein the step of establishing the template database is to classify edge pixels constituting the gesture edges according to the angle of each edge pixel in the gesture edges, gather the edge pixels within the same angle range to form the gesture template, and then store the gesture templates of different angles in the template database with different angle classes, respectively.

19. The gesture recognition method according to claim 16, wherein a skin part is filtered from the color digital image by using a skin processing method, and the skin edge of the skin part is found out with an edge detection filter.

20. The gesture recognition method according to claim 17, wherein a skin part is filtered from the color digital image by using a skin processing method, the skin part is then converted into a gray-scale image, and the skin edge of the skin part is found out with an edge detection filter.

21. The gesture recognition method according to claim 20, wherein sets of the edge pixels within the same angle ranges are respectively found out from the edge pixels constituting the skin edge, then stored as the edge parts, and further classified into the edge parts of different angle classes according to the angles of the edge pixels contained in the edge parts.

22. The gesture recognition method according to claim 21, wherein gradients of the edge pixels of the skin edge in horizontal and vertical directions are calculated by using an edge detector to obtain the angles, and then the edge pixels are classified into the edge parts at different angles according to the angles.

23. The gesture recognition method according to claim 19, wherein the edge detector is a Sobel filter.

24. The gesture recognition method according to claim 16, wherein the skin edge is classified into eight angle ranges equally dividing a 360°

round angle.

25. The gesture recognition method according to claim 16, wherein a similarity between the gesture template and the edge part is obtained by using a distance transform (DT) technology.

26. The gesture recognition method according to claim 25, wherein the DT technology performs a DT operation on the edge part to produce a DT image, and searches each position in the DT image by using the gesture template, then performs a convolution operation on the gesture template and the DT image at each of the search positions to generate an approximate value representing a similarity between the gesture template and the DT image, and finds out the gesture template with the smallest approximate value, and finally sends the gesture template with the smallest approximate value to the optimal template selection means.

27. The gesture recognition method according to claim 16, wherein several predetermined equally divided positions of the gesture template in the edge part are first searched to find out a scope where the hand is most likely to appear, and then a fine template matching is performed on the scope.

28. The gesture recognition method according to claim 27, wherein the scope where the hand is most likely to appear refers to a scope with a number of the appearing skin pixels greater than a threshold.