Method and system for gesture recognition

• US 8,290,210 B2
• Filed: 10/15/2009
• Issued: 10/16/2012
• Est. Priority Date: 06/30/2009
• Status: Expired due to Fees

• Current Assignee: National Taiwan University of Science and Technology
• Original Assignee: National Taiwan University of Science and Technology
• Inventors: Fahn, Chin-Shyurng, Yang, Chung-Nan
• Primary Examiner(s): MARIAM, DANIEL G
• Application Number:
  US12/579,409
• Publication Number:
  US 20100329509A1
• Time in Patent Office: 1,097 Days
• Field of Search: 382/103, 382/107, 382/190, 382/165, 382/181, 382/194, 715/863
• US Class Current: 382/103
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A method and a system for gesture recognition are provided for recognizing a gesture performed by a user in front of an electronic product having a video camera. In the present method, an image containing the upper body of the user is captured and a hand area in the image is obtained. The hand area is fully scanned by a first couple of concentric circles. During the scanning, a proportion of a number of skin color pixels on an inner circumference of the first couple of concentric circles and a proportion of a number of skin color pixels on an outer circumference of the first couple of concentric circles are used to determine a number of fingertips in the hand area. The gesture is recognized by the number of fingertips and an operation function of the electronic product is executed according to an operating instruction corresponding to the recognized gesture.

Claims

1. A method of gesture recognition, configured to recognize a gesture performed by a user in front of an electronic product, the method comprising:

capturing an image containing an upper body of the user;

obtaining a hand area in the image;

scanning the entire hand area by placing a first concentric center position of a first couple of concentric circles at a first predetermined position in the hand area and moving the first concentric center position toward a predetermined direction to perform the scan and determining a number of fingertips in the hand area according to a number of skin color pixels on an inner circumference of the first couple of concentric circles and a number of skin color pixels on an outer circumference of the first couple of concentric circles during the scanning, wherein the first predetermined position makes the first couple of concentric circles to be displayed in the hand area completely and an outer circle and an inner circle of the first couple of concentric circles have different sizes;

recognizing the gesture according to the number of fingertips; and

executing an operation function of the electronic product according to an operating instruction corresponding to the recognized gesture, wherein the electronic product comprises a computer system, a computer/television game, and an information appliance.

2. The method of gesture recognition as claimed in claim 1, wherein the capturing of the image containing the upper body of the user comprises:

capturing the image with a video camera equipped on the electronic product.

3. The method of gesture recognition as claimed in claim 2, wherein the scanning of the hand area by the first couple of concentric circles to determine the number of fingertips in the hand area comprises:

determining whether the current first concentric center position is a fingertip position according to a ratio of the number of skin color pixels on the inner circumference of the first couple of concentric circles to a number of pixels included on the inner circumference of the first couple of concentric circles, and a ratio of the number of skin color pixels on the outer circumference of the first couple of concentric circles to a number of pixels included on the outer circumference of the first couple of concentric circles;

moving the first concentric center position toward the predetermined direction by taking a pixel as a unit of movement; and

repeating both the determination of whether the first concentric center position is the fingertip position and the movement of the first concentric center position until a number of the fingertip positions obtained has reached a certain number or the scanning of the hand area has finished, and using the number of the fingertip positions obtained as the number of fingertips.

4. The method of gesture recognition as claimed in claim 3, wherein the determination of whether the current first concentric center position is the fingertip position comprises:

determining whether the number of skin color pixels on the inner circumference of the first couple of concentric circles equals the number of pixels included on the inner circumference of the first couple of concentric circles;

when the number of skin color pixels on the inner circumference of the first couple of concentric circles equals the number of pixels included on the inner circumference of the first couple of concentric circles, determining whether the ratio of the number of skin color pixels on the outer circumference of the first couple of concentric circles to the number of pixels included on the outer circumference of the first couple of concentric circles is smaller than a first predetermined value;

when the ratio of the number of skin color pixels on the outer circumference of the first couple of concentric circles to the number of pixels included on the outer circumference of the first couple of concentric circles is smaller than the first predetermined value, identifying the current first concentric center position to be the fingertip position; and

recording the fingertip position.

5. The method of gesture recognition as claimed in claim 4, wherein after determining whether the number of skin color pixels on the inner circumference of the first couple of concentric circles equals the number of pixels included on the inner circumference of the first couple of concentric circles, the method further comprises:

when the number of skin color pixels on the inner circumference of the first couple of concentric circles does not equal the number of pixels included on the inner circumference of the first couple of concentric circles, then identifying the current first concentric center position not to be the fingertip position.

6. The method of gesture recognition as claimed in claim 4, wherein after determining whether the ratio of the number of skin color pixels on the outer circumference of the first couple of concentric circles to the number of pixels included on the outer circumference of the first couple of concentric circles is smaller than the first predetermined value, the method further comprises:

when the ratio of the number of skin color pixels on the outer circumference of the first couple of concentric circles to the number of pixels included on the outer circumference of the first couple of concentric circles is not smaller than the first predetermined value, identifying the current first concentric center position not to be the fingertip position.

7. The method of gesture recognition as claimed in claim 3, wherein after the number of fingertip positions obtained is used as the number of fingertips, the method further comprises:

overlapping a second concentric center position of a second couple of concentric circles to a second predetermined position in the hand area and moving the second concentric center position toward the predetermined direction to scan the hand area, and obtaining a palm position in the hand area according to the number of skin color pixels on an inner circumference of the second couple of concentric circles and the number of skin color pixels on an outer circumference of the second couple of concentric circles during the scanning, wherein an outer circle and an inner circle of the second couple of concentric circles have different sizes and are both larger than those of the first couple of concentric circles, and the second predetermined position makes the second couple of concentric circles to be displayed in the hand area completely; and

recognizing the gesture according to the number of fingertips and the palm position.

8. The method of gesture recognition as claimed in claim 7, wherein the scanning of the hand area by the second couple of concentric circles to determine the palm position in the hand area comprises:

determining whether the current second concentric center position is the palm position according to a ratio of the number of skin color pixels on the inner circumference of the second couple of concentric circles to the number of pixels included on the inner circumference of the second couple of concentric circles, and a ratio of the number of skin color pixels on the outer circumference of the second couple of concentric circles to the number of pixels included on the outer circumference of the second couple of concentric circles;

moving the second concentric center position toward the predetermined direction by taking a pixel as a unit of movement; and

repeating both the determination of whether the second concentric center position is the palm position and the movement of the second concentric center position until the palm position has been obtained or the scanning of the hand area has finished.

9. The method of gesture recognition as claimed in claim 8, wherein the determination of whether the current second concentric center position is the palm position comprises:

determining whether the number of skin color pixels on the inner circumference of the second couple of concentric circles equals the number of pixels included on the inner circumference of the second couple of concentric circles;

when the number of skin color pixels on the inner circumference of the second couple of concentric circles equals the number of pixels included on the inner circumference of the second couple of concentric circles, determining whether the ratio of the number of skin color pixels on the outer circumference of the second couple of concentric circles to the number of pixels included on the outer circumference of the second couple of concentric circles is greater than a second predetermined value;

when the ratio of the number of skin color pixels on the outer circumference of the second couple of concentric circles to the number of pixels included on the outer circumference of the second couple of concentric circles is greater than the second predetermined value, identifying the current second concentric center position to be the palm position; and

recording the palm, position.

10. The method of gesture recognition as claimed in claim 9, wherein after determining whether the number of skin color pixels on the inner circumference of the second couple of concentric circles equals the number of pixels included on the inner circumference of the second couple of concentric circles, the method further comprises:

when the number of skin color pixels on the inner circumference of the second couple of concentric circles does not equal the number of pixels included on the inner circumference of the second couple of concentric circles, identifying the current second concentric center position not to be the palm position.

11. The method of gesture recognition as claimed in claim 9, wherein after determining whether the ratio of the number of skin color pixels on the outer circumference of the second couple of concentric circles to the number of pixels included on the outer circumference of the second couple of concentric circles is greater than the second predetermined value, the method further comprises:

when the ratio of the number of skin color pixels on the outer circumference of the second couple of concentric circles to the number of pixels included on the outer circumference of the second couple of concentric circles is not greater than the second predetermined value, identifying the current second concentric center position not to be the palm position.

12. The method of gesture recognition as claimed in claim 7, wherein the sizes of the first couple of concentric circles and the second couple of concentric circles are inversely proportional to a distance between the user and the video camera.

13. The method of gesture recognition as claimed in claim 7, wherein the recognition of the gesture according to the number of fingertips and the palm position comprises:

defining n sector areas by referring to the palm position as a center, wherein each of the n sector areas corresponds to a direction respectively and n is a positive integer;

calculating a number of the fingertip positions in each of the n sector areas;

obtaining a specific sector area covering a greatest number of fingertip positions in the n sector areas; and

identifying the gesture to conform with the direction corresponding to the specific sector area.

14. The method of gesture recognition as claimed in claim 13, wherein after the identification of the gesture to conform with the direction corresponding to the specific sector area, the method further comprises:

executing the operation function of the electronic product correspondingly according to the number of fingertips and the direction of a recognized gesture.

15. The method of gesture recognition as claimed in claim 7, wherein the hand area consists of a left hand area and a right hand area, and the method further comprises:

scanning the left hand area by the first couple of concentric circles to determine the number of fingertips in the left hand area;

scanning the right hand area by the first couple of concentric circles to determine the number of fingertips in the right hand area;

recognizing the gesture according to the number of fingertips in the left hand area and the number of fingertips in the right hand area; and

executing the operation function of the electronic product correspondingly according to the recognized gesture.

16. The method of gesture recognition as claimed in claim 15, further comprising:

scanning the left hand area and the right hand area respectively by the first couple of concentric circles to determine the number of fingertips in the left hand area and the number of fingertips in the right hand area;

scanning the left hand area and the right hand area respectively by the second couple of concentric circles to determine the palm position in the left hand area and the palm position in the right hand area;

determining a left hand direction corresponding to the gesture according to the palm position in the left hand area;

determining a right hand direction corresponding to the gesture according to the palm position in the right hand area;

recognizing the gesture according to the number of fingertips in the left hand area and the left hand direction as well as the number of fingertips in the right hand area and the right hand direction; and

executing the operation function of the electronic product correspondingly according to the recognized gesture.

17. The method of gesture recognition as claimed in claim 1, wherein after capturing the image containing the upper body of the user, the method further comprises:

obtaining a predetermined skin color value; and

extracting a plurality of skin color areas coinciding with the predetermined skin color value in the captured image according to the predetermined skin color value.

18. The method of gesture recognition as claimed in claim 17, wherein the acquisition of the hand area in the image comprises:

obtaining three skin color areas possessing top three largest areas from the plurality of skin color areas;

analyzing whether each of the three skin color areas has a facial feature;

labeling the area having the facial feature in the three skin color areas as a face area; and

labeling the area without the facial feature in the three skin color areas as the hand area.

19. The method of gesture recognition as claimed in claim 18, wherein the facial feature includes a hair feature and a texture feature consisting of two eyes, two eye-brows, one nose, one mouth, and two ears.

20. The method of gesture recognition as claimed in claim 1, wherein the operation function at least includes a mouse function, a keyboard function, and a joystick function.

21. A system of gesture recognition, comprising:

a hand area detection module, configured to obtain a hand area in an image captured by an image capture module, wherein the image contains the upper body of a user of an electronic product, and a gesture performed by the user in front of the electronic product, wherein the electronic product comprises a computer system, a computer/television game, and an information appliance;

a fingertip feature capture module, placing a first concentric center position of a first couple of concentric circles at a first predetermined position in the hand area and moving the first concentric center position toward a predetermined direction to scan the entire hand area, and determining a number of fingertips in the hand area according to a number of skin color pixels on an inner circumference of the first couple of concentric circles and a number of skin color pixels on an outer circumference of the first couple of concentric circles during the scanning, wherein the first predetermined position makes the first couple of concentric circles to be displayed in the hand area completely and an outer circle and an inner circle of the first couple of concentric circles have different sizes;

a gesture recognition module, recognizing the gesture according to the number of fingertips; and

a function execution module, executing an operation function of the electronic product according to an operating instruction corresponding to the recognized gesture.

22. The system of gesture recognition as claimed in claim 21, further comprising:

an image pre-processing module, obtaining a predetermined skin color value and extracting a plurality of skin color areas coinciding with the predetermined skin color value in the captured image according to the predetermined skin color value.

23. The system of gesture recognition as claimed in claim 22, wherein the hand area detection module obtains three skin color areas possessing the top three largest areas from the plurality of skin color areas, analyzes which one of the three skin color areas has a facial feature, labels the area having the facial feature in one of the three skin color areas as a face area, and labels the area without the facial feature as the hand area, the facial feature includes a hair feature and a texture feature consisting of two eyes, two eye-brows, one nose, one mouth, and two ears.

24. The system of gesture recognition as claimed in claim 21, wherein the fingertip feature capture module determines whether the current first concentric center position is a fingertip position according to a ratio of the number of skin color pixels on the inner circumference of the first couple of concentric circles to a number of pixels included on the inner circumference of the first couple of concentric circles, and a ratio of the number of skin color pixels on the outer circumference of the first couple of concentric circles to a number of pixels included on the outer circumference of the first couple of concentric circles;

when the current first concentric center position is not a fingertip position, moves the first concentric center position toward the predetermined direction by taking a pixel as a unit of movement;

the fingertip feature capture module repeats both the determination of whether the first concentric center position is the fingertip position and the movement of the first concentric center position until the number of the fingertip positions obtained has reached a certain number or the scanning of the hand area has finished, and uses the number of the fingertip positions obtained as the number of fingertips,

the fingertip feature capture module determines whether the number of skin color pixels on the inner circumference of the first couple of concentric circles equals the number of pixels included on the inner circumference of the first couple of concentric circles;

when the number of skin color pixels on the inner circumference of the first couple of concentric circles equals the number of pixels included on the inner circumference of the first couple of concentric circles, the fingertip feature capture module determines whether the ratio of the number of skin color pixels on the outer circumference of the first couple of concentric circles to the number of pixels on the outer circumference of the first couple of concentric circles is smaller than a first predetermined value;

when the ratio of the number of skin color pixels on the outer circumference of the first couple of concentric circles to the number of pixels on the outer circumference of the first couple of concentric circles is smaller than the first predetermined value, then the fingertip feature capture module identifies the current first concentric center position to be the fingertip position and records the fingertip position,

when the fingertip feature capture module determines the number of skin color pixels on the inner circumference of the first couple of concentric circles to be different from the number of pixels included on the inner circumference of the first couple of concentric circles, the current first concentric center position is identified not to be the fingertip position,

when the fingertip feature capture module determines the ratio of the number of skin color pixels on the outer circumference of the first couple of concentric circles to the number of pixels included on the outer circumference of the first couple of concentric circles to be greater than or equal to the first predetermined value, the current first concentric center position is identified not to be the fingertip position.

25. The system of gesture recognition as claimed in claim 24, further comprising:

a palm feature capture module, overlapping a second concentric center position of a second couple of concentric circles to a second predetermined position in the hand area and moving the second concentric center position toward the predetermined direction to scan the hand area, and obtaining a palm position in the hand area according to the number of skin color pixels on an inner circumference of the second couple of concentric circles and the number of skin color pixels on an outer circumference of the second couple of concentric circles during the scanning, wherein an inner circle and an outer circle of the second couple of concentric circles have different sizes and are both larger than those of the first couple of concentric circles and the gesture recognition module recognizes the gesture by the number of fingertips and the palm position, wherein the second predetermined position makes the second couple of concentric circles to be displayed in the hand area completely,

wherein the palm feature capture module determines whether the current second concentric center position is the palm position according to the ratio of the number of skin color pixels on the inner circumference of the second couple of concentric circles to the number of pixels included on the inner circumference of the second couple of concentric circles, and the ratio of the number of skin color pixels on the outer circumference of the second couple of concentric circles to the number of pixels included on the outer circumference of the second couple of concentric circles, when the current second concentric center position is not the palm position, moves the second concentric center position toward the predetermined direction by taking a pixel as a unit of movement;

the palm feature capture module repeats both the determination of whether the second concentric center position is the palm position and the movement of the second concentric center position until the palm position has been obtained or the scanning of the hand area has finished,

the palm feature capture module determines whether the number of skin color pixels on the inner circumference of the second couple of concentric circles equals the number of pixels included on the inner circumference of the second couple of concentric circles;

when the number of skin color pixels on the inner circumference of the second couple of concentric circles equals the number of pixels included on the inner circumference of the second couple of concentric circles, determines whether the ratio of the number of skin color pixels on the outer circumference of the second couple of concentric circles to the number of pixels included on the outer circumference of the second couple of concentric circles is greater than a second predetermined value;

when the ratio of the number of skin color pixels on the outer circumference of the second couple of concentric circles to the number of pixels included on the outer circumference of the second couple of concentric circles is greater than the second predetermined value, then identifies the current second concentric center position to be the palm position and records the palm position,

when the palm feature capture module determines the number of skin color pixels on the inner circumference of the second couple of concentric circles to be different from the number of pixels included on the inner circumference of the second couple of concentric circles, the current second concentric center position is identified not to be the palm position,

wherein when the palm feature capture module determines the ratio of the number of skin color pixels on the outer circumference of the second couple of concentric circles to the number of pixels included on the outer circumference of the second couple of concentric circles to be smaller than or equal to the second predetermined value, the current second concentric center position is identified not to be the palm position.

26. The system of gesture recognition as claimed in claim 25, wherein the sizes of the first couple of concentric circles and the second couple of concentric circles are inversely proportional to a distance between the user and the video camera.

27. The system of gesture recognition as claimed in claim 25, wherein the gesture recognition module defines n sector areas using the palm position as a center, wherein each of the n sector areas corresponds to a direction respectively and n is a positive integer;

calculates a number of the fingertip positions in each of then sector areas;

in the n sector areas, obtains a specific sector area covering a greatest number of fingertip positions and identifies the gesture to conform with the direction corresponding to the specific sector area, the function execution module correspondingly executes the operation function of the electronic product according to the number of fingertips and the direction of a recognized gesture.

28. The system of gesture recognition as claimed in claim 25, wherein the hand area consists of a left hand area and a right hand area, and the fingertip feature capture module scans the left hand area by the first couple of concentric circles to determine the number of fingertips in the left hand area and scans the right hand area by the first couple of concentric circles to determine the number of fingertips in the right hand area, then the gesture recognition module recognizes the gesture according to the number of fingertips in the left hand area and the number of fingertips in the right hand area, and the function execution module executes the operation function of the electronic product correspondingly according to the recognized gesture.

29. The system of gesture recognition as claimed in claim 28, wherein the fingertip feature capture module scans the left hand area and the right hand area respectively by the first couple of concentric circles to determine the number of fingertips in the left hand area and the number of fingertips in the right hand area;

the palm feature capture module scans the left hand area and the right hand area respectively by the second couple of concentric circles to determine the palm position in the left hand area and the palm position in the right hand area;

the gesture recognition module determines a left hand direction corresponding to the gesture according to the palm position in the left hand area and determines a right hand direction corresponding to the gesture according to the palm position in the right hand area, and recognizes the gesture according to the number of fingertips in the left hand area and the left hand direction as well as the number of fingertips in the right hand area and the right hand direction;

the function execution module executes the operation function of the electronic product correspondingly according to the recognized gesture.

30. The system of gesture recognition as claimed in claim 28, wherein the operation function at least possesses a mouse function and a keyboard function of the computer system as well as a joystick function of the computer/television game, and further includes an operation interface of the information appliance.

31. The system of gesture recognition as claimed in claim 21, further comprising:

a recognition result display module, when the operation function correspondingly executed by the function execution module according to the recognized gesture is the mouse function, the recognition result display module demonstrates a virtual mouse panel having a size mapped to a monitor of the electronic product to notify a cursor position of the electronic product.