System and method for dynamic gesture recognition using geometric classification

• US 8,620,024 B2
• Filed: 09/17/2010
• Issued: 12/31/2013
• Est. Priority Date: 09/17/2010
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Huang, Kuang-Man, Liu, Ming-Chang, Yu, Liangyin
• Primary Examiner(s): BHATNAGAR, ANAND P
• Application Number:
  US12/885,104
• Publication Number:
  US 20120068917A1
• Time in Patent Office: 1,201 Days
• Field of Search: 382/100, 382/103, 382/106, 382/107, 382/162, 382/164, 382170-172, 382/181, 382/190, 382/195, 382/199, 382/201, 382/206, 382/224, 348169-172, 340/5.1, 340/5.2, 340/5.82, 340/5.83
• US Class Current: 382/103
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A gesture recognition system and method that inputs videos of a moving hand and outputs the recognized gesture states for the input sequence. In each image, the hand area is segmented from the background and used to estimate parameters of all five fingers. The system further classifies the hand image as one of the postures in the pre-defined database and applies a geometric classification algorithm to recognize the gesture. The system combines a skin color model with motion information to achieve real-time hand segmentation performance, and considers each dynamic gesture as a multi-dimensional volume and uses a geometric algorithm to classify each volume.

Claims

1. A gesture recognition system for remote control of a device, comprising:

(a) a sensor for capturing video data of a user'"'"'s hand at a location near said device;

(b) a processor for processing the captured video data of said hand; and

(c) programming executable on said processor for performing steps comprising;

segmenting each hand image in the captured video based on skin color;

extracting a parameter of the segmented hand image;

said parameter relating to one of a plurality of fingers in the hand image;

tracking the parameter in the hand image;

quantizing the parameter for each of the plurality of fingers to a plurality of states by comparing the parameter against one or more thresholds;

classifying a hand gesture based on the quantized states of the one or more fingers; and

operating the device based on recognition of said hand gesture.

2. A system as recited in claim 1, wherein segmenting each hand image comprises:

separating a foreground area associated with said hand from a background area surrounding said foreground area;

training a 3-component mixture model of the background area and foreground area; and

classifying the image pixels based on one or more of motion and skin color.

3. A system as recited in claim 1, wherein the image is segmented using only Cb and Cr color channels.

4. A system as recited in claim 1, wherein extracting one or more parameters comprises:

obtaining a palm center location of the user'"'"'s hand based on an extracted contour and skeleton of the hand.

5. A system as recited in claim 4, wherein tracking the at least one of the one or more parameters comprises tracking the palm center location using a Kalman filter.

6. A system as recited in claim 4, wherein extracting the parameter further comprises:

obtaining a palm bottom location;

approximating a location of a fingertip of at least one of the plurality of fingers based on said contour; and

measuring the distance between the palm bottom location and each of the approximated fingertip locations.

7. A system as recited in claim 6, wherein quantizing the parameter to a plurality of states comprises:

comparing the measured distance for each finger to first and second threshold values;

classifying a finger as being in a first state if the measured distance for the finger is below the first threshold value;

classifying the finger as being in a third state if the measured distance for the finger is above the second threshold value; and

classifying the finger as being in a second state if the measured distance for the finger is between the first threshold value and the second threshold value.

8. A system as recited in claim 7, wherein the first state comprises a “

closed”

hand gesture, the second state comprises a “

half-open”

hand gesture, and the third state comprises an “

open”

hand gesture.

9. A system as recited in claim 8, wherein classifying a hand gesture comprises comparing the quantized states against a database of predetermined hand gestures.

10. A system as recited in claim 1, wherein the parameter is quantized to a maximum of three states.

11. A system as recited in claim 10, wherein the hand image further comprises:

a thumb four fingers;

wherein a parameter corresponding to the thumb is quantized to two states; and

wherein a parameter corresponding to each of the four fingers is quantized to three states.

12. A method for remotely controlling a device using hand gestures, the device having a user interface for visualization on a display, the method comprising:

capturing video of a user'"'"'s hand with a sensor at a location near said device;

segmenting each hand image in the captured video based on skin color;

extracting a parameter of the segmented hand image;

said parameter relating to one of a plurality of fingers in the hand image;

tracking the parameter in the hand image;

quantizing the parameter for each of the plurality of fingers to a plurality of states by comparing the parameter against one or more thresholds;

classifying a hand gesture based on the quantized states of the one or more fingers;

sending a command to the user interface based on recognition of said hand gesture and tracking of said one or more parameters; and

outputting the sent command on the display to operate the device.

13. A method as recited in claim 12, wherein the parameter is quantized to a maximum of three states.

14. A method as recited in claim 13:

wherein the hand image further comprises a thumb four fingers;

wherein a parameter corresponding to the thumb is quantized to two states; and

wherein a parameter corresponding to each of the four fingers is quantized to three states.

15. A method as recited in claim 14, wherein extracting the parameter comprises:

obtaining a palm bottom location;

approximating a location of a fingertip of at least one of the plurality of fingers based on said contour; and

measuring the distance between the palm bottom location and each of the approximated fingertip locations.

16. A method as recited in claim 15, wherein quantizing the parameter to a plurality of states comprises:

comparing the measured distance for each finger to first and second threshold values; and

classifying a finger as being in a first state if the measured distance for the finger is below the first threshold value;

classifying the finger as being in a third state if the measured distance for the finger is above the second threshold value; and

classifying the finger as being in a second state if the measured distance for the finger is between the first threshold value and the second threshold value.

17. A method as recited in claim 16, wherein the first state comprises a “

closed”

hand gesture, the second state comprises a “

half-open”

hand gesture, and the third state comprises an “

open”

hand gesture.

18. A method as recited in claim 12, wherein classifying a hand gesture comprises comparing the quantized states against a database of predetermined hand gestures.

19. A gesture recognition system for remote control of a device having a user interface for visualization on a display, comprising:

(a) a sensor for capturing video data of a user'"'"'s hand at a location near said device;

(b) a processor for processing the captured video data of said hand;

(c) programming executable on said processor for carrying out steps comprising;

segmenting each hand image in the captured video based on skin color;

extracting a parameter of the segmented hand image;

said parameter relating to one of a plurality of fingers in the hand image;

tracking the parameter in the hand image;

quantizing the parameter for each of the plurality of fingers to a maximum of three states by comparing the parameter against one or more thresholds;

classifying a hand gesture based on the quantized states of the one or more fingers; and

operating the device based on recognition of said hand gesture and tracking of a second parameter relating to said hand image;

wherein operating the device comprises sending a command to the user interface based on recognition of said hand gesture and tracking of said second parameter.

20. A system as recited in claim 19, wherein extracting the parameter comprises:

obtaining a palm bottom location;

approximating a location of a fingertip of at least one of the plurality of fingers based on said contour; and

measuring the distance between the palm bottom location and each of the approximated fingertip locations.