Gesture recognition system for TV control

• US 9,213,890 B2
• Filed: 09/17/2010
• Issued: 12/15/2015
• Est. Priority Date: 09/17/2010
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Huang, Kuang-Man, Liu, Ming-Chang, Yu, Liangyin
• Primary Examiner(s): Ustaris, Joseph
• Assistant Examiner(s): SECHSER, JILL D
• Application Number:
  US12/885,072
• Publication Number:
  US 20120069168A1
• Time in Patent Office: 1,915 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06V 10/34 Smoothing or thinning of the pattern; Morphological operations; Skeletonisation
• CPC Class Codes: G06V 10/426 Graphical representations
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence

Abstract

A gesture recognition system using a skin-color based method combined with motion information to achieve real-time segmentation. A Kalman filter is used to track the centroid of the hand. The palm center, palm bottom, as well as the largest distance from the palm center to the contour from extracted hand mask are computed. The computed distance to a threshold is then compared to decide if the current posture is “open” or “closed.” In a preferred embodiment, the transition between the “open” and “closed” posture to decide if the current gesture is in “select” or “grab” state.

Claims

1. A gesture recognition system for remote control of a device, comprising:

(a) a sensor for capturing video data of a user'"'"'s hand at a location near the device;

(b) a processor for processing the captured video data of the hand; and

(c) programming executable on said processor for carrying out steps comprising;

segmenting each hand image in the captured video based on skin color;

extracting one or more parameters of the segmented hand image;

wherein the one or more parameters comprises a palm center location of the user'"'"'s hand based on an extracted contour and skeleton of the hand;

wherein said palm center location comprises a center of the user'"'"'s palm during an “

open”

configuration of the user'"'"'s hand;

said “

open”

configuration corresponding to a condition wherein all of the user'"'"'s fingers are extended to expose the user'"'"'s palm;

wherein the one or more parameters further comprises a palm bottom location;

measuring the distance between the palm center location and a furthest contour point on the extracted image contour;

the furthest contour point being within a predetermined angular range of a vector passing through the palm bottom location and the palm center location;

tracking the one or more parameters in the hand image;

classifying a hand gesture as “

open”

or “

closed”

based on the one or more parameters; and

operating the device based on recognition of said hand gesture and tracking the position of said one or more parameters.

2. A system as recited in claim 1, wherein segmenting each hand image comprises:

separating a foreground area associated with said hand from a background area surrounding said foreground area;

training a 3-component mixture model of the background area and foreground area; and

classifying the image pixels based on one or more of motion and skin color.

3. A system as recited in claim 1, wherein the image is segmented using only Cb and Cr color channels.

4. A system as recited in claim 1, wherein tracking the one or more parameters comprises tracking the palm center location using a Kalman filter.

5. A system as recited in claim 1, wherein classifying a hand gesture as “

open”

or “

closed”

comprises;

comparing the distance to the furthest contour point to a threshold value;

classifying the image as “

open”

if the distance to the furthest contour point is above the threshold value; and

classifying the image as “

closed”

if the distance to the furthest contour point is below the threshold value.

6. A system as recited in claim 1:

wherein the device comprises a user interface coupled to a display, the user interface being output on said display; and

wherein operating the device comprises sending a command to the user interface based on recognition of said hand gesture and tracking of said one or more parameters.

7. A system as recited in claim 6, wherein a transition between an “

open”

hand gesture and a “

closed”

hand gesture is used to determine whether “

select”

or “

grab”

command is sent to the user interface.

8. A system as recited in claim 7, wherein tracking of an open hand gesture is use to determine a location of a cursor on the user interface.

9. A gesture recognition system for remote control of a device having a user interface for visualization on a display, comprising:

(a) a sensor for capturing video data of a user'"'"'s hand at a location near the device;

(b) a processor for processing the captured video data of the hand;

(c) programming executable on said processor for carrying out steps comprising;

segmenting each hand image in the captured video based on skin color;

extracting one or more parameters of the segmented hand image;

wherein the one or more parameters comprises a palm center location of the user'"'"'s hand based on an extracted contour and skeleton of the hand;

wherein said palm center location comprises a center of the user'"'"'s palm during an “

open”

configuration of the user'"'"'s hand;

said “

open”

configuration corresponding to a condition wherein all of the user'"'"'s fingers are extended to expose the user'"'"'s palm;

wherein the one or more parameters further comprises a palm bottom location;

measuring the distance between the palm center location and a furthest contour point on the extracted image contour;

the furthest contour point being within a predetermined angular range of a vector passing through the palm bottom location and the palm center location;

tracking the one or more parameters in the hand image;

classifying a hand gesture as “

open”

or “

closed”

based on said one or more parameters; and

operating the device based on recognition of said hand gesture and tracking of said one or more parameters;

wherein operating the device comprises sending a command to the user interface based on recognition of said hand gesture and tracking the position of said one or more parameters.

10. A system as recited in claim 9, wherein segmenting each hand image comprises:

separating a foreground area associated with said hand from a background area surrounding said foreground area;

training a 3-component mixture model comprising only Cb and Cr color channels of the background area and foreground area; and

classifying the image pixels based on one or more of motion and skin color.

11. A system as recited in claim 9, wherein tracking the one or more parameters comprises tracking the palm center location using a Kalman filter.

12. A system as recited in claim 9, wherein classifying a hand gesture as “

open”

or “

closed”

comprises;

comparing the distance to the furthest contour point to a threshold value;

classifying the image as “

open”

if the distance to the furthest contour point is above the threshold value; and

classifying the image as “

closed”

if the distance to the furthest contour point is below the threshold value.

13. A system as recited in claim 12, wherein a transition between an “

open”

hand gesture and a “

closed”

hand gesture is used to determine whether a “

select”

or “

grab”

command is sent to the user interface.

14. A system as recited in claim 13, wherein tracking of an open hand gesture is use to determine a location of a cursor on the user interface.

15. A method for remotely controlling a device using hand gestures, the device having a user interface for visualization on a display, the method comprising:

capturing video of a user'"'"'s hand with a sensor at a location near said device;

segmenting each hand image in the captured video based on skin color;

extracting one or more parameters of the segmented hand image;

wherein the one or more parameters comprises a palm center location of the user'"'"'s hand based on an extracted contour and skeleton of the hand;

wherein said palm center location comprises a center of the user'"'"'s palm during an “

open”

configuration of the user'"'"'s hand;

said “

open”

configuration corresponding to a condition wherein all of the user'"'"'s fingers are extended to expose the user'"'"'s palm;

wherein the one or more parameters further comprises a palm bottom location;

measuring the distance between the palm center location and a furthest contour point on the extracted image contour;

the furthest contour point being within a predetermined angular range of a vector passing through the palm bottom location and the palm center location;

tracking the at least one of the one or more parameters in the hand image;

classifying a hand gesture as “

open”

or “

closed”

based on said one or more parameters;

wherein classifying a hand gesture as “

open”

or “

closed”

comprises;

comparing the distance to the furthest contour point to a threshold value; and

classifying the image as “

open”

if the distance to the furthest contour point is above the threshold value; and

classifying the image as “

closed”

if the distance to the furthest contour point is below the threshold value;

operating the device based on recognition of said hand gesture and tracking of said one or more parameters;

sending a command to the user interface based on recognition of said hand gesture and tracking of said one or more parameters; and

outputting the sent command on the display to operate the device.

16. A method as recited in claim 15:

wherein a transition between an “

open”

hand gesture and a “

closed”

hand gesture is used to determine whether a “

select”

or “

grab”

command is sent to the user interface; and

wherein tracking the palm center of an open hand gesture is use to determine a location of a cursor on the user interface.