System and method for hand gesture recognition for remote control of an internet protocol TV

• US 8,582,037 B2
• Filed: 03/09/2011
• Issued: 11/12/2013
• Est. Priority Date: 03/11/2010
• Status: Active Grant

• Current Assignee: Deutsche Telekom AG
• Original Assignee: Deutsche Telekom AG
• Inventors: Stern, Helman, Berman, Sigal, Frolova, Daria, Stegmann, Joachim, Weiss, Christian, Lonczewski, Frank
• Primary Examiner(s): Harold, Jefferey
• Assistant Examiner(s): SANDERS, JUSTIN B
• Application Number:
  US13/043,645
• Publication Number:
  US 20110221974A1
• Time in Patent Office: 979 Days
• Field of Search: 348/734
• US Class Current: 348/734
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/66 of image moments or centre of gravity
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Gesture recognition system for remote controlling a TV is disclosed. The system comprises: (a) a webcam for capturing a video sequence of a user which is composed of video frames (b) a hand tracker for receiving the video sequence, and when a hand is determined within a frame of the video sequence, the hand tracker calculates the hand centroid; (c) a trajectory buffer for receiving plurality of the hand centroids as determined from plurality of frames respectively; (d) trajectory segmentation module for continuously inspecting the buffer for possible inclusion of relevant trajectories that relate to a hand gesture pattern, and whenever a relevant trajectory is detected, it extracts a respective trajectory segment, and conveys the segment into a gesture classifier; and (e) the gesture classifier for verifying whether the segment relates to a specific gesture command, and when affirmative, the gesture classifier transfers the respective command to the TV.

Claims

1. Gesture recognition system for remote controlling a TV using a webcam and low-grade computational equipment, which comprises:

a single webcam for capturing video sequence of a user which is composed of video frames, and conveying the same into a hand tracker;

hand tracker for receiving said video sequence, inspecting each of said video frames for possible inclusion of a user hand within it, and when the hand is determined to be found in a frame from said video frames, calculating a centroid of the hand, and outputting coordinates of the hand centroid into a trajectory buffer;

the trajectory buffer for receiving from said hand tracker each of said hand centroid coordinates, and maintaining in storage at least several tens of recent hand centroids, that altogether define a trajectory of the hand;

trajectory segmentation module for continuously inspecting said buffer for possible inclusion of relevant trajectories of at least pre-defined lengths that are suspected to relate to a hand gesture pattern, and whenever a relevant trajectory is detected, extracting a trajectory segment of the same, and conveying it into a gesture classifier; and

the gesture classifier for extracting from each received trajectory segment global feature parameters of the same, comparing said extracted global feature parameters with gesture global predefinitions, concluding whether the segment relates to a specific gesture command, and in the affirmative case, transferring the command to the TV;

wherein said hand tracker comprises a switch for switching the hand tracker between a color-mode and a motion-mode, wherein during a stationary state the hand tracker is in color-mode inspecting the scene to identify existence of the user hand based on an expected color of the user hand, and upon such user hand identification, the switch switches the hand tracker to a motion-mode for tracking a motion of the user hand by inspecting the scene using a dynamic fusion of both color and motion cues as determined by frame differencing, said hand tracker remains in the motion mode as long as an amount of motion between proximate centroids is determined to be above a motion energy threshold, and furthermore, said motion energy is inspected only within a window which is set around said calculated centroid.

2. The gesture recognition system according to claim 1, wherein the buffer maintains in the storage 100-300 of the recent hand centroids.

3. The gesture recognition system according to claim 1, wherein said global feature parameters comprises one or more of:

aspect ratio of the segment, variability of the segment, start and end point locations of the segment, intersection of the segment, and switch back traces.

4. The gesture recognition system according to claim 1, wherein the hand tracker bases its operation on hand predefinitions, and on statistical calculations relating to the possibility that each pixel in the inspected frame relates to a hand.

5. The gesture recognition system according to claim 4, wherein statistical calculations consider the similarity of said pixel color to a hand color.

6. The gesture recognition system according to claim 1, wherein the trajectory segment which is extracted by the trajectory segmentation module from the buffer is the whole buffer content or a portion thereof.

7. The gesture recognition system according to claim 1, wherein the trajectory segmentation module bases its detection of relevant trajectory segment on movement predefinitions.

8. The gesture recognition system according to claim 1, wherein the hand tracker employs a switch adaptively turned on and off by the amount of the motion energy in the frame to enable dynamic fusion of motion and color cues.

9. The gesture recognition system according to claim 1, for controlling electrical devices from a group comprising:

TVs, IPTVs, PCs, DVDs, music players, video players, mobile screens, video consoles, wall switches, lamps, heating systems, and security systems.

10. The gesture recognition system according to claim 1, which is training free.

11. The gesture recognition system according to claim 1, which is feedback free.