Hand gesture machine control system

• US 5,594,469 A
• Filed: 02/21/1995
• Issued: 01/14/1997
• Est. Priority Date: 02/21/1995
• Status: Expired due to Fees

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Information Technology Center America, Inc. (Mitsubishi Electric Corporation)
• Inventors: Freeman, William T., Weissman, Craig D.
• Primary Examiner(s): Hjerpe, Richard
• Assistant Examiner(s): Liang, Regina
• Application Number:
  US08/391,955
• Time in Patent Office: 693 Days
• Field of Search: 345/157, 345/156, 345/158, 345/145, 345/146, 395/159, 382/291, 382/203
• US Class Current: 345/158
• CPC Class Codes: G05B 19/106 for selecting a programme, variable or parameter
• CPC Class Codes: G05B 2219/23021 Gesture programming, camera sees hand, displays it on screen, grasp buttons
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H01H 2217/032 Feedback about selected symbol, e.g. display
• CPC Class Codes: H04N 21/42201 biosensors, e.g. heat sensor for presence detection, EEG sensors or any limb activity sensors worn by the user input arrangements for interaction with the human body based on nervous system activity detection G06F3/015
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 5/44 Receiver circuitry for the reception of television signals according to analogue transmission standards H04N5/14 takes precedence
• CPC Class Codes: H04N 5/445 for displaying additional information H04N5/50 takes precedence

Abstract

A system for the control from a distance of machines having displays incls hand gesture detection in which the hand gesture causes movement of an on-screen hand icon over an on-screen machine control icon, with the hand icon moving the machine control icon in accordance with sensed hand movements to effectuate machine control. In one embodiment, TV control led by hand signals includes detecting a single hand gesture and providing a hand icon on the screen along with the provision of icons representing TV controls such as volume, channel, color, density, etc., in which a television camera detects the hand in a noisy background through correlation techniques based on values of local image orientation. In order to trigger the system into operation, a trigger gesture such as the "how" sign is distinguished from the background through the utilization of orientation angle differences. From correlation values based on correlating local orientations between a mask defining a particular hand and the later acquired image of the hand, normalized correlation scores for each pixel are obtained, with the correlation peak being detected and then thresholded to eliminate false alarms.

Claims

1. A system for control of a machine having a display screen and means for controlling said machine via actuating on-screen machine control icons, comprising:

means for detecting the existence of and position in space of a predetermined hand gesture and for generating corresponding existence and position signals responsive thereto;

means for providing an on-screen hand icon corresponding in on-screen position to the detected position of said predetermined hand gesture in space, such that changes in position of said predetermined hand gesture result in a corresponding change in position of said on-screen hand icon, whereby the position of said on-screen hand icon is controllable by movement of said predetermined hand gesture; and

,

means for detecting coincidence of said on-screen hand icon with an on-screen machine control icon as said hand icon made to move over said machine control icon by movement of said predetermined hand gesture and for generating corresponding machine control signal responsive to said detected coincidence wherein said means for detecting the existence and position of said predetermined hand gesture includes;

means for providing a hand gesture template including orientation vectors corresponding to orientations of edges of said predetermined hand gesture corresponding to said predetermined hand gesture;

means for optically scanning a region at which said predetermined hand gesture is expected to exist and for providing orientation vectors corresponding to orientations of edges of images within said scanned region;

means for correlating the orientation vectors of said hand template with orientations vectors of the images within said scanned region to provide a correlation values; and

,

means for providing a trigger signal when said correlation value exceeds a predetermined threshold.

2. The system of claim 1, wherein said machine control icon represents a switch and wherein said corresponding machine control signal generating means includes means for generating a switch closure signal.

3. The system of claim 1, wherein said machine control icon includes a moveable element and wherein said coincidence detecting means includes means for detecting coincidence of said hand icon with said moveable element, thus to represent capture of said moveable element, and means responsive to the movement of said hand icon for effectuating corresponding movement of said moveable element and generation of the corresponding machine control signal.

4. The system of claim 3 wherein said machine control icon is a slider bar and wherein said moveable element is a slider.

5. The system of claim 3 wherein said moveable element moves in a predetermined direction and wherein said means for effectuating corresponding movement of said moveable element is actuated only by movement of said hand icon in said predetermined direction.

6. The system of claim 1 and further including means for terminating said machine control upon detection of non-coincidence of said hand icon with said machine control icon.

7. The system of claim 5 said further including means for terminating said machine control upon movement of said hand icon in a direction different from said predetermined direction.

8. The system of claim 1 wherein said means for detecting the existence and position of said predetermined hand gesture includes:

a video camera directed towards a region in space at which said predetermined hand gesture is expected to exist, means coupled to said camera for analyzing the output of said camera for the presence of said predetermined hand gesture in the scene scanned by said vidio camera and for narrowing the scene scanned by said video camera to a region around the position of a detected hand gesture when said predetermined hand gesture is detected, thereby to minimize the effect of background noise in the further detection of said predetermined hand gesture once said predetermined hand gesture has been detected.

9. The system of claim 8 wherein said predetermined hand gesture is a flat hand presented with palm facing said camera.

10. The system of claim 1 wherein said predetermined hand gesture is the "how" sign.

11. The system of claim 1 wherein said predetermined hand gesture is the "how" sign, said predetermined gesture functioning as a trigger gesture, the detection of which establishes said machine control.

12. The system of claim 11, and further including means for detecting a second predetermined gesture and means responsive thereto for terminating said hand gesture machine control.

13. The system of claim 12 wherein said second predetermined hand gesture is a clenched fist.

14. The system of claim 1 and further including means for determining a maximum correlation value and the location thereof within said scanned region.

15. The system of claim 14 and further including means for delimiting the region scanned to that region in the vicinity of the position associated with said maximum correlation value.

16. The system of claim 14, wherein the position of said predetermined hand gesture is taken to be the position of said maximum correlation, thereby to facilitate hand gesture tracking.