Hand gesture control system

• US 6,002,808 A
• Filed: 07/26/1996
• Issued: 12/14/1999
• Est. Priority Date: 07/26/1996
• Status: Expired due to Term

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Information Technology Center America, Inc. (Mitsubishi Electric Corporation)
• Inventors: Freeman, William T.
• Primary Examiner(s): Tran, Phuoc
• Assistant Examiner(s): Mariam, Daniel G.
• Application Number:
  US08/690,297
• Time in Patent Office: 1,236 Days
• Field of Search: 382/100, 382/103, 382/107, 382/174, 382/288, 382/291, 382/286, 382/141, 382/216, 382/287, 382/289, 382/181, 382/190, 382/203, 382/206, 382/204, 382/218, 382/278, 382/170 , 382/189, 348/77, 348/170, 348/171, 345/145, 345/358
• US Class Current: 382/288
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/8041 Skating using skis, skates or board
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A system is provided for rapidly recognizing hand gestures for the control of computer graphics, in which image moment calculations are utilized to determine an overall equivalent rectangle corresponding to hand position, orientation and size, with size in one embodiment correlating to the width of the hand. In a further embodiment, a hole generated through the utilization of the touching of the forefinger with the thumb provides a special trigger gesture recognized through the corresponding hole in the binary representation of the hand. In a further embodiment, image moments of images of other objects are detected for controlling or directing onscreen images.

Claims

1. A system for rapidly recognizing non-regular objects in space for the control of computer graphics, comprising:

means for scanning an area at which said object is expected to exist and for generating an electronic image in the form of a pixel representation of said object;

means for detecting and calculating image moments from pixel intensities of pixels in said pixel representation;

means for determining, based upon the calculated image moments, an overall rectangle equivalent to the object and having a position, orientation, length and width corresponding to object position, orientation, length and width; and

,

means for identifying parameters of said rectangle and generating a corresponding control signal.

2. A system for rapidly recognizing hand gestures for the control computer graphics, comprising:

means for scanning an area at which a hand is expected to exist and for generating an electronic image in the form of a pixel representation of said hand;

means for calculating image moments from pixel intensities of pixels in said pixel representation;

means for determining, based upon the calculated image moments, an overall rectangle equivalent to the hand and having a position, orientation, length and width corresponding to a position, orientation, length and width of the hand; and

,

means for correlating said rectangle with a predetermined hand gesture and for providing a signal representing the result of said correlation.

3. The system of claim 2 wherein the correlating means responds to at least one of the position, the orientation, the length and the width of the rectangle exceeding a predetermined threshold by generating the control signal to control said computer graphics in a predetermined manner.

4. The system of claim 2, wherein said size corresponds to the width of the hand making said hand gesture.

5. The system of claim 2, wherein said correlating means responds to the existence of a hole within a region of said pixels generated through touching of a finger and a thumb of said hand by providing a predetermined signal upon correlating the corresponding rectangle to that hand gesture.

6. The system of claim 5 further comprising means for determining that the hole exist by counting a number of holes in said pixel representation.

7. A system for rapidly recognizing a non-regular image, comprising:

means for generating a representation of said image;

means for detecting and calculating image moments from pixel intensities of pixels in said pixel representation;

means for determining, based on the calculated image moments, an overall geometric structure equivalent to the image and having length, width and height characteristics corresponding to parameters of said image; and

,

means for correlating said structure with a predetermined structure and for providing a signal representing the result of said correlating.

8. The system of claim 7 wherein the correlating means responds to at least one of the length, the width and the height characteristics of the structure exceeding a predetermined threshold by providing the signal to control computer graphics in a particular manner.

9. The system of claim 7, wherein said image is that of a hand, wherein said pixel generating means includes means for generating an electronic binary pixel image corresponding to said hand;

and further comprising;

means for determining a change in image topology corresponding to the exact time a portion of said image is in the form of a completed circle corresponding to that time at which a hole generated through touching of a finger of with a thumb of the hand is completed; and

means for providing a predetermined signal upon detection of said completed circle, whereby said completed circle can be used as a staccato button-like-push to effectuate a hand generated on/off control action.

10. The system of claim 9 wherein said predetermined signal is a trigger signal.

11. The system of claim 10, wherein said trigger signal is generated in response to the detection of a hole within a region of said generated pixels.

12. The system of claim 11, wherein said hole is detected by counting the number of holes in said binary pixel image.

13. A method of controlling a digital representation, comprising the steps of:

imaging an object to generate object images;

determining a movement of the object based upon at least one of an orientation and a position of a first geometric configuration having a contour different than a contour of the object and corresponding to image intensities of a first of the object images and at least one of an orientation and a position of a second geometric configuration having a contour different than a contour of the object and corresponding to image intensities of a second of the object images; and

controlling a digital representation based upon the determined movement.

14. The method of claim 13, further comprising the steps of:

computing a first spatially weighted average intensity of the generated first image in its entirety;

computing a second spatially weighted average intensity of the generated second image in its entirety;

generating the first geometric configuration based upon the computed first spatially weighted average; and

generating the second geometric configuration based upon the computed second spatially weighted average.

15. The method of claim 13, wherein:

the imaged object is a hand;

the geometric configuration is a rectangle; and

the movement corresponds to a gesture.

16. The method of claim 15, wherein:

the gesture is a circle formed with a thumb and a finger of the hand; and

the digital representation is controlled such that an action is triggered responsive to the thumb contacting the finger to form the circle.

17. The method of claim 13, further comprising the steps of:

computing a first spatially weighted average intensity of the generated first image;

computing a second spatially weighted average intensity of the generated second image;

generating the first geometric configuration based upon the computed first spatially weighted average intensity, wherein the center of mass of the object in the first image corresponds to the center of the first geometric configuration; and

generating the second geometric configuration based upon the computed second spatially weighted average intensity, wherein the center of mass of the object in the second image corresponds to the center of the second geometric configuration;

wherein the orientation and the position of the second geometric configuration is determined based on a difference in a location of the center of the first geometric configuration and a location of the center of the second geometric configuration.

18. The method of claim 13, further comprising the steps of:

computing first summations respectively summing intensities of the generated first image in an x direction, in a y direction and in a diagonal direction;

computing second summations respectively summing intensities of the generated second image in the x direction, in the y direction and in the diagonal direction;

generating the first geometric configuration based upon the computed first summations; and

generating the second geometric configuration based upon the computed second summations.

19. The method of claim 13, further comprising the steps of:

determining another movement of the object in a third of the object images based upon a distance between (i)a third geometric configuration having a contour different than a contour of the object and corresponding to the object in the third image and (ii) a fixed reference; and

controlling the digital representation based upon the determined other movement.

20. The method of claim 19, wherein:

the distance is a vertical distance and exceeds a threshold; and

the digital representation is controlled to perform a jump sequence based upon the determined other movement.

21. A system for controlling a digital representation, comprising:

an imager configured to generate image intensity data representing an object in an orientation; and

at least one processor configured (i) to process the generated image intensity data to generate a geometric configuration, corresponding to the generated image intensity data, representing the object, and having a contour different than a contour of the represented object and an orientation corresponding to the orientation of the represented object, and (ii) to generate signals to control a digital representation based upon the orientation of the geometric configuration.

22. The system of claim 21, wherein the object is a body part.

23. The system of claim 21, wherein the geometric configuration is a rectangle.

24. The system of claim 21, wherein:

the represented object has a position;

the generated geometric configuration has a position corresponding to the position of the represented object; and

the at least one processor is further configured to generate the signals to control the digital representation based upon the position of the geometric configuration.

25. The system of claim 21, wherein:

the orientation of the represented object is a first object orientation, the image data is first image data, and the geometric configuration is a first geometric configuration and the orientation of the first geometric configuration is a first orientation;

the imager is further configured to generate second image data representing the object in a second object orientation; and

the at least one processor is further configured (i) to process the generated second image data to generate a second geometric configuration representing the object, and having a contour different than the contour of the object represented by the second image data and a second orientation corresponding to the second object orientation of the object, (ii) to determine a movement of the object from the second object orientation to the first object orientation based upon the first orientation, and (iii) to generate the signals to control the digital representation in accordance with the determined movement.

26. The system of claim 21, wherein said at least one processor is further configured to:

compute a spatially weighted average intensity of the generated image data in its entirety; and

generate the geometric configuration based upon the computed spatially weighted average.

27. The system of claim 21, wherein:

the object is a hand;

the image data represents a circle formed with a thumb and a finger of the hand; and

the at least one processor is configured to generate the control signal to thereby trigger an action responsive to the thumb contacting the finger to form the circle.

28. The system of claim 21, wherein

the object has a center of mass and the geometric configuration has a center point;

and

the center point of the geometric configuration corresponds to the center of mass of the object.

29. The system of claim 21, wherein:

the at least one processor is further configured to compute summations respectively summing intensities of the generated image data in an x direction, in a y direction and in a diagonal direction and to generate the geometric configuration based upon the computed summations.

30. The system of claim 21, wherein:

the represented object has a position;

the generated geometric configuration has a position corresponding to the position of the represented object;

the at least one processor is further configured to determine a distance between the position of the geometric configuration and a fixed reference, and to generate the signals to control the digital representation based upon the determined distance.

31. The system of claim 30, wherein:

the distance is a vertical distance and exceeds a threshold; and

the digital representation is controlled to perform a jump sequence.

32. The system of claim 21, wherein:

the at least one processor is two processors; and

the imager and one of the processors form an artificial retina.