Hand gesture recognizing device

• US 6,215,890 B1
• Filed: 09/25/1998
• Issued: 04/10/2001
• Est. Priority Date: 09/26/1997
• Status: Expired due to Fees

• Current Assignee: National Institute of Information and Communications Technology, Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Original Assignee: Communications Research Laboratory of The Ministry of Posts and Telecommunications Japan, Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Inventors: Imagawa, Kazuyuki, Teshima, Terutaka, Takata, Yuji, Matsuo, Hideaki, Lu, Shan, Igi, Seiji
• Primary Examiner(s): Bella, Matthew C.
• Assistant Examiner(s): CHOOBIN, BARRY
• Application Number:
  US09/160,048
• Time in Patent Office: 928 Days
• Field of Search: 235/64.7, 384/15, 395/333, 395/125, 382/103, 382/288, 382/203, 382/100, 382/190, 382/195, 345/158, 345/501, 345/358, 345/351, 73/865 A, 348/459, 348/56, 704/3, 434/307
• US Class Current: 382/103
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A hand gesture recognizing device is provided which can correctly recognize hand gestures at high speed without requiring users to be equipped with tools. A gesture of a user is stereoscopically filmed by a photographing device 1 and then stored in an image storage device 2. A feature image extracting device 3 transforms colors of the stereoscopic image data read from the image storage device 2 in accordance with color transformation tables created by a color transformation table creating device 13, and disassembles and outputs the feature image of the user in corresponding channels. A spatial position calculating device 4 calculates spatial positions of feature parts of the user by utilizing parallax of the feature image outputted from the feature image extracting device 4. A region dividing device 5 defines the space around the user with spatial region codes. A hand gesture detecting device 6 detects how the hands of the user move in relation to the spatial region codes. A category is detected first on the basis of the detected hand gesture, and then a sign language word in that category is specified.

Claims

1. A hand gesture recognizing device for recognizing hand gestures performed by a user, comprising:

a photographing device having at least two cameras and operable to take stereoscopic pictures of the user;

an image storage device operable to store stereoscopic image data of the user outputted from the photographing device at an arbitrary sampling interval;

a feature image extracting device operable to remove the stereoscopic image data in order from the image storage device, extract a feature image showing body features of the user from each of the stereoscopic image data, and disassemble and output the feature image in a plurality of channels;

a spatial position calculating device operable to detect three-dimensional spatial positions of body parts of the user based on a parallax of the feature image outputted from said feature image extracting device;

a region dividing device operable to divide a space surrounding the user into a plurality of regions related to the body of the user based on the parallax of the feature image outputted from said feature image extracting device;

a hand gesture detecting device operable to detect how three-dimensional spatial positions corresponding to hands of the user, in the three-dimensional spatial positions of the body parts calculated by said spatial position calculating device, move with respect to the regions divided by said region dividing device;

a hand gesture word determining device operable to determine a corresponding hand gesture word based on the movement of the hands detected by said hand gesture detecting device; and

an output device operable to output a result determined by said hand gesture word determining device in a form which can be recognized by an operator.

2. The hand gesture recognizing device according to claim 1, wherein said feature image extracting device is operable to output the feature image in said corresponding channels based on color information of individual picture elements forming the stereoscopic image data.

3. The hand gesture recognizing device according to claim 2, wherein said feature image extracting device is operable to set a color transformation table for each of said channels based on a color to be extracted and a color not to be outputted which are specified by the operator,

transform the color information of the individual picture elements forming the stereoscopic image data according to the color transformation table, and

discriminate values transformed according to the color transformation table with a predetermined threshold to output the feature image in said corresponding channels.

4. The hand gesture recognizing device according to claim 1, wherein said region dividing device is operable to estimate a position of a body part which does not appear in the feature image based on the three-dimensional spatial positions of the body parts calculated by said spatial position calculating device and divide the space surrounding the user into still smaller regions based on the estimated position.

5. The hand gesture recognizing device according to claim 1, wherein said region dividing device is operable to calculate a difference value between the feature images adjacent in time and perform the process of dividing regions only when the difference value is equal to or larger than a predetermined threshold.

6. The hand gesture recognizing device according to claim 1, wherein said region dividing device is operable to divide a space extending in front of and in rear of the body of the user into a plurality of layers and further to divide each of the layers into a plurality of regions.

7. The hand gesture recognizing device according to claim 6, wherein said region dividing device is operable to divide the layers into different numbers of regions.

8. The hand gesture recognizing device according to claim 7, wherein said region dividing device is operable to divide the layers into decreasing numbers of regions as going forward, seen from the body of the user, from a backmost layer to a front layer.

9. The hand gesture recognizing device according to claim 1, wherein a plurality of hand gesture words as objects of recognition are classified into a plurality of categories in advance,

and wherein said hand gesture word determining device comprises:

a category dictionary in which features of movements common among hand gesture words belonging to respective categories are previously recorded for each of the categories,

a word dictionary in which more detailed features of the movements of individual hand gesture words are stored for each of the categories,

a category detecting device operable to detect which of the categories the movement of the hands detected by said hand gesture detecting device belongs to, from the category dictionary, and

a word recognizing device operable to recognize which of the hand gesture words belonging to the category detected by said category detecting device corresponds to the movement of the hands detected by said hand gesture detecting device.

10. The hand gesture recognizing device according to claim 9, wherein sandwort recognizing device is operable to output, as a recognition result, one hand gesture word having a highest degree of similarity with respect to the movement of the hands detected by said hand gesture detecting device from among the hand gesture words belonging to the category detected by said category detecting device.

11. The hand gesture recognizing device according to claim 9, wherein sandwort recognizing device outputs, as a recognition result, one or a plurality of hand gesture words having a degree of similarity equal to or higher than a given threshold with respect to the movement of the hands detected by said hand gesture detecting device from among the hand gesture words belonging to the category detected by said category detecting device.

12. The hand gesture recognizing device according to claim 1, further comprising a start-of-gesture informing device operable to inform the user when to start a hand gesture.

13. The hand gesture recognizing device according to claim 1, wherein said hand gesture detecting device is operable to extract a sampling point at which a direction of movement changes greatly as a control point from among sampling points showing three-dimensional spatial positions detected between a start point and an end point of the movement and to represent the hand movement of the user by using the start point, the end point, and the control point.

14. The hand gesture recognizing device according to claim 13, wherein said hand gesture detecting device is operable to detect:

a sampling point existing between the start point and the end point, having a maximum distance, which is equal to or larger than a predetermined threshold, to a straight line connecting the start point and the end point,

a sampling point existing between the start point and AN adjacent control point, having a maximum distance, which is equal to or larger than the predetermined threshold, to a straight line connecting the start point and the adjacent control point,

a sampling point existing between the end point and an adjacent control point, having a maximum distance, which is equal to or larger than the predetermined threshold, to a straight line connecting the end point and the adjacent control point, and

a sampling point existing between two adjacent control points, having a maximum distance, which is equal to or larger than the predetermined threshold, to a straight line connecting said two adjacent control points; and

further operable to define these detected sampling points as the control points.

15. The hand gesture recognizing device according to claim 14, wherein said hand gesture detecting device is operable to hierarchically detect the control points by using a plurality of thresholds to hierarchically represent the hand movement of the user, and

said hand gesture word determining device is operable to hierarchically specify a corresponding hand gesture word on the basis of the hand movement of the user hierarchically represented by said hand gesture detecting device.