Method and device for segmenting hand gestures

• US 6,256,400 B1
• Filed: 09/28/1999
• Issued: 07/03/2001
• Est. Priority Date: 09/28/1998
• Status: Expired due to Term

• Current Assignee: National Institute of Information and Communications Technology, Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Original Assignee: Communications Research Laboratory Independent Administrative Institution, Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Inventors: Nagashima, Yuji, Matsuo, Hideaki, Lu, Shan, Igi, Seiji, Takata, Yuji
• Primary Examiner(s): Bella, Matthew C.
• Assistant Examiner(s): CHOOBIN, BARRY
• Application Number:
  US09/406,733
• Time in Patent Office: 644 Days
• Field of Search: 382/103, 382/276, 382/236, 382/107, 382/199, 382/171, 379/52, 379/219, 348/20, 348/700, 704/271, 434/112, 434/307, 345/9, 345/156
• US Class Current: 382/103
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

An object of the present invention is to provide a method of segmenting hand gestures which automatically segments hand gestures to be detected into words or apprehensible units structured by a plurality of words when recognizing the hand gestures without the user'"'"'s presentation where to segment. Transition feature data in which a feature of a transition gesture being not observed during a gesture representing a word but is described when transiting from a gesture to another is previously stored. Thereafter, a motion of image corresponding to the part of body in which the transition gesture is observed is detected (step S106), the detected motion of image is compared with the transition feature data (step S107), and a time position where the transition gesture is observed is determined so as to segment the hand gestures (step S108).

Claims

1. A method of automatically segmenting a subject'"'"'s hand gestures into words or apprehensible units structured as a plurality of words when recognizing the subject'"'"'s hand gestures, said method comprising:

storing transition feature data including a feature of a transition gesture which is not observed in the subject'"'"'s body during a gesture representing a word, but is observed when transiting from one gesture to another;

photographing the subject, and storing image data thereof;

extracting an image corresponding to a part of the body in which the transition gesture is observed from the image data;

detecting a motion of the image corresponding to the part of the body in which the transition gesture is observed; and

segmenting the hand gestures by comparing the motion of the image corresponding to the part of the body in which the transition gesture is observed with the transition feature data, and then finding a time position where the transition gesture is observed.

2. A method of automatically segmenting a subject'"'"'s hand gestures according to claim 1, wherein the transition gesture comprises blinking.

3. A method of automatically segmenting a subject'"'"'s hand gestures according to claim 1, wherein the transition gesture comprises nodding.

4. A method of automatically segmenting a subject'"'"'s hand gestures according to claim 1, wherein the transition gesture comprises closing a mouth.

5. A method of automatically segmenting a subject'"'"'s hand gestures according to claim 1, wherein the transition gesture comprises stopping a motion of at least one hand.

6. A method of automatically segmenting a subject'"'"'s hand gestures according to claim 1, wherein the transition gesture comprises stopping a motion of the body.

7. A method of automatically segmenting a subject'"'"'s hand gestures according to claim 1, wherein the transition gesture comprises touching a face with at least one hand.

8. A method of automatically segmenting a subject'"'"'s hand gestures according to claim 1, further comprising setting a meaningless-hand region around the subject in which no hand gesture is considered effective even if a hand is observed, wherein the transition gesture comprises a movement of the hand into or out from the meaningless-hand region.

9. A method of automatically segmenting a subject'"'"'s hand gestures according to claim 1, wherein said segmenting of the hand gestures comprises measuring a duration of the transition gesture, and then segmenting the hand gestures in relation to the duration.

10. A method of automatically segmenting a subject'"'"'s hand gestures according to claim 1, further comprising:

storing non-transition feature data including a feature of a non-transition gesture which is not observed in the body when transiting from a gesture representing a word to another, but is observed during a gesture representing a word;

extracting an image corresponding to a part of the body in which the non-transition gesture is observed from the image data;

detecting a motion of the image corresponding to the part of the body in which the non-transition gesture is observed; and

finding a time position where the non-transition gesture is observed by comparing the motion of the image corresponding to the part of the body in which the non-transition gesture is observed with the non-transition feature data, wherein said segmenting of the hand gestures does not occur at the time position where the non-transition gesture is observed.

11. A method of automatically segmenting a subject'"'"'s hand gestures according to claim 10, wherein the non-transition gesture comprises bringing hands closer to each other than a value predetermined for a distance between the hands.

12. A method of automatically segmenting a subject'"'"'s hand gestures according to claim 10, wherein the non-transition gesture comprises changing a shape of a mouth.

13. A method of automatically segmenting a subject'"'"'s hand gestures according to claim 10, wherein the non-transition gesture comprises a motion of moving a right hand symmetrical to a left hand, or vice-versa.

14. A method of automatically segmenting a subject'"'"'s hand gestures according to claim 13, wherein said photographing of the subject and storing image data thereof comprises stereoscopically photographing the subject and storing 3D image data thereof;

said extracting of the image corresponding to a part of the body in which the non-transition gesture is observed comprises extracting a 3D image corresponding to the part of the body in which the non-transition gesture is observed from the 3D image data;

said detecting of the motion of the image corresponding to the part of the body in which the non-transition gesture is observed comprises detecting a motion of the 3D image; and

said finding of the time position where the non-transition gesture is observed comprises;

detecting whether changes in a gesture plane for the right hand and a gesture plane for the left hand are in accordance with the motion of the 3D image, and

if neither of the gesture planes shows a change, determining that the non-transition gesture is being observed and finding a time position of the non-transition gesture.

15. A method of automatically segmenting a subject'"'"'s hand gestures according to claim 14, wherein said detecting of whether changes in a gesture plane for the right hand and a gesture plane for the left hand comprises detecting whether changes in a gesture plane for the right hand and a gesture plane for the left hand are in accordance with a change in a normal vector to the gesture planes.

16. A method of automatically segmenting a subject'"'"'s hand gestures according to claim 14, further comprising:

generating, as to a plurality of 3D gesture codes corresponding to a 3D vector whose direction is varying, a single-motion plane table in which a combination of the 3D gesture codes found in a single plane is included; and

converting the motion of the 3D image into a 3D gesture code string represented by the plurality of 3D gesture codes, wherein said detecting of whether changes in a gesture plane for the right hand and a gesture plane for the left hand comprises detecting whether changes in a gesture plane for the right hand and a gesture plane for the left hand are in accordance with the single-motion plane table.

17. A method of automatically segmenting a subject'"'"'s hand gestures according to claim 1, further comprising:

storing image data of an animation representing the transition gesture;

detecting a status of the transition gesture'"'"'s detection and a status of the hand gesture'"'"'s recognition; and

visually displaying the animation representing the transition gesture to the subject in relation to the status of the transition gesture'"'"'s detection and the status of the hand gesture'"'"'s recognition.

18. A method of automatically segmenting a subject'"'"'s hand gestures according to claim 17, wherein said visually displaying of the animation comprises changing a speed of the animation in accordance with the status of the hand gesture'"'"'s recognition.

19. A method of automatically segmenting a subject'"'"'s hand gestures according to claim 1, wherein said storing transition feature data comprises previously storing transition feature data.

20. A method of automatically segmenting a subject'"'"'s hand gestures according to claim 10, wherein said storing non-transition feature data comprises previously storing non-transition feature data.

21. A computer program embodied on a computer readable medium for use with a computer for automatically segmenting a subject'"'"'s hand gestures into words or apprehensible units structured by a plurality of words, said computer program comprising:

computer readable program code operable to instruct the computer to store transition feature data including a feature of a transition gesture which is not observed in the subjcct'"'"'s body during a gesture representing a word, but is observed when transiting from one gesture to another;

computer readable program code operable to instruct the computer to instruct a camera to photograph the subject and store image data thereof;

computer readable program code operable to instruct the computer to extract an image corresponding to a part of the body in which the transition gesture is observed from the image data;

computer readable program code operable to instruct the computer to detect a motion of the image corresponding to the part of the body in which the transition gesture is observed; and

computer readable program code operable to instruct the computer to segment the hand gestures by comparing the motion of the image corresponding to the part of the body in which the transition gesture is observed with the transition feature data, and then find a time position where the transition gesture is observed.

22. A computer program according to claim 21, further comprising:

computer readable program code operable to instruct the computer to store non-transition feature data including a feature of a non-transition gesture which is not observed in the body when transiting from a gesture representing a word to another, but is observed during a gesture representing a word;

computer readable program code operable to instruct the computer to extract an image corresponding to a part of the body in which the non-transition gesture is observed from the image data;

computer readable program code operable to instruct the computer to detect a motion of the image corresponding to the part of the body in which the non-transition gesture is observed; and

computer readable program code operable to instruct the computer to find a time position where the non-transition gesture is observed by comparing the motion of the image corresponding the part of the body in which the non-transition gesture is observed with the non-transition feature data, wherein said computer readable program code operable to instruct the computer to segment the hand gestures is further operable to instruct the computer to not segment the hand gestures at the time position where the non-transition gesture is observed.

23. A computer program according to claim 21, further comprising:

computer readable program code operable to instruct the computer to store image data of an animation representing the transition gesture;

computer readable program code operable to instruct the computer to detect a status of the transition gesture'"'"'s detection and a status of the hand gesture'"'"'s recognition; and

computer readable program code operable to instruct the computer to visually display the animation representing the transition gesture to the subject in relation to the status of the transition gesture'"'"'s detection and the status of the hand gesture'"'"'s recognition.

24. A computer program according to claim 21, wherein said computer readable program code operable to instruct the computer to store transition feature data comprises computer readable program code operable to instruct the computer to previously store transition feature data.

25. A computer program according to claim 22, wherein said computer readable program code operable to instruct the computer to store non-transition feature data comprises computer readable program code operable to instruct the computer to previously store non-transition feature data.

26. A computer program according to claim 23, wherein said computer readable program code operable to instruct the computer to store image data of an animation representing the transition gesture comprises computer readable program code operable to instruct the computer to previously store image data of an animation representing the transition gesture.

27. A hand gesture segmentation device for automatically segmenting a subject'"'"'s hand gestures into words or apprehensible units structured by a plurality of words when recognizing the subject'"'"'s hand gestures, said device comprising:

means for storing transition feature data including a feature of a transition gesture which is not observed in the subject'"'"'s body during a gesture representing a word, but is observed when transiting from one gesture to another;

means for photographing the subject, and storing image data thereof;

means for extracting an image corresponding to a part of the body in which the transition gesture is observed;

means for detecting a motion of the image corresponding to the part of the body in which the transition gesture is observed; and

means for segmenting the hand gestures by comparing the motion of the image corresponding to the part of the body in which the transition gesture is observed with the transition feature data, and then finding a time position where the transition gesture is observed.

28. A hand gesture segmentation device according to claim 27, further comprising:

means for storing non-transition data including a feature of a non-transition gesture which is not observed in the body when transiting from a gesture representing a word to another, but is observed during a gesture representing a word;

means for extracting an image corresponding to a part if the body in which the non-transition gesture is observed from the image data;

means for detecting a motion of the image corresponding to the part of the body in which the non-transition gesture is observed; and

means for finding a time position where the non-transition gesture is observed by comparing the motion of the image corresponding to the part of the body in which the non-transition gesture is observed with the non-transition feature data, wherein said means for segmenting the hand gestures does not execute segmentation with respect to the hand gestures at the time position where the non-transition gesture is observed.

29. A motion induction device being incorporated in a hand gesture recognition device for recognizing a subject'"'"'s hand gestures, and in a hand gesture segmentation device for automatically segmenting the hand gesture into words or apprehensible units structured by a plurality of words to visually guide the subject to make a predetermined gesture, said hand gesture segmentation device including a function of detecting a transition gesture which is not observed in the subject'"'"'s body during a gesture representing a word, but is observed when transiting from one gesture to another, and then segmenting the hand gestures, said motion induction device comprising:

means for storing image data of an animation representing the transition gesture;

means for detecting a status of the transition gesture'"'"'s detection and a status of the hand gesture'"'"'s recognition by monitoring said hand gesture segmentation device and said hand gesture recognition device; and

means for visually displaying the animation representing the transition gesture to the subject in relation to the status of the transition gesture'"'"'s detection and the status of the hand gesture'"'"'s recognition.

30. A motion induction device according to claim 29, wherein said means for visually displaying the animation comprises means for changing a speed of the animation according to the status of the hand gesture'"'"'s recognition.

31. A motion induction device according to claim 29, wherein said means for storing image data of an animation representing the transition gesture comprises means for previously storing image data of an animation representing the transition gesture.

32. A hand gesture segmentation device for automatically segmenting a subject'"'"'s hand gestures into words or apprehensible units structured by a plurality of words when recognizing the subject'"'"'s hand gestures, said device comprising:

means for storing transition feature data including a feature of a transition gesture which is not observed in the subject'"'"'s body during a gesture representing a word, but is observed when transiting from one gesture to another;

means for photographing the subject with a camera placed in a position opposite to the subject, and storing image data thereof;

means for extracting an image corresponding to a part of the body in which the transition gesture is observed from the image data;

means for detecting a motion of the image corresponding to the part of the body in which the transition gesture is observed;

means for segmenting the hand gesture by comparing the motion of the image corresponding to the part of the body in which the transition gesture is observed with the transition feature data, and then finding a time position where the transition gesture is observed;

means for visually displaying the animation representing the transition gesture to the subject in relation to the status of the transition gesture'"'"'s detection and the status of the hand gesture'"'"'s recognition; and

means for concealing said camera from the subject'"'"'s view.

33. A hand gesture segmentation device according to claim 32, wherein said animation display means comprises an upward-facing monitor in a vertically lower position from a straight line between the subject and said camera, and said means for concealing said camera comprises a half mirror operable to allow light coming from a forward direction to pass and through and to reflect light coming from a reverse direction, wherein said half mirror is placed on the straight line between the subject and said camera in a vertically upper position from said monitor where an angle of 45°

is obtained with respect to the straight line.