Method and apparatus for translating hand gestures

• US 8,140,339 B2
• Filed: 07/21/2009
• Issued: 03/20/2012
• Est. Priority Date: 08/28/2003
• Status: Expired due to Fees

• Current Assignee: George Washington University
• Original Assignee: George Washington University
• Inventors: Hernandez-Rebollar, Jose L.
• Primary Examiner(s): Abebe, Daniel D
• Application Number:
  US12/506,677
• Publication Number:
  US 20100063794A1
• Time in Patent Office: 973 Days
• Field of Search: 704/270, 704/271, 382/126, 382/182
• US Class Current: 704/271
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G09B 21/009 Teaching or communicating with deaf persons G09B21/04 takes precedence; deaf and dumb persons G09B21/00
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems

Abstract

A sign language recognition apparatus and method is provided for translating hand gestures into speech or written text. The apparatus includes a number of sensors on the hand, arm and shoulder to measure dynamic and static gestures. The sensors are connected to a microprocessor to search a library of gestures and generate output signals that can then be used to produce a synthesized voice or written text. The apparatus includes sensors such as accelerometers on the fingers and thumb and two accelerometers on the back of the hand to detect motion and orientation of the hand. Sensors are also provided on the back of the hand or wrist to detect forearm rotation, an angle sensor to detect flexing of the elbow, two sensors on the upper arm to detect arm elevation and rotation, and a sensor on the upper arm to detect arm twist. The sensors transmit the data to the microprocessor to determine the shape, position and orientation of the hand relative to the body of the user.

Claims

1. A gesture recognition apparatus comprising:

an input assembly configured to detect a user gesture, a computer connected to said input assembly configured to generate an output signal based on said user gesture, said input assembly comprising;

a glove to be worn by a user, said glove having sensors configured to detect dynamic hand movements of each finger and thumb;

an elbow sensor configured to detect and measure flexing and positioning of the forearm about the elbow;

a shoulder sensor configured to detect movement and position of the user'"'"'s arm with respect to the user'"'"'s shoulder; and

a frame having a first section configured to couple to the upper arm of the user and a second section configured to couple to the forearm of the user, said first sections being coupled together by a hinge, said elbow sensor being positioned on said frame configured to measure flexing and positioning of the forearm and second section.

2. The apparatus of claim 1, wherein said input assembly comprises at least one sensor configured to be positioned on each of the user'"'"'s finger and thumb and to detect motion and position of each of the user'"'"'s finger and thumb.

3. The apparatus of claim 2, wherein said input assembly further includes a sensor configured to determine orientation and movement of the user'"'"'s arm with respect to the user'"'"'s body.

4. The apparatus of claim 1, wherein at least one of said elbow sensor and said shoulder sensor comprise an accelerometer.

5. The apparatus of claim 1, wherein said input assembly further comprises two accelerometers configured to be positioned on a distal phalanx of the user'"'"'s thumb.

6. The apparatus of claim 4, wherein said input assembly further comprises two accelerometers configured to be positioned on a back of the user'"'"'s hand to detect orientation and movement of a palm of the user.

7. The apparatus of claim 6, wherein said shoulder sensor includes a first angular sensor configured to measure arm rotation with respect to the user'"'"'s shoulder.

8. The apparatus of claim 7, wherein said shoulder sensor further comprises a second angular sensor configured to measure flexing of the user'"'"'s forearm about the user'"'"'s elbow.

9. The apparatus of claim 8, wherein said shoulder sensor further comprises an accelerometer configured to measure elevation and rotation of the user'"'"'s arm with respect to the user'"'"'s shoulder.

10. The apparatus of claim 1, said input assembly comprising:

an arm sensor for detecting movement and orientation of the user'"'"'s arm with respect to the user'"'"'s shoulder.

11. The apparatus of claim 1, wherein said shoulder sensor is coupled to said first section of said frame.

12. The apparatus of claim 1, wherein said shoulder sensor comprises a first sensor configured to detect twisting of the arm.

13. The apparatus of claim 12, wherein said first sensor of said shoulder sensor comprises a resistive angular sensor.

14. The apparatus of claim 12, wherein said shoulder sensor further comprises an accelerometer configured to detect motion, elevation and position of the upper arm with respect to the shoulder.

15. The apparatus of claim 1, wherein said glove includes a first accelerometer on each finger and thumb, a second accelerometer on the back of said glove configured to detect vertical orientation and movement of a palm of the user, and a third accelerometer on the back of said glove configured to detect axial orientation and movement of said glove with respect to the forearm.

16. The apparatus of claim 1, wherein the output signal is a visual output or an audible output.

17. The apparatus of claim 1, wherein said input assembly further comprises at least one accelerometer configured to be positioned on a back of the user'"'"'s hand to detect orientation and movement of a palm of the user.

18. A gesture recognition apparatus comprising:

an input assembly comprising a glove having hand sensors configured to detect hand movement, an elbow sensor configured to detect forearm movement, an arm sensor configured to detect arm orientation, and a shoulder sensor configured to detect arm rotation,

said input assembly further comprising a frame having an upper arm section and a forearm section, said upper arm section and said forearm section coupled together by a hinge; and

a computer connected to said input assembly and configured to generate an output corresponding to the detected hand movement, forearm movement, or arm rotation.

19. The apparatus of claim 18, wherein said shoulder sensor is coupled to said upper arm section.

20. The apparatus of claim 19, wherein said shoulder sensor comprises a first sensor configured to detect arm twisting.

21. The apparatus of claim 20, wherein said first sensor of said shoulder sensor comprises a resistive angular sensor.

22. The apparatus of claim 20, wherein said shoulder sensor further comprises an accelerometer configured to detect motion, elevation and position of the user'"'"'s upper arm with respect to the user'"'"'s shoulder.

23. The apparatus of claim 22, wherein said elbow sensor is coupled to said arm section.

24. The apparatus of claim 23, wherein said elbow sensor further comprises a resistive angular sensor.

25. The apparatus of claim 18, wherein said glove includes a finger accelerometer, a thumb accelerometer, a first back hand accelerometer on a back of said glove configured to detect vertical orientation and movement of said glove, and a second back hand accelerometer on the back of said glove configured to detect axial orientation and movement of said glove with respect to the forearm sensor.

26. A method for translating a user'"'"'s gesture composed of an initial pose, movement, hand location, and a final pose, the method comprising:

determining the initial pose, the hand location and the final pose of the gesture, and a movement of the gesture, the movement occurring between the initial pose and the final pose;

matching the determined initial pose with one or more initial poses of all known gestures, and defining a first list of candidate gestures as those whose pose matches the determined initial pose or, if there is only one match, returning a first most likely gesture corresponding to the match;

matching the determined hand location with one or more hand locations of the first list of candidate gestures, and defining a second list of candidate gestures as those whose hand locations match the determined hand location, or, if there is only one match, returning a second most likely gesture corresponding to the match;

matching the determined movement with one or more movements of the second list of candidate gestures, and defining a third list of candidate gestures as those more than one gestures whose movements match the determined movement, or, if there is only one match, returning a third most likely gesture corresponding to the match;

matching the determined final pose with one or more poses of the third list of candidate gestures, and defining a fourth list of candidate gestures as those more than one gestures whose final pose matches the determined final pose, or, if there is only one match, returning a fourth most likely gesture corresponding to the match;

matching the determined final hand location of the gesture with a hand location of the fourth list of candidate gestures, and returning a fifth most likely gesture corresponding to the match; and

converting the first, second, third, fourth or fifth gesture into an output based on the gesture.

27. The method for translating of claim 26, wherein said method transmits said most likely gesture as a stream of ASCII characters to be displayed as text or synthesized as voice in any language.

28. The method for translating of claim 26, wherein only the first step is used to translate postures as letters of finger-spelled words.

29. The method of claim 26, wherein hand posture is the combination of hand orientation and hand shape, and hand pose is the combination of hand posture and hand location with respect to the user'"'"'s body, movement is the trajectory defined by the hand when going from one pose to the next, static phoneme is a pose, dynamic phoneme is movement, and a gesture is a sequence of static and dynamic phonemes.

30. A hand gesture recognition apparatus comprising:

an input assembly configured to detect a hand gesture, said input assembly comprising a glove configured to be worn on a hand of a user, said glove having a first sensor on the back of said glove for detecting position, orientation and movement of a user'"'"'s palm;

a processor connected to said input assembly and configured to generate an output signal corresponding to the detected hand gesture based on the detected hand movement, glove position, glove orientation and glove movement,

wherein said processor is further configured to translate the detected hand movement, glove position, glove orientation and glove movement into a meaning, and said processor is further configured to generate an output corresponding to the meaning.

31. The apparatus of claim 30, wherein the meaning is a language and the output is an audible or visual representation of the language.

32. The apparatus of claim 30, further comprising an output device located on the glove for providing the output.

33. The apparatus of claim 30, further comprising at least one second sensor on each finger and thumb for detecting movement of each finger and thumb with respect to the hand.

34. The apparatus of claim 30, wherein said first sensor comprises a first hand sensor having two axes configured to be parallel to a plane defined by the user'"'"'s palm, and a second hand sensor having at least one axes configured to be perpendicular to the plane of the user'"'"'s palm.