METHOD AND APPARATUS FOR TRANSLATING HAND GESTURES

• US 20100063794A1
• Filed: 07/21/2009
• Published: 03/11/2010
• Est. Priority Date: 08/28/2003
• Status: Active Grant

• Current Assignee: George Washington University
• Original Assignee: George Washington University
• Inventors: HERNANDEZ-REBOLLAR, Jose L.
• Granted Patent: US 8,140,339 B2
• Time in Patent Office: Days
• US Class Current: 704/3
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G09B 21/009 Teaching or communicating with deaf persons G09B21/04 takes precedence; deaf and dumb persons G09B21/00
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems

Abstract

A sign language recognition apparatus and method is provided for translating hand gestures into speech or written text. The apparatus includes a number of sensors on the hand, arm and shoulder to measure dynamic and static gestures. The sensors are connected to a microprocessor to search a library of gestures and generate output signals that can then be used to produce a synthesized voice or written text. The apparatus includes sensors such as accelerometers on the fingers and thumb and two accelerometers on the back of the hand to detect motion and orientation of the hand. Sensors are also provided on the back of the hand or wrist to detect forearm rotation, an angle sensor to detect flexing of the elbow, two sensors on the upper arm to detect arm elevation and rotation, and a sensor on the upper arm to detect arm twist. The sensors transmit the data to the microprocessor to determine the shape, position and orientation of the hand relative to the body of the user.

Claims

1. A gesture recognition apparatus for use by a user, comprising:

an input assembly detecting movement and position of each finger with respect to the user'"'"'s hand, and movement and position of the user'"'"'s hand with respect to the user'"'"'s body, said input assembly generating at least one value corresponding to one or more phonemes; and

a word storage device for storing words, each word associated with a stored value corresponding to a sequence of one or more phonemes, receiving said at least one value from said input assembly, matching said at least one received value with the stored value, and producing an output value corresponding to the word associated with the matched stored value.

2. The apparatus of claim 1, wherein said input assembly comprises at least one sensor on each of the user'"'"'s finger and thumb, for detecting motion and position of each of the user'"'"'s finger and thumb.

3. The apparatus of claim 2, wherein said sensors each comprise an accelerometer.

4. The apparatus of claim 1, wherein said input assembly comprises two accelerometers on a distal phalanx of the user'"'"'s thumb.

5. The apparatus of claim 3, wherein said input assembly further comprises two accelerometers on a back of the user'"'"'s hand to detect orientation and movement of the user'"'"'s hand.

6. The apparatus of claim 2, wherein said input assembly further includes a sensor to determine orientation and movement of the user'"'"'s arm with respect to the user'"'"'s body.

7. The apparatus of claim 5, wherein said input assembly includes a first angular sensor positioned to measure arm rotation with respect to the user'"'"'s shoulder.

8. The apparatus of claim 7, wherein said input assembly further comprises a second angular sensor positioned to measure flexing of the user'"'"'s forearm about the user'"'"'s elbow.

9. The apparatus of claim 8, wherein said input assembly further comprises an accelerometer positioned to measure elevation and rotation of the user'"'"'s arm with respect to the user'"'"'s shoulder.

10. The apparatus of claim 1, said input assembly comprising:

a glove to be worn by the user, said glove having sensors for detecting dynamic hand movements of each of the user'"'"'s finger and the user'"'"'s thumb;

an elbow sensor for detecting and measuring flexing and positioning of the user'"'"'s forearm about the user'"'"'s elbow;

an arm sensor for detecting movement and orientation of the user'"'"'s arm with respect to the user'"'"'s shoulder; and

a shoulder sensor for detecting movement and position of the user'"'"'s arm with respect to the user'"'"'s shoulder.

11. A gesture recognition apparatus comprising:

an input assembly comprising a glove having hand sensors for detecting hand movement, an elbow sensor for detecting forearm movement, an arm sensor for detecting arm orientation, and a shoulder sensor for detecting arm rotation,

said input assembly further comprising a frame having an upper arm section and a forearm section, said upper arm section and said forearm section coupled together by a hinge, and

a computer connected to said input assembly and generating an output signal for producing a visual or audible output corresponding to the detected hand movement, forearm movement, or arm rotation.

12. The apparatus of claim 11, wherein said shoulder sensor is coupled

to said upper arm section.

13. The apparatus of claim 12, wherein said shoulder sensor comprises a first sensor for detecting arm twisting.

14. The apparatus of claim 13, wherein said first sensor of said shoulder sensor

comprises a resistive angular sensor.

15. The apparatus of claim 13, wherein said shoulder sensor further comprises an accelerometer for detecting motion, elevation and position of the user'"'"'s upper arm with respect to the user'"'"'s shoulder.

16. The apparatus of claim 11, wherein said glove includes a finger accelerometer, a thumb accelerometer, a first back hand accelerometer on a back of said glove to detect vertical orientation and movement of said glove, and a second back hand accelerometer on the back of said glove to detect axial orientation and movement of said glove with respect to the forearm sensor.

21. The apparatus of claim 15, wherein said elbow sensor is coupled to said arm section.

22. The apparatus of claim 21, wherein said elbow sensor further comprises a resistive angular sensor.

17. A method for translating a user'"'"'s gesture of a signed language composed of an initial pose, movement, hand location, and a final pose, the method comprising:

determining the initial pose, the hand location and the final pose of the gesture, and a movement of the gesture, the movement occurring between the initial pose and the final pose;

matching the determined initial pose with one or more initial poses of all known gestures, and defining a first list of candidate gestures as those whose pose matches the determined initial pose or, if there is only one match, returning a first most likely gesture corresponding to the match;

matching the determined hand location with one or more hand locations of the first list of candidate gestures, and defining a second list of candidate gestures as those whose hand locations match the determined hand location, or, if there is only one match, returning a second most likely gesture corresponding to the match;

matching the determined movement with one or more movements of the second list of candidate gestures, and defining a third list of candidate gestures as those more than one gestures whose movements match the determined movement, or, if there is only one match, returning a third most likely gesture corresponding to the match;

matching the determined final pose with one or more poses of the third list of candidate gestures, and defining a fourth list of candidate gestures as those more than one gestures whose final pose matches the determined final pose, or, if there is only one match, returning a fourth most likely gesture corresponding

to the match;

matching the determined final hand location of the gesture with a hand location of the fourth list of candidate signs, and returning a fifth most likely gesture corresponding to the match; and

converting the first, second, third, fourth or fifth gesture into a stream of ASCII characters to be displayed as text and/ or to a voice synthesizer to be reproduced as speech.

18. The method for translating a sign of claim 17, wherein said method transmits said most likely gesture as a stream of ASCII characters to be displayed as text or synthesized as voice in any language.

19. The method for translating signs of claim 17, wherein only the first step is

used to translate postures as letters of finger-spelled words.

20. The method of claim 17, wherein hand posture is the combination of hand orientation and hand shape, and hand pose is the combination of hand posture and hand location with respect to the user'"'"'s body, movement is the trajectory defined by the hand when going from one pose to the next, static phoneme is a pose, dynamic phoneme is movement, and a gesture is a sequence of static and dynamic phonemes.