ASL Glove with 3-Axis Accelerometers

• US 20100023314A1
• Filed: 08/08/2007
• Published: 01/28/2010
• Est. Priority Date: 08/13/2006
• Status: Abandoned Application

• Current Assignee: George Washington University
• Original Assignee: George Washington University
• Inventors: Hernandez-Rebollar, Jose
• Application Number:
  US11/836,136
• Publication Number:
  US 20100023314A1
• Time in Patent Office: Days
• US Class Current: 704/3
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 30/268 Lexical context
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems

Abstract

A sign language recognition apparatus and method is provided for translating hand gestures into speech or written text. The apparatus includes a number of 3-axis accelerometers on fingers and back of the palm to measure dynamic and static gestures, an analog multiplexer and a programmable micro controller to detect hand postures of American Sign Language and send them to a host via serial communication. The sensors are connected to a microprocessor to search a library of gestures and generate output signals that can then be used to produce a synthesized voice or written text. The apparatus includes sensors such as accelerometers on the fingers and thumb and two accelerometers on the back of the hand to detect motion and orientation of the hand. Sensors are also provided on the back of the hand or wrist to detect forearm rotation, an angle sensor to detect flexing of the elbow, two sensors on the upper arm to detect arm elevation and rotation, and a sensor on the upper arm to detect arm twist. The sensors transmit the data to the microprocessor to determine the shape, position and orientation of the hand relative to the body of the user.

Claims

1. A sign language recognition apparatus comprising an input assembly for detecting sign language, a computer connected to said input assembly and generating an output signal for producing a visual or audible output corresponding to said sign language, said input assembly comprising:

a glove to be worn by a user, said glove having 3-axis sensors for detecting dynamic hand movements of each finger and thumb;

an elbow sensor for detecting and measuring flexing and positioning of the forearm about the elbow; and

a shoulder sensor for detecting movement and position of the arm with respect to the shoulder.

2. The apparatus of claim 1, wherein said input assembly further comprises a frame having a first section for coupling to the upper arm of the user and a second section for coupling to the forearm of the user, said first sections being coupled together by a hinge, said elbow sensor being positioned on said frame for measuring flexing and positioning of the forearm, and second section.

3. The apparatus of claim 2, wherein said shoulder sensor is coupled to said first section of said frame.

4. The apparatus of claim 3, wherein said shoulder sensor comprises a first sensor for detecting twisting of the arm.

5. The apparatus of claim 4, wherein said first sensor of said shoulder sensor comprises a resistive angular sensor.

6. The apparatus of claim 4, wherein said shoulder sensor further comprises an accelerometer for detecting motion, elevation and position of the upper arm with respect to the shoulder.

7. The apparatus of claim 1, wherein said glove includes a first accelerometer on each finger and thumb, and a second accelerometer on the back of said glove to detect vertical orientation and movement of said glove, lateral orientation and movement of said glove, and longitudinal orientation and movement of said glove.

8. A method for translating a sign into a phoneme, comprising:

determining an initial and final pose of the sign, and a movement of the sign, the movement occurring between the initial and final pose, the pose of the sign comprised of an initial posture part and a hand location part, the initial and final pose and the movement measured by a sign language apparatus which uses a plurality of 3-axis sensors;

matching a determined initial posture of the sign with one or more initial postures of all known signs, and defining a first list of candidate signs as those more than one signs whose posture matches the determined initial posture or, if there is only one match, returning a first most likely sign corresponding to the match;

matching a captured initial hand location of the sign with one or more hand locations of the first list of candidate signs, and defining a second list of candidate signs as those more than one signs whose hand locations matches the determined initial hand location, or, if there is only one match, returning a second most likely sign corresponding to the match;

matching a captured movement of the sign with one or more movements of the second list of candidate signs, and defining a third list of candidate signs as those more than one signs whose movements matches the determined movements, or, if there is only one match, returning a third most likely sign corresponding to the match;

matching a determined final posture of the sign with one or more postures of the third list of candidate signs, and defining a fourth list of candidate signs as those more than one signs whose final posture matches the determined final posture, or, if there is only one match, returning a fourth most likely sign corresponding to the match;

matching a determined final hand location of the sign with one hand location of the fourth list of candidate signs, and returning a fifth most likely sign corresponding to the match; and

converting the first, second, third, fourth or fifth sign into a stream of ASCII characters to be displayed as text and/or to a voice synthesizer to be reproduced as speech.

9. The method for translating a sign of claim 8, wherein said method transmits said most likely sign as a stream of ASCII characters to be displayed as text or synthesized as voice in a language other than English

10. The method for translating signs of claim 8, wherein only the first step is used to translate postures as letters of finger-spelled words.