System and Method For Automated Sign Language Recognition

• US 20160307469A1
• Filed: 04/15/2016
• Published: 10/20/2016
• Est. Priority Date: 04/16/2015
• Status: Active Grant

• Current Assignee: Robert Bosch GmbH
• Original Assignee: Robert Bosch GmbH
• Inventors: Zhou, Zhengyu, Menne, Tobias, Li, Kehuang, Xu, Kui, Feng, Zhe, Lee, Chin-Hui
• Granted Patent: US 10,109,219 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/21 Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
• CPC Class Codes: G06F 18/295 Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G09B 21/009 Teaching or communicating with deaf persons G09B21/04 takes precedence; deaf and dumb persons G09B21/00
• CPC Class Codes: G10L 15/142 Hidden Markov Models [HMMs]
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features

Abstract

A method for sign language recognition includes receiving, with an input device, an input based on a plurality of hand movements and postures of a user that correspond to a sequence of signs, extracting a plurality of features from the input corresponding to the plurality of hand movements and postures, identifying, a start of the sequence of signs in the input based on a first set of features in the plurality of features and a first Hidden Markov Model (HMM) stored in the memory and identifying a first sign in the input based on a second set of features in the plurality of features and a second HMM stored in the memory. The method also includes generating an output corresponding to the first sign from the input.

Claims

1. A training method for automated sign language recognition comprising:

receiving, with an input device, a plurality of training inputs corresponding to a plurality of predetermined sequences of signs from hand movements and postures of a user;

extracting, with the processor, a first set of features from the training inputs corresponding to hand movements and postures of the user for each sign in the plurality of predetermined sequences of signs;

generating, with the processor, a first Hidden Markov Model (HMM) based on the first set of features from the training inputs corresponding to hand movements and postures for each sign in the predetermined sequences of signs;

extracting, with the processor, a second set of features from the training inputs corresponding to hand movements and postures of the user for transitions between signs in the plurality of predetermined sequences of signs;

generating, with the processor, a second HMM based on the second set of features from the training inputs and the predetermined sequences of signs;

extracting, with the processor, a third set of features from the training inputs corresponding to hand movements and postures of the user for starting and ending each predetermined sequence of signs in the plurality of predetermined sequences of signs;

generating with the processor a third HMM based on the third set of features from the training inputs and the predetermined sequences of signs; and

storing, with the processor, the first HMM, second HMM, and third HMM in a memory for recognition of additional signs received from the input device.

2. The method of claim 1, the receiving of the plurality of training inputs further comprising:

receiving, with the input device, a first plurality of inputs from a dominant hand of the user corresponding to the predetermined sequences of signs;

receiving, with the input device, a second plurality of inputs from a non-dominant hand of the user corresponding to the predetermined sequences of signs.

3. The method of claim 2, the receiving of the training inputs further comprising

receiving, from a first plurality of sensors in a glove worn on the dominant hand of the user, a first set of finger position data corresponding to at least two fingers in the dominant hand of the user;

receiving, from the first plurality of sensors in the glove worn on the dominant hand of the user, a first set of hand angle data corresponding to an angle of the dominant hand of the user; and

receiving, from a second plurality of sensors in a glove worn on the non-dominant hand of the user, a second set of hand angle data corresponding to an angle of the non-dominant hand of the user.

4. The method of claim 3, the extraction of the first set of features further comprising:

identifying, with the processor, at least one feature in the first set of features as a cosine distance between a first finger and a second finger in the at least two fingers of the dominant hand based on the first set of finger position data.

5. The method of claim 3, the extraction of the second set of features further comprising:

identifying, with the processor, a first hand angle in the first set of hand angle data generated at a first time, the first hand angle being parallel to a predetermined direction of gravity;

identifying, with the processor, a second hand angle in the first set of hand angle data generated at a second time; and

extracting, with the processor, an angular position of the dominant hand as a feature in the second set of features based on the first hand angle, the second hand angle, and a quaternion.

6. The method of claim 3, the extraction of the third set of features further comprising:

identifying, with the processor, a first hand angle in the second set of hand angle data generated at a first time, the first hand angle being parallel to a predetermined direction of gravity;

identifying, with the processor, a second hand angle in the second set of hand angle data generated at a second time; and

extracting, with the processor, an angular position of the non-dominant hand as a feature in the second set of features based on the first hand angle, the second hand angle, and a quaternion.

7. The method of claim 1, the generation of the first HMM further comprising:

generating, with the processor, the first HMM based on a first portion of the first set of features from the training inputs corresponding to the hand movements and postures for a dominant hand of the user for each sign in the plurality of predetermined sequences of signs and a second portion of the second set of features from the training data corresponding to the hand movements and postures of a non-dominant hand of the user for the transitions between signs in the plurality of predetermined sequences of signs.

8. A method for automated sign language recognition comprising:

receiving, with an input device, an input based on a plurality of hand movements and postures of a user that correspond to a sequence of signs;

extracting, with a processor, a plurality of features from the input corresponding to the plurality of hand movements and postures;

identifying, with the processor, a start of the sequence of signs in the input based on a first set of features in the plurality of features and a first Hidden Markov Model (HMM) stored in the memory;

identifying, with the processor, a first sign in the input based on a second set of features in the plurality of features and a second HMM stored in the memory; and

generating, with an output device, an output corresponding to the first sign from the input.

9. The method of claim 8 further comprising:

identifying, with the processor, a first transition corresponding to hand movements and postures that occur between signs in the input based on a third set of features in the plurality of features and a third HMM stored in the memory;

identifying, with the processor, a second sign in the input based on a fourth set of features in the plurality of features and the second HMM stored in the memory; and

10. The method of claim 9 further comprising:

identifying, with the processor, an end of the sequence of signs in the input based on a fifth set of features in the plurality of features and the first HMM stored in the memory; and

generating, with the output device, the output corresponding to the first sign and the second from the input after identifying the end of the sequence of signs.

11. The method of claim 8, the receiving of the input further comprising:

receiving, with the input device, a first plurality of inputs corresponding to a plurality of movements of a dominant hand of the user; and

receiving, with the input device, a second plurality of inputs corresponding to a plurality of movements of a non-dominant hand of the user.

12. The method of claim 11, the receiving of the input further comprising

receiving, from a first plurality of sensors in a glove worn on the dominant hand of the user, a first set of finger position data corresponding to at least two fingers in the dominant hand of the user;

receiving, from the first plurality of sensors in the glove worn on the dominant hand of the user, a first set of hand angle data corresponding to an angle of the dominant hand of the user; and

receiving, from a second plurality of sensors in a glove worn on the non-dominant hand of the user, a second set of hand angle data corresponding to an angle of the non-dominant hand of the user.

13. The method of claim 12, the extraction of the features further comprising:

identifying, with the processor, at least one feature as a cosine distance between a first finger and a second finger in the at least two fingers of the dominant hand based on the first set of finger position data.

14. The method of claim 12, the extraction of the features further comprising:

identifying, with the processor, a first hand angle in the first set of hand angle data generated at a first time, the first hand angle being parallel to a predetermined direction of gravity;

identifying, with the processor, a second hand angle in the first set of hand angle data generated at a second time; and

extracting, with the processor, a position of the dominant hand as a feature based on the first hand angle, the second hand angle, and a quaternion.

15. The method of claim 12, the extraction of the features further comprising:

identifying, with the processor, a first hand angle in the second set of hand angle data generated at a first time, the first hand angle being parallel to a predetermined direction of gravity;

identifying, with the processor, a second hand angle in the second set of hand angle data generated at a second time; and

extracting, with the processor, a position of the non-dominant hand as a feature in the second set of features based on the first hand angle, the second hand angle, and a quaternion.

16. A system for automated sign language recognition comprising:

an input device configured to receive input corresponding to a plurality of hand movements and postures from a user corresponding to a sequence of signs;

an output device;

a memory; and

a processor operatively connected to the input device, the output device, and the memory, the processor being configured to;

receive an input from the input device based on a plurality of hand movements and postures of a user that correspond to a sequence of sign;

extract a plurality of features from the input corresponding to the plurality of hand movements and postures;

identify a start of the sequence of signs in the input based on a first set of features in the plurality of features and a first Hidden Markov Model (HMM) stored in the memory;

identify a first sign in the input based on a second set of features in the plurality of features and a second HMM stored in the memory; and

generate an output with the output device corresponding to the first sign from the input.

17. The system of claim 16, the processor being further configured to:

identify a first transition corresponding to hand movements and postures that occur between signs in the input based on a third set of features in the plurality of features and a third HMM stored in the memory;

identify a second sign in the input based on a fourth set of features in the plurality of features and the second HMM stored in the memory; and

generate the output corresponding to the first sign and the second from the input.

18. The system of claim 17, the processor being further configured to:

identify an end of the sequence of signs in the input based on a fifth set of features in the plurality of features and the first HMM stored in the memory; and

generate the output corresponding to the first sign and the second from the input after identifying the end of the sequence of signs.

19. The system of claim 16, the processor being further configured to:

receive a first plurality of inputs corresponding to a plurality of movements of a dominant hand of the user from the input device; and

receive a second plurality of inputs corresponding to a plurality of movements of a non-dominant hand of the user from the input device.

20. The system of claim 19 the input device further comprising:

a first glove including a first plurality of sensors;

a second glove including a second plurality of sensors; and

the processor being further configured to;

receive, from the first plurality of sensors in the first glove worn on the dominant hand of the user, a first set of finger position data corresponding to at least two fingers in the dominant hand of the user;

receive, from the first plurality of sensors in the first glove worn on the dominant hand of the user, a first set of hand angle data corresponding to an angle of the dominant hand of the user; and

receive, from the second plurality of sensors in the glove worn on the non-dominant hand of the user, a second set of hand angle data corresponding to an angle of the non-dominant hand of the user.