Sign language translation system and method that includes analysis of dependence relationships between successive words

• US 5,473,705 A
• Filed: 03/09/1993
• Issued: 12/05/1995
• Est. Priority Date: 03/10/1992
• Status: Expired due to Fees

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Nitin, Indurkhya, Abe, Masahiro, Sagawa, Hirohiko, Sakou, Hiroshi
• Primary Examiner(s): Boudreau, Leo H.
• Assistant Examiner(s): Johns, Andrew W.
• Application Number:
  US08/029,046
• Time in Patent Office: 1,001 Days
• Field of Search: 382/1, 382/100, 382/103, 382/107, 382/229, 348/77, 364/419.03, 364/419.04, 395/2.8, 395/2.86, 434/112, 434/116
• US Class Current: 382/100
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06F 40/55 Rule-based translation
• CPC Class Codes: G06F 40/56 Natural language generation
• CPC Class Codes: G06V 10/811 the classifiers operating on different input data, e.g. multi-modal recognition
• CPC Class Codes: G06V 30/248 involving plural approaches, e.g. verification by template match; Resolving confusion among similar patterns, e.g. "O" versus "Q" G06V30/242 takes precedence
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G09B 19/06 Foreign languages with audible presentation of material to be studied G09B5/04
• CPC Class Codes: G09B 21/00 Teaching, or communicating with, the blind, deaf or mute audible presentation of material to be studied G09B5/04; devices or methods for replacing direct visual or auditory perception by another kind of perception A61F9/08, A61F11/04; audible indication of meter readings or of colour G01D7/12; watches for blind persons G04B25/02; speech analysis, speech recognition G10L; sound-recording or reproducing, per se G11B
• CPC Class Codes: G09B 21/009 Teaching or communicating with deaf persons G09B21/04 takes precedence; deaf and dumb persons G09B21/00

Abstract

A sign language translation system and method that not only recognizes words of a sign language but also supplements omitted words between the words of the sign language, to thereby generate a spoken language. The sign language translation system has an input unit for inputting at least the motion of hands, a language generating unit responsive to the inputted motion of hands for recognizing the words corresponding to the motion of hands and generating a spoken language using the relationship between the recognized words, and an output unit for outputting the generated spoken language. The sign language translation system and method can translate a sign language into an easy-to-understand spoken language.

Claims

1. A sign language translation system, comprising:

input means for inputting motion of hands as electric signals;

sign language word generating means for recognizing words in accordance with said input electric signals and generating sign language words;

storage means for storing conjugations or translations of said generated sign language words and postpositions or auxiliary verbs to be supplemented between said generated sign language words;

dependence analyzing means for analyzing a dependence relationship between successive ones of said recognized words in accordance with said stored translations of said recognized words and outputting analyzed results;

spoken language generating means for generating, in accordance with said analyzed results, an audibly communicated sentence by supplementing said stored postpositions or auxiliary verbs and providing said stored conjugations of conjugative words; and

output means for outputting said generated spoken language.

2. A sign language translation system according to claim 1, wherein said input means comprises an input device for inputting the motion of hands at a predetermined time interval.

3. A sign language translation system according to claim 1, wherein said sign language word generating means recognizes sign language words, using a neural network which learns hand motions corresponding to said sign language words, by sequentially performing critical matching comparison between said learned hand motions and said input motion of hands.

4. A sign language translation system according to claim 1, wherein said sign language word generating means recognizes sign language words by comparing input motion of hands with stored hand motions based on a DP matching scheme.

5. A sign language translation system according to claim 1, wherein when the recognition result of said word recognizing means is unable to identify a final single word, a plurality of word candidates are sent to said dependence analyzing means.

6. A sign language translation system according to claim 1, wherein said dependence analyzing means analyzes the dependence relationship by performing the steps of:

storing said recognized words into a stack memory in the sequence of word recognition;

producing an analyzing table containing cases, parts of speech and translations of said stacked word;

determining a word to be analyzed in the stack;

searching words, from the stack, which are capable of depending on said determined word in accordance with said analyzing table; and

storing said searched words into a memory.

7. A sign language translation system according to claim 6, wherein said dependence analyzing means determines a word to be analyzed in the stack when the word is a declinable word, wherein a declinable word indicates a word describing an action, function, existence, property or condition thereof.

8. A sign language translation system according to claim 7, wherein said dependence analyzing means moves said searched words corresponding to said determined word from the stack to said analyzing table together with information concerning the dependence relationship between the determined word and the searched words, wherein said determined word remains in the stack.

9. A sign language translation system according to claim 8, wherein said spoken language generating means supplements said stored postpositions or auxiliary verbs in accordance with said moved words in said analyzing table and corresponding information concerning a dependence relationship between the determined word and the moved words.

10. A sign language translation system according to claim 7, wherein said dependence analyzing means further performs dependence analysis with respect to substantives.

11. A sign language translation system according to claim 1, wherein said omission/supplement means supplements omitted words and provides conjugations of conjugative words, in accordance with the words recognized by said word recognizing means and an omission/supplement rule determined by said dependence analyzing means.

12. A sign language translation system according to claim 1, wherein each said omitted word to be supplemented is at least one of an auxiliary verb, postposition, pseudo pronoun, and conjunction, respectively representing a semantic and time sequential relationships between words.

13. A sign language translation system according to claim 1, wherein said output means outputs said generated spoken language in at least one of a text representation and voice expression wherein the dependence between said recognized words is analyzed using said recognized results, and a predetermined process is executed using said analyzed results to generate said spoken language by supplementing omitted words and providing the conjugations of a conjugative word.

14. A sign language translation system according to claim 1, wherein said output means outputs synthesized images of the hand motion corresponding to a sign language wherein the dependence between said recognized words is analyzed using said recognized results, and a predetermined process is executed using said analyzed results to generate said spoken language by supplementing omitted words and providing the conjugations of a conjugative word.

15. A sign language translation system according to claim 1, wherein said output means outputs a train of words recognized when a spoken language cannot be generated.

16. A sign language translation system according to claim 1, wherein said language generating means further comprises:

word reading means for reading the words recognized by said word recognizing means;

analysis stacking means for individually stacking the read words comprising a sentence;

speech determining means for determining the part of speech of each word stacked in said analysis stacking means;

a case dictionary for determining the conjugation of each word in the analysis stacking means relative to the verb determined by said determining means; and

an analysis table for receiving the properly conjugated words from said analysis stacking means relative to the verb.

17. A sign language translation system according to claim 1, wherein, when said dependence analyzing means fails to analyze a dependence relationship between said determined word and said searched words, said spoken language generating means outputs recognized words without supplementing said stored postpositions or auxiliary verbs and without providing said stored conjugations of conjugative words.

18. A sign language translation method comprising the steps of:

inputting motion of hands;

recognizing words corresponding to said input motion of hands;

generating a spoken language using a dependence relationship between said recognized words;

wherein the dependence between said recognized words is analyzed using said recognized results, and a predetermined process is executed using said analyzed results to generate said spoken language by supplementing omitted words and providing the conjugations of a conjugative word; and

outputting said generated spoken language.

19. A sign language translation method according to claim 18, wherein spatial information representing a pronoun and time lapse entered in a sign language is recognized to translate said spatial information into a corresponding word.