Sign language generation apparatus and sign language translation apparatus

• US 5,659,764 A
• Filed: 02/23/1994
• Issued: 08/19/1997
• Est. Priority Date: 02/25/1993
• Status: Expired due to Fees

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Abe, Masahiro, Inoue, Kiyoshi, Oohira, Eiji, Obuchi, Yasunari, Sagara, Kazuhiko, Sagawa, Hirohiko, Ohki, Masaru, Sakiyama, Tomoko, Toda, Yuji
• Primary Examiner(s): Hayes, Gail O.
• Assistant Examiner(s): Tkacs, Stephen R.
• Application Number:
  US08/200,702
• Time in Patent Office: 1,273 Days
• Field of Search: 364/419.01, 364/419.03, 364/513.5, 340/365, 340/709, 340/706, 340/825.19, 341/20, 434/229, 434/185, 434/169, 434/112, 395/752, 395/753, 395/757, 395/760, 395/173, 395/174, 395/175, 395/2.8
• US Class Current: 704/3
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G09B 21/009 Teaching or communicating with deaf persons G09B21/04 takes precedence; deaf and dumb persons G09B21/00

Abstract

A sign language interpretation apparatus for performing sign language recognition and sign language generation generates easily read sign language computer graphics (CG) animation by preparing sign language word CG patterns on the basis of actual motion of the hand through the use of a glove type sensor to generate natural sign language CG animation, and by applying correction to the sign language word CG patterns. Further, in the sign language interpretation apparatus, results of translation of inputted sign language or voice language are confirmed and modified easily by the individual input persons, whereby results of translation of the inputted sign language or voice language are displayed in a combined form desired by the user to realize smooth communication. Also, candidates obtained as a result of translation are all displayed and can be selected easily by the input person with a device such as a mouse. Further, when a polysemous word is available, the word is displayed while being changed in its display form, and other expressions are confirmed and modified with the mouse.

Claims

1. A sign language interpretation apparatus, comprising:

a sign language recognition and translation apparatus including;

sign language input means for inputting a sign language;

means for storing standard sign language patterns;

first translation means for recognizing the sign language input by said sign language input means by sign said standard sign language patterns, and for translating the recognized sign language into a voice language;

first display means for displaying results of translation of sign language into voice language; and

means for controlling display of the translation results displayed on said first display means, and for controlling modification of results displayed on said first display means; and

a sign language generation apparatus including;

voice language input means for inputting a voice language;

second translation means for translating a voice language input by said voice language input means into a series of sign language words;

pattern memory means for storing computer graphics patterns of sign language words;

means for generating sign language computer graphics by retrieving computer graphic patterns of sign language words from said pattern memory means in accordance with the translated sign language word series and by interpolating hand motion between sign language word computer graphic patterns;

second display means for displaying the generated sign language computer graphics; and

means for controlling display of the translation results displayed on said second display means, and for controlling modification of results displayed on said second display means;

wherein said first display means includes means for providing the following displays from which a user selects at least one;

results of said translation of sign language into a sentence in voice language, results of said translation of sign language into words in voice language, illustrations of words of said sign language input by said sign language input means, and sign language computer graphics generated by translating back the sentence in voice language, translated from the sign language input by said sign language input means, into sign language; and

wherein said second display means includes means for providing the following displays, from which a user selects at least one;

sign language computer graphics translated from the voice language input by the voice language input means, voice language words corresponding to sign language words constituting said sign language computer graphics, an illustration of sign language words, and voice language sentence.

2. A sign language interpretation apparatus according to claim 1,

wherein said first display means is provided for an aurally normal person and said second display means is provided for an aurally handicapped person;

wherein said second display means includes;

means for displaying results of said translation of sign language into voice language in the form of at least one of an image of a sign language input using a video camera as said said language input means;

means for displaying a voice language sentence resulting from the translation of the sign language input by the sign language input means into voice language, said illustrations of words in voice language corresponding to sign language words constituting the sign language input by said sign language input means;

means for displaying illustrations of said sign language words; and

means for displaying results of translating the translated voice language back into sign language computer graphics; and

wherein said first display means includes;

means for displaying results of translating the translated sign language back into voice language, in respect of only results of correction of erroneous recognition, in the form of at least one of illustrations of voice language words constituting the sign language input by said sign language input means and of sign language words, and a voice language sentence resulting from said translation of the sign language input by said sign language input means into voice language; and

means for displaying results of said translation of voice language into sign language in the form of at least one of a sentence resulting from recognition of a voice language sentence or direct input of a voice language sentence and illustrations of words constituting the voice language sentence, and of sign language words; and

wherein said second display means further displays said results of said translation of voice language into sign language, in respect of only results of correction of erroneous recognition, in the form of at least one of said voice language sentence, illustrations of word names constituting the voice language sentence, and sign language words and results of said translation of voice language into sign language computer graphics.

3. A sign language interpretation apparatus according to claim 1, wherein said second display means includes means for dividing, word-by-word, image data of the sign language, on the basis of results of recognition of individual sign language words constituting the sign language, when results of translation of sign language into voice language are displayed;

and means for displaying the divided image data together with interpreted voice language.

4. A sign language interpretation apparatus according to claim 1, wherein said second display means includes means for displaying, as an illustration of a sign language word, a still image of sign language stored in advance in respect of each word, or a key frame data for sign language computer graphics important to the action of sign language from data for sign language;

and means for displaying a setting indication.

5. A sign language interpretation apparatus according to claim 4, wherein said second display means further displays, as an illustration of a sign language word, an indication of a locus of motion.

6. A sign language interpretation apparatus according to claim 1, wherein said first display means includes:

means for displaying an indication representative of a polysemous voice language word upon display of results of translation of sign language into voice language, when a plurality of voice language words corresponding to a sign language word are available, together with one of the plurality of words or an illustration of the sign language word; and

means for displaying a pull-down menu containing the plurality of voice language words other than said one of the plurality of voice language words, by which said one of the plurality of voice language words is confirmed and changed to another of said plurality of voice language words if necessary.

7. A sign language interpretation apparatus according to claim 6, wherein the displayed selected word indicates a voice language word used in a voice language sentence translated from sign language input by said sign language input means, and wherein when a different voice language word is selected, the first display means changes the display to a voice language sentence utilizing the selected word if the results of the selection lead to a corrected sentence in voice language.

8. A sign language interpretation apparatus according to claim 1, wherein said second display means includes:

means for displaying an indication representative of a polysemous sign language word upon display of results of translation of voice language into sign language, when a plurality of sign language words corresponding to the voice language word to the translated are available, together with one of the plurality of sign language words or an illustration of one of the plurality of sign language words; and

means for displaying a pull-down menu containing illustrations of the plurality of sign language words other than said one of the plurality of sign language words, by which said one of the plurality of sign language words is confirmed and changed if necessary.

9. A sign language interpretation apparatus according to claim 8, wherein the displayed selected illustration of a sign language word indicates an illustration of a sign language word used in a sign language sentence translated from voice language input by said voice language input means, and wherein when a different sign language word is selected, the second display means changes the display to a sign language sentence utilizing the selected sign language word when the results of the selection lead to a corrected sentence in sign language.

10. A sign language interpretation apparatus according to claim 1, wherein by changing the display form of a word or illustration of a sign language word corresponding to displayed sign language computer graphics, a sign language word displayed in sign language computer graphics is confirmed.

11. A sign language interpretation apparatus according to claim 1, further comprising means for displaying a history of translation on the screen, by which the conversation that has been carried out is displayed.

12. A sign language interpretation apparatus according to claim 1, wherein said first display means and said second display means face each other.

13. A sign language interpretation apparatus according to claim 1, wherein said first display means provides, for said selection by said user, a combination of said displays of said first display means, and at least one of said displays of said second display means.