Apparatus for bi-directional sign language/speech translation in real time and method

• US 10,089,901 B2
• Filed: 06/21/2016
• Issued: 10/02/2018
• Est. Priority Date: 02/11/2016
• Status: Active Grant

• Current Assignee: Hyundai Motor Company (Hyundai Motor Group), Kia Corp.
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: Jung, Woo Sug, Kim, Hwa Suk, Jeon, Jun Ki, Kim, Sun Joong, Lee, Hyun Woo
• Primary Examiner(s): Wozniak, James
• Application Number:
  US15/188,099
• Publication Number:
  US 20170236450A1
• Time in Patent Office: 833 Days
• Field of Search: 704235, 704251, 704271
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G09B 21/00 Teaching, or communicating with, the blind, deaf or mute audible presentation of material to be studied G09B5/04; devices or methods for replacing direct visual or auditory perception by another kind of perception A61F9/08, A61F11/04; audible indication of meter readings or of colour G01D7/12; watches for blind persons G04B25/02; speech analysis, speech recognition G10L; sound-recording or reproducing, per se G11B
• CPC Class Codes: G09B 21/009 Teaching or communicating with deaf persons G09B21/04 takes precedence; deaf and dumb persons G09B21/00
• CPC Class Codes: G10L 13/027 Concept to speech synthesisers; Generation of natural phrases from machine-based concepts generation of parameters for speech synthesis out of text G10L13/08
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2021/065 Aids for the handicapped in understanding
• CPC Class Codes: G10L 21/06 Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids G10L15/26 takes precedence

Abstract

Provided is an apparatus for bi-directional sign language/speech translation in real time and method that may automatically translate a sign into a speech or a speech into a sign in real time by separately performing an operation of recognizing a speech externally made through a microphone and outputting a sign corresponding to the speech, and an operation of recognizing a sign sensed through a camera and outputting a speech corresponding to the sign.

Claims

1. An apparatus for bi-directional sign language/speech translation in real time comprising:

a processor; and

a non-transitory computer readable medium storing program instructions, when executed, causing the processor to;

analyze a used pattern of sign language by a user in view of current surrounding environment information of the user to generate sign category information via a pattern analyzer;

recognize a speech externally made through a microphone via a speech-sign outputter and output a sign corresponding to the speech via a display of the speech-sign outputter; and

recognize a sign sensed through a camera via a sign-speech outputter and output a speech corresponding to the sign via a speaker of the sign-speech outputter,

wherein the pattern analyzer transmits the sign category information to the speech-sign outputter to output a text or a sign corresponding to the sign category information via the display or to the sign-speech outputter to output a speech corresponding to the sign category information via the speaker,

wherein the pattern analyzer comprises a sign category keyword comparator configured to compare the sign category information with the sign corresponding to the speech or the speech corresponding to the sign to determine whether the sign category information is correct, and

wherein the sign category keyword comparator transmits a signal related to whether the sign category information is correct to the speech-sign outputter or the sign-speech outputter to block the text, the sign, or the speech corresponding to the sign category information.

2. The apparatus for bi-directional sign language/speech translation in real time of claim 1, wherein the speech-sign outputter comprises:

a speech recognizer configured to recognize the speech;

an index generator configured to generate a sign index to translate into a sign corresponding to the recognized speech; and

a sign outputter configured to output the sign corresponding to the recognized speech based on the generated sign index.

3. The apparatus for bi-directional sign language/speech translation in real time of claim 2, wherein the index generator comprises:

a speech-text converter configured to convert the recognized speech into a text using a predefined sign language dictionary; and

an index determiner configured to determine a sign index with respect to the text based on the text and the used pattern of sign language by the user.

4. The apparatus for bi-directional sign language/speech translation in real time of claim 2, wherein the sign outputter comprises:

a display mode controller configured to control an output based on a display mode to display one of a sign and a text corresponding to the recognized speech; and

an outputter configured to output a sign mapped to the generated sign index or a text corresponding to the sign mapped to the generated sign index based on the display mode.

5. The apparatus for bi-directional sign language/speech translation in real time of claim 4, wherein the display mode controller is configured to control the display mode of the outputter based on a sign display event or a generation period of the sign mapped to the sign index.

6. The apparatus for bi-directional sign language/speech translation in real time of claim 5, wherein the outputter is configured to output the sign mapped to the generated sign index or the text corresponding to the sign mapped to the generated sign index on a display of the apparatus for bi-directional sign language/speech translation in real time based on the display mode which is controlled based on the sign display event or the generation period of the sign mapped to the sign index.

7. The apparatus for bi-directional sign language/speech translation in real time of claim 1, wherein the pattern analyzer is configured to analyze the used pattern of sign language by the user by analyzing at least one of location information of the user, surrounding environment information of the user corresponding to the location information, a life pattern of the user, or a behavior pattern of the user.

8. The apparatus for bi-directional sign language/speech translation in real time of claim 1, wherein the sign-speech outputter comprises:

a sign recognizer configured to recognize the sign sensed through the camera;

an index generator configured to generate a sign index to translate into the speech corresponding to the recognized sign; and

a speech outputter configured to output the speech corresponding to the sign based on the generated sign index.

9. The apparatus for bi-directional sign language/speech translation in real time of claim 8, wherein the sign recognizer comprises:

a wearing sensor configured to sense whether the user is wearing gesture gloves; and

a sign collector configured to collect the sign based on whether the user is wearing the gesture gloves.

10. The apparatus for bi-directional sign language/speech translation in real time of claim 9, wherein the sign collector is configured to collect the sign by removing a background from a color image acquired by the camera when the user is not wearing the gesture gloves.

11. The apparatus for bi-directional sign language/speech translation in real time of claim 9, wherein the sign collector is configured to collect the sign based on a finger motion of the user collected from the camera when the user is wearing the gesture gloves.

12. The apparatus for bi-directional sign language/speech translation in real time of claim 8, wherein the index generator comprises:

an index determiner configured to determine the sign index with respect to the recognized sign using a predefined sign language dictionary;

a sign-text converter configured to convert the recognized sign to a text based on the determined sign index; and

a sentence generator configured to generate a sentence associated with the sign with respect to the text through a keyword combination corresponding to the text and the used pattern of sign language by the user.

13. The apparatus for bi-directional sign language/speech translation in real time of claim 12, wherein the speech outputter is configured to output a speech corresponding to the sentence associated with the sign with respect to the text.

14. A method for bi-directional sign language/speech translation in real time performed by an apparatus for bi-directional sign language/speech translation in real time, the method comprising:

analyzing a used pattern of sign language by a user who uses the apparatus for bi-directional sign language/speech translation in real time in view of current surrounding environment information of the user to generate a sign category information;

recognizing a first sign or a first speech externally made by the user through a camera or a microphone, respectively;

identifying a second sign corresponding to the first speech or a second speech corresponding to the first sign using the sign category information; and

outputting the second sign or the second speech through a different translation path based on a result of the identifying,

wherein the analyzing comprises;

comparing the sign category information with the second sign corresponding to the first speech or the second speech corresponding to the first sign to determine whether the sign category information is correct, and

transmitting a signal related to whether the sign category information is correct to the speech-sign outputter or the sign-speech outputter to block the second sign or the second speech corresponding to the sign category information.

15. The method for bi-directional sign language/speech translation in real time of claim 14, wherein the outputting the second sign or the second speech comprises:

generating a sign index to translate into the second sign corresponding to the recognized first speech; and

outputting the second sign corresponding to the recognized first speech based on the generated sign index.

16. The method for bi-directional sign language/speech translation in real time of claim 15, wherein the generating comprises:

converting the recognized first speech into a text using a predefined sign language dictionary; and

determining a sign index with respect to the text based on the text and the used pattern of sign language by the user.

17. The method for bi-directional sign language/speech translation in real time of claim 15, wherein the outputting of the second sign corresponding to the recognized first speech comprises:

controlling whether to display a sign or a text based on a display mode to display one of a sign and a text corresponding to the recognized first speech; and

outputting a sign mapped to the generated sign index or a text corresponding to the sign mapped to the generated sign index based on the display mode.

18. The method for bi-directional sign language/speech translation in real time of claim 14, wherein the outputting the second sign or the second speech comprises:

recognizing the first sign externally made by the user;

generating a sign index to translate into the second speech corresponding to the recognized first sign; and

outputting the second speech corresponding to the recognized first sign based on the generated sign index.

19. The method for bi-directional sign language/speech translation in real time of claim 18, wherein the recognizing comprises:

sensing whether the user is wearing gesture gloves; and

collecting the first sign based on whether the user is wearing the gesture gloves.

20. The method for bi-directional sign language/speech translation in real time of claim 18, wherein the generating comprises:

determining the sign index with respect to the recognized first sign using a predefined sign language dictionary;

converting the recognized first sign to a text based on the determined sign index; and

generating a sentence associated with the first sign with respect to the text through a keyword combination corresponding to the text and the used pattern of sign language by the user.