Electronic translator for assisting communications
First Claim
1. An electronic translator for translating a text data stream into one or more forms in real time comprising:
- a database containing a plurality of video clips of an actual person signing words, each of said video clips showing the signing of at least one word, said database also including a plurality of finger spelling video clips of an actual person signing individual letters of the alphabet;
a processor, said processor including a text-to-sign language translator for converting said text data stream into a video stream of an actual person using sign language to sign words in said data stream, said text-to-sign language translator including programming responsive to said text data stream for detecting words therein, sequentially retrieving video clips in said database showing signing of said words, and displaying said video clips sequentially on said video display, said text-to-sign language translator further including programming for determining whether a word in said text data stream does not have a corresponding signing video clip in said database, and if this is the case, accessing said finger spelling video clips to display sign language finger spelling of said word; and
a video display interfaced to said processor for selectively displaying said video stream and text corresponding to said text data stream.
2 Assignments
0 Petitions
Accused Products
Abstract
An electronic translator translates input speech into multiple streams of data that are simultaneously delivered to the user, such as a hearing impaired individual. Preferably, the data is delivered in audible, visual and text formats. These multiple data streams are delivered to the hearing-impaired individual in a synchronized fashion, thereby creating a cognitive response. Preferably, the system of the present invention converts the input speech to a text format, and then translates the text to any of three other forms, including sign language, animation and computer generated speech. The sign language and animation translations are preferably implemented by using the medium of digital movies in which videos of a person signing words, phrase and finger spelled words, and of animations corresponding to the words, are selectively accessed from databases and displayed. Additionally the received speech is converted to computer-generated speech for input to various hearing enhancement devices used by the deaf or hearing-impaired, such as cochlear implants and hearing aids, or other output devices such as speakers, etc. The data streams are synchronized utilizing a high-speed personal computer to facilitate sufficiently fast processing that the text, video signing and audible streams can be generated simultaneously in real time. Once synchronized the data streams are presented to the subject concurrently in a method that allows the process of mental comprehension to occur. The electronic translator can also be interfaced to other communications devices, such as telephones. Preferably, the hearing-impaired person is also able to use the system'"'"'s keyboard or mouse to converse or respond.
-
Citations
68 Claims
-
1. An electronic translator for translating a text data stream into one or more forms in real time comprising:
-
a database containing a plurality of video clips of an actual person signing words, each of said video clips showing the signing of at least one word, said database also including a plurality of finger spelling video clips of an actual person signing individual letters of the alphabet;
a processor, said processor including a text-to-sign language translator for converting said text data stream into a video stream of an actual person using sign language to sign words in said data stream, said text-to-sign language translator including programming responsive to said text data stream for detecting words therein, sequentially retrieving video clips in said database showing signing of said words, and displaying said video clips sequentially on said video display, said text-to-sign language translator further including programming for determining whether a word in said text data stream does not have a corresponding signing video clip in said database, and if this is the case, accessing said finger spelling video clips to display sign language finger spelling of said word; and
a video display interfaced to said processor for selectively displaying said video stream and text corresponding to said text data stream. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34)
a keyboard interfaced to said processor for entering a text data stream to be translated by said text-to-sign language translator;
an audio receiver for detecting a person'"'"'s speech and generating a first speech signal in response thereto;
a speech-to-text translator for converting said first speech signal into a text data stream to be translated by said text-to-sign language translator;
a text-to-speech translator in said processor for translating said text data stream into a second speech signal;
at least one audio output device interfaced to said processor for receiving said second speech signal and generating a speech sound simultaneously with the display of said video stream on said video display; and
wherein each of said video clips begins and ends with said person signing said words in a selected position to facilitate seamless sequencing of said clips in said video stream.
-
-
23. The electronic translator of claim 22, further including at least one additional translator in said processor and associated database interfaced to said processor selected from the group comprising a text-to-image translator for translating said text into one or more images contained in an image database that are associated with words in said text data stream, and displaying said images simultaneously with said words on said video display;
- a text-to-phonetic spelling translator for translating words in said text data stream into text of phonetic spellings of said words contained in a phonetic spelling database, and displaying said phonetic spelling text on said video display; and
, a text-to-audio sound translator for translating words in said text data stream into environmental sounds that are related to said words and are contained in an environmental sounds database, and playing said audio sounds on said audio output device.
- a text-to-phonetic spelling translator for translating words in said text data stream into text of phonetic spellings of said words contained in a phonetic spelling database, and displaying said phonetic spelling text on said video display; and
-
24. The electronic translator of claim 22, wherein said audio output device comprises a cochlear implant, hearing aid or other hearing enhancement device, whereby, a hearing impaired person can identify speech sounds generated by said cochlear implant, hearing aid or other hearing enhancement device as corresponding to words that are signed in said video stream.
-
25. The electronic translator of claim 22, wherein said database further includes a plurality of finger spelling video movies for signing individual letters of the alphabet, and said text-to-sign language translator further includes programming for determining that a word in said text data stream does not have a corresponding signing video in said database, and accessing said finger spelling videos to display sign language spelling of said word.
-
26. The electronic translator of claim 22, wherein said text-to-sign language translator includes a speed control for adjusting a play speed of said video clips to enable synchronization of said video clips with said text data stream as they are simultaneously displayed on said video display.
-
27. The electronic translator of claim 22, wherein said processor further includes programming for simultaneously displaying text, corresponding to said text data stream, on said display in a first, scrolling marquee format and in a second, static format.
-
28. The electronic translator of claim 22, wherein said processor further includes programming for displaying a plurality of information windows on said video display, including at least a first window for displaying said video stream and a second window for displaying text corresponding to said text data stream.
-
29. The electronic translator of claim 22, wherein said at least one audio output device comprises a cochlear implant, hearing aid or other hearing enhancement device and one or more audio speakers, and said translator further comprises a sound card interfaced to said processor, said sound card having a first output channel interfaced to said cochlear implant, hearing aid or other hearing enhancement device and a second output channel interfaced to said audio speaker, and, wherein, said processor is programmed to switch on said second channel only if said text data stream translated by said text-to-speech translator has been generated by said keyboard to avoid generation of feedback between said audio speaker and said audio receiver when said text data stream is generated by said speech-to-text translator.
-
30. The electronic translator of claim 22, wherein said processor further includes programming for displaying text corresponding to said text data stream simultaneously with the generation of said speech sound and display of said video stream on said video display.
-
31. The electronic translator of claim 22, wherein said processor further includes programming for highlighting words in said text being displayed on said video display as said highlighted words are being signed by said text-to-sign language translator, and generated by said audio output device.
-
32. The electronic translator of claim 31, wherein said audio output device comprises a cochlear implant, hearing aid or other hearing enhancement device, whereby, a hearing impaired person can identify speech sounds generated by said cochlear implant, hearing aid or other hearing enhancement device as corresponding to words that are signed in said video stream and highlighted in said text on said video display.
-
33. The electronic translator of claim 22, wherein said processor further includes programming for establishing a telephone connection with a telephone, transmitting said second speech signal over a transmitter in said telephone, and receiving a third speech signal from a receiver in said telephone for translation by said speech-to-text translator and display on said video display.
-
34. The electronic translator of claim 22, wherein said processor and said display are incorporated in a laptop computer.
-
35. An electronic translator for translating speech into multiple forms in real time comprising:
-
an audio receiver for detecting a person'"'"'s speech and generating a first speech signal in response thereto;
a keyboard for entering a first text data stream to be translated;
a processor, said processor including;
a speech-to-text translator for converting said first speech signal into a second text data stream; and
a text-to-speech translator for selectively converting said first or said second text data streams into a second speech signal;
a video display interfaced to said processor for displaying text that corresponds to said first and second text data streams; and
an audio output device interfaced to said processor for receiving said second speech signal and generating a speech sound simultaneously with the display of said text. - View Dependent Claims (36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48)
-
-
49. A method for translating a text data stream in real time comprising the steps of:
-
generating a text data stream;
translating said text data stream into a video stream of a person using sign language to sign words in said data stream by accessing a database containing a plurality of video clips of a person signing words, each of said video clips showing the signing for at least one particular word, said database also containing a plurality of finger spelling video clips showing a person signing individual letters of the alphabet;
sequentially retrieving video clips in said database showing signing of said words in said text data stream for any of said words that are determined to have a corresponding signing video in said database, and/or finger spelling video clips corresponding to the spelling of any of said words in said text data stream that are determined not to have a corresponding video in said database; and
, combining said sequentially retrieved video clips to form said video stream;
displaying text that corresponds to said text data stream on a video display; and
displaying said video stream simultaneously with said text on said display. - View Dependent Claims (50, 51, 52, 53, 54, 55, 56, 57, 58)
1) displaying a phonetic spelling representation of a word as it is signed in said video stream;
2) playing an audio sound that is related to a word as it is signed in said video stream; and
3) displaying one or more images that are associated with said text as said text is displayed.
-
-
57. The method of claim 49, wherein the step of generating a text data stream further comprises entering said text data stream with a keyboard.
-
58. The method of claim 49, wherein the step of generating a text data stream further comprises generating a speech signal in response to a person speaking, and translating said speech signal into said text data stream.
-
59. A method for translating speech into multiple formats in real time comprising the steps of:
-
generating a first speech signal in response to a person speaking;
translating said speech signal into a text data stream;
translating said text data stream into a second speech signal;
driving an audio output device with said second speech signal; and
simultaneously displaying text that corresponds to said text data stream on a video display. - View Dependent Claims (60, 61, 62, 63, 64, 65, 66, 67, 68)
translating said text data stream into a video stream of a person using sign language to sign words in said data stream by accessing a database containing a plurality of video clips of a person signing words, each of said video clips showing the signing for at least one particular word; and
sequentially retrieving video clips in said database showing signing of said words in said text data stream to form said video stream; and
displaying said video stream simultaneously with said text on said display.
-
-
63. The method of claim 62, wherein said database further includes a plurality of finger spelling video movies for signing individual letters of the alphabet, and said method further comprise the steps of determining whether a word in said text data stream has a corresponding signing video in said database, if not, sequentially retrieving finger spelling videos corresponding to the spelling of said word.
-
64. The method of claim 62, further comprising the step of adjusting a playback speed of said video clips to synchronize of said video clips with said text as said video stream and said text are simultaneously displayed on said video display.
-
65. The method of claim 62, wherein each of said video clips is selected to begin and end with a person in a selected position to facilitate seamless sequencing of said clips in said video stream.
-
66. The method of claim 62, further including the step of highlighting words in said text being displayed on said video display as said highlighted words are being signed in said video stream.
-
67. The method of claim 59, further comprising one or more of the steps selected from the group comprising:
-
1) displaying a phonetic spelling representation of a word as it is signed in said video stream;
2) playing an audio sound that is related to a word as it is signed in said video stream; and
3) displaying one or more images that are associated with said text as said text is displayed.
-
-
68. The method of claim 59, further comprising the steps of establishing a telephone connection with a telephone, transmitting said audio signal over a transmitter in said telephone, receiving a second speech input signal from a receiver in said telephone, translating said second speech signal into a second text data stream, and displaying said second text data stream on said video display.
Specification