Electronic translator for assisting communications

• US 6,377,925 B1
• Filed: 07/07/2000
• Issued: 04/23/2002
• Est. Priority Date: 12/16/1999
• Status: Expired due to Term

• Current Assignee: P.P.R. Direct, Inc.
• Original Assignee: Interactive Solutions
• Inventors: Dorety, Michael F., Greene, Virginia, Newman, Harry E., Yuhas, Mark J., Greene, Morgan Jr.
• Primary Examiner(s): {haeck over (S)}mits, Ta̅livaldis Ivars
• Application Number:
  US09/610,675
• Time in Patent Office: 655 Days
• Field of Search: 704/235, 704/260, 704/271, 379/52
• US Class Current: 704/271
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 21/06 Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids G10L15/26 takes precedence
• CPC Class Codes: H04M 2201/50 Telephonic communication in combination with video communication
• CPC Class Codes: H04M 2201/60 Medium conversion
• CPC Class Codes: H04M 3/42391 where the subscribers are hearing-impaired persons, e.g. telephone devices for the deaf
• CPC Class Codes: H04R 1/1041 Mechanical or electronic switches, or control elements switches in general H01H
• CPC Class Codes: H04R 2205/041 Adaptation of stereophonic signal reproduction for the hearing impaired
• CPC Class Codes: H04R 25/505 using digital signal processing
• CPC Class Codes: H04R 25/70 Adaptation of deaf aid to hearing loss, e.g. initial electronic fitting

Abstract

An electronic translator translates input speech into multiple streams of data that are simultaneously delivered to the user, such as a hearing impaired individual. Preferably, the data is delivered in audible, visual and text formats. These multiple data streams are delivered to the hearing-impaired individual in a synchronized fashion, thereby creating a cognitive response. Preferably, the system of the present invention converts the input speech to a text format, and then translates the text to any of three other forms, including sign language, animation and computer generated speech. The sign language and animation translations are preferably implemented by using the medium of digital movies in which videos of a person signing words, phrase and finger spelled words, and of animations corresponding to the words, are selectively accessed from databases and displayed. Additionally the received speech is converted to computer-generated speech for input to various hearing enhancement devices used by the deaf or hearing-impaired, such as cochlear implants and hearing aids, or other output devices such as speakers, etc. The data streams are synchronized utilizing a high-speed personal computer to facilitate sufficiently fast processing that the text, video signing and audible streams can be generated simultaneously in real time. Once synchronized the data streams are presented to the subject concurrently in a method that allows the process of mental comprehension to occur. The electronic translator can also be interfaced to other communications devices, such as telephones. Preferably, the hearing-impaired person is also able to use the system'"'"'s keyboard or mouse to converse or respond.

Claims

1. An electronic translator for translating a text data stream into one or more forms in real time comprising:

a database containing a plurality of video clips of an actual person signing words, each of said video clips showing the signing of at least one word, said database also including a plurality of finger spelling video clips of an actual person signing individual letters of the alphabet;

a processor, said processor including a text-to-sign language translator for converting said text data stream into a video stream of an actual person using sign language to sign words in said data stream, said text-to-sign language translator including programming responsive to said text data stream for detecting words therein, sequentially retrieving video clips in said database showing signing of said words, and displaying said video clips sequentially on said video display, said text-to-sign language translator further including programming for determining whether a word in said text data stream does not have a corresponding signing video clip in said database, and if this is the case, accessing said finger spelling video clips to display sign language finger spelling of said word; and

a video display interfaced to said processor for selectively displaying said video stream and text corresponding to said text data stream.

2. The electronic translator of claim 1, wherein said processor further includes a text-to-speech translator for converting said text data stream into a speech signal, and said translator further comprises an audio output device for receiving said speech signal, and generating speech sounds in response thereto.

3. The electronic translator of claim 2, wherein said audio output device comprises a cochlear implant, hearing aid or other hearing enhancement device whereby, a hearing impaired person can identify speech sounds generated by said cochlear implant, hearing aid or other hearing enhancement device as corresponding to words that are signed in said video stream.

4. The electronic translator of claim 1, wherein said text-to-sign language translator includes a speed control for adjusting a play speed of said video clips to enable synchronization of said video clips with said text data stream as they are simultaneously displayed on said video display.

5. The electronic translator of claim 1, wherein each of said video clips begin and end with a person in a selected position to facilitate seamless sequencing of said clips in said video stream.

6. The electronic translator of claim 1, further comprising a text-to-image translator in said processor for translating said text into one or more images that are associated with words in said text data stream, and displaying said images simultaneously with said words on said video display;

and, an image database interfaced to said processor containing said images.

7. The electronic translator of claim 1, wherein said processor further includes programming for simultaneously displaying text corresponding to said text data stream on said display in a first, scrolling marquee format and in a second, static format.

8. The electronic translator of claim 1, wherein said processor further includes programming for displaying a plurality of information windows on said video display, including at least a first window for displaying said video stream and a second window for simultaneously displaying text corresponding to said text data stream.

9. The electronic translator of claim 8, wherein said processor further includes programming for selectively latching and unlatching said windows to and from one another to facilitate arrangement of said windows on said display either as a group, or individually.

10. The electronic translator of claim 1, further comprising a text-to-phonetic spelling translator in said processor for translating words in said text data stream into text of phonetic spelling of said words, and displaying said phonetic spelling text on said video display, and a phonetic spelling database interfaced to said processor containing said phonetic spelling text.

11. The electronic translator of claim 1, further comprising a text-to-audio sound translator in said processor for translating words in said text data stream into environmental sounds that are related to said words, and playing said audio sounds on an audio output device, and an audio sound database interfaced to said processor containing said environmental sounds.

12. The electronic translator of claim 1, further comprising an audio receiver for generating a speech signal in response to a person speaking, and a speech-to-text translator for converting said input speech signal into said text data stream.

13. The electronic translator of claim 12, further comprising a text-to-speech translator in said processor for translating said text data stream back into a second speech signal, and at least one audio output device interfaced to said processor for receiving said second speech signal and generating a speech sound simultaneously with the display of said video stream on said video display.

14. The electronic translator of claim 13, wherein said processor further includes programming for displaying text corresponding to said text data stream simultaneously with the generation of said speech sound and display of said video stream on said video display.

15. The electronic translator of claim 14, wherein said processor further includes programming for highlighting words in said text being displayed on said video display as said highlighted words are being signed by said text-to-sign language translator, and generated by said audio output device.

16. The electronic translator of claim 15, wherein said audio output device comprises a cochlear implant, hearing aid or other hearing enhancement device whereby, a hearing impaired person can identify speech sounds generated by said cochlear implant, hearing aid or other hearing enhancement device as corresponding to words that are signed in said video stream and highlighted in said text on said video display.

17. The electronic translator of claim 12, wherein said audio output device comprises a cochlear implant, hearing aid or other hearing enhancement device, whereby, a hearing impaired person can identify speech sounds generated by said cochlear implant, hearing aid or other hearing enhancement device as corresponding to words that are signed in said video stream.

18. The electronic translator of claim 12, wherein said processor further includes programming for establishing a telephone connection with a telephone, transmitting said second speech signal over a transmitter in said telephone, and receiving a third speech signal from a receiver in said telephone for translation by said speech-to-text translator and display on said video display.

19. The electronic translator of claim 1, wherein said processor and said display are incorporated in a laptop computer.

20. The electronic translator of claim 1, further comprising a keyboard interfaced to said processor for entering said text data stream, and wherein said processor is further programmed to display text corresponding to said text data stream on said display simultaneously with said video stream.

21. The electronic translator of claim 20, wherein said at least one audio output device comprises a cochlear implant, hearing aid or other hearing enhancement device and one or more audio speakers, and said translator further comprises a sound card interfaced to said processor, said sound card having a first output channel interfaced to said cochlear implant, hearing aid or other hearing enhancement device and a second output channel interfaced to said audio speaker, and, wherein, said processor is programmed to switch on said second channel only if said text data stream translated by said text-to-speech translator has been generated by said keyboard to avoid generation of feedback between said audio speaker and said audio receiver when said text data stream is generated by said speech-to-text translator.

22. The electronic translator of claim 1, further comprising:

a keyboard interfaced to said processor for entering a text data stream to be translated by said text-to-sign language translator;

an audio receiver for detecting a person'"'"'s speech and generating a first speech signal in response thereto;

a speech-to-text translator for converting said first speech signal into a text data stream to be translated by said text-to-sign language translator;

a text-to-speech translator in said processor for translating said text data stream into a second speech signal;

at least one audio output device interfaced to said processor for receiving said second speech signal and generating a speech sound simultaneously with the display of said video stream on said video display; and

wherein each of said video clips begins and ends with said person signing said words in a selected position to facilitate seamless sequencing of said clips in said video stream.

23. The electronic translator of claim 22, further including at least one additional translator in said processor and associated database interfaced to said processor selected from the group comprising a text-to-image translator for translating said text into one or more images contained in an image database that are associated with words in said text data stream, and displaying said images simultaneously with said words on said video display;

a text-to-phonetic spelling translator for translating words in said text data stream into text of phonetic spellings of said words contained in a phonetic spelling database, and displaying said phonetic spelling text on said video display; and

, a text-to-audio sound translator for translating words in said text data stream into environmental sounds that are related to said words and are contained in an environmental sounds database, and playing said audio sounds on said audio output device.

24. The electronic translator of claim 22, wherein said audio output device comprises a cochlear implant, hearing aid or other hearing enhancement device, whereby, a hearing impaired person can identify speech sounds generated by said cochlear implant, hearing aid or other hearing enhancement device as corresponding to words that are signed in said video stream.

25. The electronic translator of claim 22, wherein said database further includes a plurality of finger spelling video movies for signing individual letters of the alphabet, and said text-to-sign language translator further includes programming for determining that a word in said text data stream does not have a corresponding signing video in said database, and accessing said finger spelling videos to display sign language spelling of said word.

26. The electronic translator of claim 22, wherein said text-to-sign language translator includes a speed control for adjusting a play speed of said video clips to enable synchronization of said video clips with said text data stream as they are simultaneously displayed on said video display.

27. The electronic translator of claim 22, wherein said processor further includes programming for simultaneously displaying text, corresponding to said text data stream, on said display in a first, scrolling marquee format and in a second, static format.

28. The electronic translator of claim 22, wherein said processor further includes programming for displaying a plurality of information windows on said video display, including at least a first window for displaying said video stream and a second window for displaying text corresponding to said text data stream.

29. The electronic translator of claim 22, wherein said at least one audio output device comprises a cochlear implant, hearing aid or other hearing enhancement device and one or more audio speakers, and said translator further comprises a sound card interfaced to said processor, said sound card having a first output channel interfaced to said cochlear implant, hearing aid or other hearing enhancement device and a second output channel interfaced to said audio speaker, and, wherein, said processor is programmed to switch on said second channel only if said text data stream translated by said text-to-speech translator has been generated by said keyboard to avoid generation of feedback between said audio speaker and said audio receiver when said text data stream is generated by said speech-to-text translator.

30. The electronic translator of claim 22, wherein said processor further includes programming for displaying text corresponding to said text data stream simultaneously with the generation of said speech sound and display of said video stream on said video display.

31. The electronic translator of claim 22, wherein said processor further includes programming for highlighting words in said text being displayed on said video display as said highlighted words are being signed by said text-to-sign language translator, and generated by said audio output device.

32. The electronic translator of claim 31, wherein said audio output device comprises a cochlear implant, hearing aid or other hearing enhancement device, whereby, a hearing impaired person can identify speech sounds generated by said cochlear implant, hearing aid or other hearing enhancement device as corresponding to words that are signed in said video stream and highlighted in said text on said video display.

33. The electronic translator of claim 22, wherein said processor further includes programming for establishing a telephone connection with a telephone, transmitting said second speech signal over a transmitter in said telephone, and receiving a third speech signal from a receiver in said telephone for translation by said speech-to-text translator and display on said video display.

34. The electronic translator of claim 22, wherein said processor and said display are incorporated in a laptop computer.

35. An electronic translator for translating speech into multiple forms in real time comprising:

an audio receiver for detecting a person'"'"'s speech and generating a first speech signal in response thereto;

a keyboard for entering a first text data stream to be translated;

a processor, said processor including;

a speech-to-text translator for converting said first speech signal into a second text data stream; and

a text-to-speech translator for selectively converting said first or said second text data streams into a second speech signal;

a video display interfaced to said processor for displaying text that corresponds to said first and second text data streams; and

an audio output device interfaced to said processor for receiving said second speech signal and generating a speech sound simultaneously with the display of said text.

36. The electronic translator of claim 35, further including at least one additional translator in said processor selected from the group comprising a text-to-image translator for translating said text data streams into one or more images that are associated with words in said text data streams, and displaying said images simultaneously with said words on said video display;

a text-to-phonetic spelling translator for translating words in said text data streams into text of phonetic spellings of said words, and displaying said phonetic spelling text on said video display; and

, a text-to-audio sound translator for translating words in said text data streams into environmental sounds that are related to said words, and playing said audio sounds on said audio output device.

37. The electronic translator of claim 35, wherein said audio output device comprises a cochlear implant, hearing aid or other hearing enhancement device, whereby, a hearing impaired person can identify speech sounds generated by said cochlear implant, hearing aid or other hearing enhancement device as corresponding to words in said text data streams.

38. The electronic translator of claim 35, further including a text-to-sign language translator in said processor for converting said text data streams into a video stream of a person using sign language to sign words in said data streams, and a database interfaced to said processor and containing a plurality of video clips of a person signing words, each of said video clips showing the signing at least one word;

wherein, said text-to-sign language translator further includes programming responsive to said text data stream for detecting words therein, sequentially retrieving video clips in said database showing signing of said words, and playing said video clips sequentially on said video display as a continuous video stream.

39. The electronic translator of claim 38, wherein said database further includes a plurality of finger spelling video movies for signing individual letters of the alphabet, and said text-to-sign language translator further includes programming for determining whether a word in said text data stream has a corresponding signing video clip in said database, and if not, accessing said finger spelling videos to display sign language spelling of said word.

40. The electronic translator of claim 38, wherein said text-to-sign language translator includes a speed control for adjusting a play speed of said video clips to enable synchronization of said video clips with said text data stream as they are simultaneously displayed on said video display.

41. The electronic translator of claim 38, wherein each of said video clips begin and end with a person in a selected position to facilitate seamless sequencing of said clips in said video stream.

42. The electronic translator of claim 38, wherein said processor further includes programming for highlighting words in said text being displayed on said video display as said highlighted words are being signed by said text-to-sign language translator, and generated by said audio output device.

43. The electronic translator of claim 42, wherein said audio output device comprises a cochlear implant, hearing aid or other hearing enhancement device, whereby, a hearing impaired person can identify speech sounds generated by said cochlear implant, hearing aid or other hearing enhancement device as corresponding to words that are signed in said video stream and highlighted in said text on said video display.

44. The electronic translator of claim 35, wherein said processor further includes programming for simultaneously displaying text, corresponding to said text data streams, on said display in a first, scrolling marquee format and in a second, static format.

45. The electronic translator of claim 35, wherein said at least one audio output device comprises a cochlear implant, hearing aid or other hearing enhancement device and one or more audio speakers, and said translator further comprises a sound card interfaced to said processor, said sound card having a first output channel interfaced to said cochlear implant, hearing aid or other hearing enhancement device and a second output channel interfaced to said audio speaker, and, wherein, said processor is programmed to switch on said second channel only if said text data stream translated by said text-to-speech translator has been generated by said keyboard to avoid generation of feedback between said audio speaker and said audio receiver when said text data stream is generated by said speech-to-text translator.

46. The electronic translator of claim 35, wherein said processor further includes programming for establishing a telephone connection with a telephone, transmitting said second speech signal over a transmitter in said telephone, and receiving a third speech signal from a receiver in said telephone for translation by said speech-to-text translator and display on said video display.

47. The electronic translator of claim 35, wherein said processor and said display are incorporated in a laptop computer.

48. The electronic translator of claim 35, wherein said processor further includes programming for displaying a plurality of information windows on said video display, including at least a first window for displaying translated text and a second window for displaying control functions.

49. A method for translating a text data stream in real time comprising the steps of:

generating a text data stream;

translating said text data stream into a video stream of a person using sign language to sign words in said data stream by accessing a database containing a plurality of video clips of a person signing words, each of said video clips showing the signing for at least one particular word, said database also containing a plurality of finger spelling video clips showing a person signing individual letters of the alphabet;

sequentially retrieving video clips in said database showing signing of said words in said text data stream for any of said words that are determined to have a corresponding signing video in said database, and/or finger spelling video clips corresponding to the spelling of any of said words in said text data stream that are determined not to have a corresponding video in said database; and

, combining said sequentially retrieved video clips to form said video stream;

displaying text that corresponds to said text data stream on a video display; and

displaying said video stream simultaneously with said text on said display.

50. The method of claim 49, further comprising the steps of translating said text data stream into a speech signal;

and, driving an audio output device with said speech signal simultaneously with the display of said video stream and said text.

51. The method of claim 50, wherein the step of driving an audio output device with said speech signal further comprises driving a cochlear implant, hearing aid or other hearing enhancement device with said speech signal.

52. The method of claim 51, further including the step of highlighting words in text being displayed on said video display as said highlighted words are being generated by said cochlear implant, hearing aid or other hearing enhancement device so that a hearing impaired person using said cochlear implant, hearing aid or other hearing enhancement device can identify a sound generated by the cochlear implant, hearing aid or other hearing enhancement device as corresponding to said highlighted words.

53. The method of claim 49, further comprising the step of adjusting a playback speed of said video clips to synchronize of said video clips with said text as said video stream and said text are simultaneously displayed on said video display.

54. The method of claim 49, wherein each of said video clips is selected to begin and end with a person in a selected position to facilitate seamless sequencing of said clips in said video stream.

55. The method of claim 49, further including the step of highlighting words in said text being displayed on said video display as said highlighted words are being signed in said video stream.

56. The method of claim 49, further comprising one or more of the steps selected from the group comprising:

1) displaying a phonetic spelling representation of a word as it is signed in said video stream;

2) playing an audio sound that is related to a word as it is signed in said video stream; and

3) displaying one or more images that are associated with said text as said text is displayed.

57. The method of claim 49, wherein the step of generating a text data stream further comprises entering said text data stream with a keyboard.

58. The method of claim 49, wherein the step of generating a text data stream further comprises generating a speech signal in response to a person speaking, and translating said speech signal into said text data stream.

59. A method for translating speech into multiple formats in real time comprising the steps of:

generating a first speech signal in response to a person speaking;

translating said speech signal into a text data stream;

translating said text data stream into a second speech signal;

driving an audio output device with said second speech signal; and

simultaneously displaying text that corresponds to said text data stream on a video display.

60. The method of claim 59, wherein the step of driving an audio output device with said second speech signal further comprises driving a cochlear implant, hearing aid or other hearing enhancement device with said second speech signal.

61. The method of claim 60, further including the step of highlighting words in text being displayed on said video display as said highlighted words are being generated by said cochlear implant, hearing aid or other hearing enhancement device so that a hearing impaired person wearing said cochlear implant can identify a sound generated by the cochlear implant, hearing aid or other hearing enhancement device as corresponding to said highlighted words.

62. The method of claim 59, further comprising the steps of:

translating said text data stream into a video stream of a person using sign language to sign words in said data stream by accessing a database containing a plurality of video clips of a person signing words, each of said video clips showing the signing for at least one particular word; and

sequentially retrieving video clips in said database showing signing of said words in said text data stream to form said video stream; and

displaying said video stream simultaneously with said text on said display.

63. The method of claim 62, wherein said database further includes a plurality of finger spelling video movies for signing individual letters of the alphabet, and said method further comprise the steps of determining whether a word in said text data stream has a corresponding signing video in said database, if not, sequentially retrieving finger spelling videos corresponding to the spelling of said word.

64. The method of claim 62, further comprising the step of adjusting a playback speed of said video clips to synchronize of said video clips with said text as said video stream and said text are simultaneously displayed on said video display.

65. The method of claim 62, wherein each of said video clips is selected to begin and end with a person in a selected position to facilitate seamless sequencing of said clips in said video stream.

66. The method of claim 62, further including the step of highlighting words in said text being displayed on said video display as said highlighted words are being signed in said video stream.

67. The method of claim 59, further comprising one or more of the steps selected from the group comprising:

1) displaying a phonetic spelling representation of a word as it is signed in said video stream;

2) playing an audio sound that is related to a word as it is signed in said video stream; and

3) displaying one or more images that are associated with said text as said text is displayed.

68. The method of claim 59, further comprising the steps of establishing a telephone connection with a telephone, transmitting said audio signal over a transmitter in said telephone, receiving a second speech input signal from a receiver in said telephone, translating said second speech signal into a second text data stream, and displaying said second text data stream on said video display.