Multimodal speech-to-speech language translation and display

• US 20040111272A1
• Filed: 12/10/2002
• Published: 06/10/2004
• Est. Priority Date: 12/10/2002
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Sorensen, Jeffrey, Gu, Liang, Gao, Yuqing, Liu, Fu-Hua
• Application Number:
  US10/315,732
• Publication Number:
  US 20040111272A1
• Time in Patent Office: Days
• US Class Current: 704/277
• CPC Class Codes: G06F 40/55 Rule-based translation
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation

Abstract

A multimodal speech-to-speech language translation system and method for translating a natural language sentence of a source language into a symbolic representation and/or target language is provided. The system includes an input device for inputting a natural language sentence of a source language into the system; a translator for receiving the natural language sentence in machine-readable form and translating the natural language sentence into a symbolic representation and/or a target language; and an image display for displaying the symbolic representation of the natural language sentence. Additionally, the image display indicates a correlation between text of the target language, the symbolic representation and the text of the source language.

Claims

1. A language translation system comprising:

an input device for inputting a natural language sentence of a source language into the system;

a translator for receiving the natural language sentence in machine-readable form and translating the natural language sentence into a symbolic representation; and

an image display for displaying the symbolic representation of the natural language sentence.

2. The system as in claim 1, further comprising a text-to-speech synthesizer for audibly producing the natural language sentence in a target language.

3. The system as in claim 1, wherein the input device is an automatic speech recognizer for converting spoken words into machine recognizable text.

4. The system as in claim 1, wherein the translator further comprises

a natural language understanding parser for parsing structural information from the natural language sentence and outputting a semantic parse tree representation of the natural language sentence.

5. The system as in claim 1, wherein the translator further comprises

a natural language understanding statistical classer for classifying elements of the natural language sentence and tagging the elements by category;

and

a natural language understanding parser for parsing structural information from the classed sentence and outputting a semantic parse tree representation of the classed sentence.

6. The system as in claim 5, wherein the translator further comprises an interlingua information extractor for extracting a language independent representation of the natural language sentence.

7. The system as in claim 6, wherein the translator further comprises a symbolic image generator for generating the symbolic representation of the natural language sentence by associating elements of the language independent representation to visual depictions.

8. The system as in claim 6, wherein the translator further comprises a natural language generator for converting the language independent representation into a target language.

9. The system as in claim 1, wherein the translator translates the natural language sentence into text of a target language and the image display displays the text of the target language along with the symbolic representation.

10. The system as in claim 3, wherein the translator translates the natural language sentence into text of a target language and the image display displays the text of the target language, the symbolic representation and the text of the source language.

11. The system as in claim 10, wherein the image display indicates a correlation between the text of the target language, the symbolic representation and the text of the source language.

12. A method for translating a language, the method comprising the steps of:

receiving a natural language sentence of a source language;

translating the natural language sentence into a symbolic representation; and

displaying the symbolic representation of the natural language sentence.

13. The method as in claim 12, wherein the receiving step includes the steps of:

receiving a spoken natural language sentence as acoustic signals; and

converting the spoken natural language sentence into machine recognizable text.

14. The method as in claim 13, further comprising the steps of:

parsing structural information from the natural language sentence and outputting a semantic parse tree representation of the natural language sentence.

15. The method as in claim 16, further comprising the step of extracting a language independent representation of the natural language sentence from the semantic parse tree.

16. The method as in claim 13, further comprising the steps of:

classifying elements of the natural language sentence and tagging the elements by category; and

parsing structural information from the classed sentence and outputting a semantic parse tree representation of the classed sentence.

17. The method as in claim 16, further comprising the step of extracting a language independent representation of the natural language sentence from the semantic parse tree.

18. The method as in claim 17, further comprising the step of generating the symbolic representation of the natural language sentence by associating elements of the language independent representation to visual depictions.

19. The method as in claim 18, further comprising the steps of converting the language independent representation into text of a target language and displaying the text of the target language along with the symbolic representation.

20. The method as in claim 19, further comprising the step of audibly producing the text of the target language.

21. The method as in claim 20, further comprising the step of highlighting elements of the displayed symbolic representation corresponding to the audible text of the target language.

22. The method as in claim 19, further comprising the steps of correlating the text of the target language, the symbolic representation and the text of the source language and displaying the correlation with the text of the target language, the symbolic representation and the text of the source language.

23. A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform the method steps for translating a language, the method steps comprising:

receiving a natural language sentence of a source language;

translating the natural language sentence into a symbolic representation; and

displaying the symbolic representation of the natural language sentence.