TRANSLATION TRAINING WITH CROSS-LINGUAL MULTI-MEDIA SUPPORT

• US 20150154185A1
• Filed: 01/05/2015
• Published: 06/04/2015
• Est. Priority Date: 06/11/2013
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Waibel, Alexander
• Granted Patent: US 9,892,115 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 40/166 Editing, e.g. inserting or deleting
• CPC Class Codes: G06F 40/20 Natural language analysis semantic analysis of natural language G06F40/30
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06F 40/44 Statistical methods, e.g. probability models
• CPC Class Codes: G06F 40/45 Example-based machine translation; Alignment
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G10L 15/005 Language recognition

Abstract

An improved lecture support system integrates multi-media presentation materials with spoken content so that the listener can follow with both the speech and the supporting materials that accompany the presentation to provide additional understanding. Computer-based systems and methods are disclosed for translation of a spoken presentation (e.g., a lecture, a video) along with the accompanying presentation materials. The content of the presentation materials can be used to improve presentation translation, as it extracts supportive material from the presentation materials as they relate to the speech.

Claims

1. A method comprising:

recognizing, by an automatic speech recognition module of a translation system, speech by a speaker in a first language, the automatic speech recognition module comprising a language model;

transcribing the recognized speech in the first language;

translating, by a machine translation module of the translation system, the transcribed speech into a second language, the machine translation module comprising a language model;

receiving presentation materials associated with the speech in the first language;

extracting a portion of text in the first language from the presentation materials;

translating, by the machine translation module, the extracted text into the second language;

generating translated presentation materials in the second language based on the text in the second language; and

modifying, based on the extracted text, at least one selected from the group consisting of the automatic speech recognition language model, the automatic speech recognition vocabulary, the machine translation language model, and the machine translation model.

2. The method of claim 1, wherein modifying the automatic speech recognition language model comprises:

identifying a first unknown word in the extracted original text;

generating a pronunciation for the first unknown word; and

modifying an automatic speech recognition language model probability associated with the first unknown word.

3. The method of claim 2, wherein modifying the automatic speech recognition language model further comprises:

receiving, based on an internet search, materials related to the extracted original text;

identifying a second unknown word in the related materials;

generating a pronunciation for the second unknown word; and

modifying an automatic speech recognition language model probability associated with the second unknown word.

4. The method of claim 1, wherein modifying the machine translation language model comprises:

identifying a third unknown word in the extracted original text;

receiving, based on an internet search, a translation of the third unknown word; and

adding the translation to the machine translation language model.

5. The method of claim 1, wherein the original presentation materials comprise slides, the method further comprising generating a time-stamp associated with a transition from a first slide to a second slide.

6. The method of claim 5, further comprising determining a paragraph break in the transcription based on the time-stamp.

7. The method of claim 5, further comprising inserting a punctuation mark in the transcription based on the time-stamp.

8. The method of claim 1, further comprising:

identifying a mathematical formula in the translated speech; and

generating an associated transcription using mathematical notation.

9. The method of claim 1, further comprising:

identifying a first element in the transcription of the translated speech;

identifying a second element in the translated presentation materials, the second element corresponding to the first element; and

generating a hyperlink between the first element and the second element.

10. The method of claim 1, wherein the speaker is a user of the translation system.

11. A method comprising:

recognizing, by an automatic speech recognition module of a translation system, speech by a speaker in a first language;

transcribing, by a transcription module of the translation system, the recognized speech in the first language;

receiving presentation materials associated with the speech in the first language;

extracting a portion of text in the first language from the presentation materials;

modifying, based on the extracted text, at least one selected from the group consisting of the automatic speech recognition language model and the automatic speech recognition vocabulary.

12. A computer program product for translating a multimedia translation, the computer program product comprising a computer-readable storage medium containing computer program code for:

recognizing, by an automatic speech recognition module of a translation system, speech by a speaker in a first language, the automatic speech recognition module comprising a language model;

transcribing the recognized speech in the first language;

translating, by a machine translation module of the translation system, the transcribed speech into a second language, the machine translation module comprising a language model;

receiving presentation materials associated with the speech in the first language;

extracting a portion of text in the first language from the presentation materials;

translating, by the machine translation module, the extracted text into the second language;

generating translated presentation materials in the second language based on the text in the second language; and

modifying, based on the extracted text, at least one selected from the group consisting of the automatic speech recognition language model, the automatic speech recognition vocabulary, the machine translation language model, and the machine translation model.

13. The computer program product of claim 12, wherein modifying the automatic speech recognition language model comprises:

identifying a first unknown word in the extracted original text;

generating a pronunciation for the first unknown word; and

modifying an automatic speech recognition language model probability associated with the first unknown word.

14. The computer program product of claim 13, wherein modifying the automatic speech recognition language model further comprises:

receiving, based on an internet search, materials related to the extracted original text;

identifying a second unknown word in the related materials;

generating a pronunciation for the second unknown word; and

modifying an automatic speech recognition language model probability associated with the second unknown word.

15. The computer program product of claim 12, wherein modifying the machine translation language model comprises:

identifying a third unknown word in the extracted original text;

receiving, based on an internet search, a translation of the third unknown word; and

adding the translation to the machine translation language model.

16. The computer program product of claim 12, wherein the original presentation materials comprise slides, the method further comprising generating a time-stamp associated with a transition from a first slide to a second slide.

17. The computer program product of claim 16, further comprising determining a paragraph break in the transcription based on the time-stamp.

18. The computer program product of claim 16, further comprising inserting a punctuation mark in the transcription based on the time-stamp.

19. The computer program product of claim 12, further comprising:

identifying a mathematical formula in the translated speech; and

generating an associated transcription using mathematical notation.

20. The computer program product of claim 12, further comprising:

identifying a first element in the transcription of the translated speech;

identifying a second element in the translated presentation materials, the second element corresponding to the first element; and

generating a hyperlink between the first element and the second element.