Translation training with cross-lingual multi-media support

• US 9,892,115 B2
• Filed: 01/05/2015
• Issued: 02/13/2018
• Est. Priority Date: 06/11/2013
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Waibel, Alexander
• Primary Examiner(s): Tzeng, Forrest F
• Application Number:
  US14/589,658
• Publication Number:
  US 20150154185A1
• Time in Patent Office: 1,135 Days
• Field of Search: 704 2, 704 3, 704 4, 704 5, 704235, 704246, 704270, 704275, 704277
• CPC Class Codes: G06F 40/166 Editing, e.g. inserting or deleting
• CPC Class Codes: G06F 40/20 Natural language analysis semantic analysis of natural language G06F40/30
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06F 40/44 Statistical methods, e.g. probability models
• CPC Class Codes: G06F 40/45 Example-based machine translation; Alignment
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G10L 15/005 Language recognition

Abstract

An improved lecture support system integrates multi-media presentation materials with spoken content so that the listener can follow with both the speech and the supporting materials that accompany the presentation to provide additional understanding. Computer-based systems and methods are disclosed for translation of a spoken presentation (e.g., a lecture, a video) along with the accompanying presentation materials. The content of the presentation materials can be used to improve presentation translation, as it extracts supportive material from the presentation materials as they relate to the speech.

Claims

1. A method comprising:

receiving, by an interface of a translation system, speech by a speaker in a first language;

recognizing, by an automatic speech recognition module of the translation system, the received speech;

transcribing the recognized speech in the first language;

translating, by a machine translation module of the translation system, the speech transcription into a second language, the translated speech transcription stored in a file of a database of the translation system;

receiving, by the interface of the translation system, presentation materials associated with the speech in the first language;

extracting text in the first language from the presentation materials;

translating, by the machine translation module, the extracted text into the second language;

generating translated presentation materials in the second language based on the text in the second language;

modifying the file storing the translated speech transcription by importing at least one formatting feature from the original or translated presentation materials into the translated speech transcription; and

displaying, by a display device, the modified translated speech transcription.

2. The method of claim 1, further comprising modifying an automatic speech recognition language model of the translation system by:

identifying a first unknown word in the extracted text;

generating a pronunciation for the first unknown word; and

modifying an automatic speech recognition language model probability associated with the first unknown word.

3. The method of claim 2, wherein modifying the automatic speech recognition language model further comprises:

receiving, based on an internet search, materials related to the extracted text;

identifying a second unknown word in the related materials;

generating a pronunciation for the second unknown word; and

modifying an automatic speech recognition language model probability associated with the second unknown word.

4. The method of claim 1, further comprising modifying a machine translation language model of the translation system by:

identifying a third unknown word in the extracted original text;

receiving, based on an internet search, a translation of the third unknown word; and

adding the translation to the machine translation language model.

5. The method of claim 1, wherein the original presentation materials comprise slides, the method further comprising generating a time-stamp associated with a transition from a first slide to a second slide.

6. The method of claim 5, wherein the modification comprises determining a paragraph break in the translated speech based on the time-stamp.

7. The method of claim 5, wherein the modification comprises inserting a punctuation mark in the transcription based on the time-stamp.

8. The method of claim 1, wherein the modification comprises:

identifying a mathematical formula in the translated speech; and

generating an associated transcription using mathematical notation.

9. The method of claim 1, wherein the modification comprises:

identifying a first element in the transcription of the translated speech;

identifying a second element in the translated presentation materials, the second element corresponding to the first element; and

generating a hyperlink between the first element and the second element.

10. The method of claim 1, wherein the speaker is a user of the translation system.

11. A computer program product for translating a multimedia presentation, the computer program product comprising a non-transitory computer-readable storage medium containing computer program code for:

receiving, by an interface of a translation system, speech by a speaker in a first language;

recognizing, by an automatic speech recognition module of the translation system, the received speech

transcribing the recognized speech in the first language;

translating, by a machine translation module of the translation system, the speech transcription into a second language, the translated speech transcription stored in a file of a database of the translation system;

receiving presentation materials associated with the speech in the first language;

extracting text in the first language from the presentation materials;

translating, by the machine translation module, the extracted text into the second language;

generating translated presentation materials in the second language based on the text in the second language;

modifying the file storing the translated speech transcription by importing at least one formatting feature from the original or translated presentation materials into the translated speech transcription; and

displaying, by a display device, the modified translated speech transcription.

12. The computer program product of claim 11, further comprising modifying an automatic speech recognition language model of the translation system by:

identifying a first unknown word in the extracted original text;

generating a pronunciation for the first unknown word; and

modifying an automatic speech recognition language model probability associated with the first unknown word.

13. The computer program product of claim 12, wherein modifying the automatic speech recognition language model further comprises:

receiving, based on an internet search, materials related to the extracted original text;

identifying a second unknown word in the related materials;

generating a pronunciation for the second unknown word; and

modifying an automatic speech recognition language model probability associated with the second unknown word.

14. The computer program product of claim 11, further comprising modifying a machine translation language model of the translation system by:

identifying a third unknown word in the extracted original text;

receiving, based on an internet search, a translation of the third unknown word; and

adding the translation to the machine translation language model.

15. The computer program product of claim 11, wherein the original presentation materials comprise slides, the method further comprising generating a time-stamp associated with a transition from a first slide to a second slide.

16. The computer program product of claim 15, wherein the modification comprises determining a paragraph break in the transcription based on the time-stamp.

17. The computer program product of claim 15, wherein the modification comprises inserting a punctuation mark in the transcription based on the time-stamp.

18. The computer program product of claim 11, wherein the modification comprises:

identifying a mathematical formula in the translated speech; and

generating an associated transcription using mathematical notation.

19. The computer program product of claim 11, wherein the modification comprises:

identifying a first element in the transcription of the translated speech;

identifying a second element in the translated presentation materials, the second element corresponding to the first element; and

generating a hyperlink between the first element and the second element.

20. The computer program product of claim 11, wherein the speaker is a user of the translation system.