Translation training with cross-lingual multi-media support

• US 10,331,796 B1
• Filed: 02/12/2018
• Issued: 06/25/2019
• Est. Priority Date: 06/11/2013
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Waibel, Alexander
• Primary Examiner(s): Tzeng, Feng-Tzer
• Application Number:
  US15/894,662
• Time in Patent Office: 498 Days
• Field of Search: 704 2, 704 3, 704 4, 704 5, 704235, 704246, 704270, 704275, 704277
• CPC Class Codes: G06F 40/166 Editing, e.g. inserting or deleting
• CPC Class Codes: G06F 40/20 Natural language analysis semantic analysis of natural language G06F40/30
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06F 40/44 Statistical methods, e.g. probability models
• CPC Class Codes: G06F 40/45 Example-based machine translation; Alignment
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G10L 15/005 Language recognition

Abstract

An improved lecture support system integrates multi-media presentation materials with spoken content so that the listener can follow with both the speech and the supporting materials that accompany the presentation to provide additional understanding. Computer-based systems and methods are disclosed for translation of a spoken presentation (e.g., a lecture, a video) along with the accompanying presentation materials. The content of the presentation materials can be used to improve presentation translation, as it extracts supportive material from the presentation materials as they relate to the speech.

Claims

1. A method comprising:

recognizing speech of a presentation;

generating a transcription based on the recognized speech;

receiving original presentation materials presented with the speech;

extracting at least one formatting feature from the original presentation materials;

modifying the transcription by importing the at least one formatting feature extracted from the original presentation materials; and

storing the modified transcription.

2. The method of claim 1, wherein extracting the at least one formatting feature comprises:

determining time-stamps recording transition times of slides in the presentation;

and wherein modifying the transcription by importing the at least one formatting feature comprises;

adding paragraph ends to the transcription based on the time-stamps.

3. The method of claim 2, wherein adding paragraph ends to the transcription is further based on breaks of the speech.

4. The method of claim 1, wherein extracting the at least one formatting feature comprises:

identifying bullet points in the original presentation materials;

and wherein modifying the transcription by importing the at least one formatting feature comprises;

aligning the recognized speech with contents associated with the bullet points; and

adding punctuations to the aligned recognized speech based on the bullet points.

5. The method of claim 1, wherein extracting the at least one formatting feature comprises:

identifying mathematical formula in the original presentation materials;

and wherein modifying the transcription by importing the at least one formatting feature comprises;

identifying the mathematical formula spoken in the recognized speech; and

converting the mathematical formula in a prose form in the transcription to the mathematical formula in a mathematical form identified in the original presentation materials.

6. The method of claim 1, wherein extracting the at least one formatting feature comprises:

identifying a table of data in the original presentation materials;

and wherein modifying the transcription by importing the at least one formatting feature comprises;

identifying a portion of transcription corresponding to one or more data in the table spoken in the recognized speech; and

inserting the table in the transcription at the portion.

7. The method of claim 1, wherein extracting the at least one formatting feature comprises:

identifying a graph in the original presentation materials;

and wherein modifying the transcription by importing the at least one formatting feature comprises;

identifying a portion of transcription corresponding to the recognized speech to which the graph is referred; and

inserting the graph in the transcription at the portion.

8. The method of claim 1, wherein extracting the at least one formatting feature comprises:

identifying a hyperlink in the original presentation materials;

and wherein modifying the transcription by importing the at least one formatting feature comprises;

identifying a portion of transcription corresponding to the recognized speech to which the hyperlink is referred; and

replacing the portion of transcription with a hyperlinked text linking to the hyperlink.

9. The method of claim 1, further comprising:

translating the transcription into another language different from a language of the recognized speech;

displaying the translated transcription.

10. The method of claim 9, wherein translating the transcription into the another language comprises:

identifying an unknown word in the recognized speech;

conducting an internet search for words in the another language based on the original presentation materials; and

determining a translation of the unknown word based on the internet search.

11. A non-transitory computer readable medium storing program code comprising instructions, when executed by one or more processors, cause the processors to:

recognize speech of a presentation;

generate a transcription based on the recognized speech;

receive original presentation materials presented with the speech;

extract at least one formatting feature from the original presentation materials;

modify the transcription by importing the at least one formatting feature extracted from the original presentation materials; and

store the modified transcription.

12. The non-transitory computer readable medium of claim 11, wherein extract the at least one formatting feature comprises:

determine time-stamps recording transition times of slides in the presentation;

and wherein modify the transcription by importing the at least one formatting feature comprises;

add paragraph ends to the transcription based on the time-stamps.

13. The non-transitory computer readable medium of claim 12, wherein add paragraph ends to the transcription is further based on breaks of the speech.

14. The non-transitory computer readable medium of claim 11, wherein extract the at least one formatting feature comprises:

identify bullet points in the original presentation materials;

and wherein modify the transcription by importing the at least one formatting feature comprises;

align the recognized speech with contents associated with the bullet points; and

add punctuations to the aligned recognized speech based on the bullet points.

15. The non-transitory computer readable medium of claim 11, wherein extract the at least one formatting feature comprises:

identify mathematical formula in the original presentation materials;

and wherein modify the transcription by importing the at least one formatting feature comprises;

identify the mathematical formula spoken in the recognized speech; and

convert the mathematical formula in a prose form in the transcription to the mathematical formula in a mathematical form identified in the original presentation materials.

16. The non-transitory computer readable medium of claim 11, wherein extract the at least one formatting feature comprises:

identify a table of data in the original presentation materials;

and wherein modify the transcription by importing the at least one formatting feature comprises;

identify a portion of transcription corresponding to one or more data in the table spoken in the recognized speech; and

insert the table in the transcription at the portion.

17. The non-transitory computer readable medium of claim 11, wherein extract the at least one formatting feature comprises:

identify a graph in the original presentation materials;

and wherein modify the transcription by importing the at least one formatting feature comprises;

identify a portion of transcription corresponding to the recognized speech to which the graph is referred; and

insert the graph in the transcription at the portion.

18. The non-transitory computer readable medium of claim 11, wherein extract the at least one formatting feature comprises:

identify a hyperlink in the original presentation materials;

and wherein modify the transcription by importing the at least one formatting feature comprises;

identify a portion of transcription corresponding to the recognized speech to which the hyperlink is referred; and

replace the portion of transcription with a hyperlinked text linking to the hyperlink.

19. The non-transitory computer readable medium of claim 11, wherein the instructions further cause the processors to:

translate the transcription into another language different from a language of the recognized speech;

display the translated transcription.

20. The non-transitory computer readable medium of claim 19, wherein translate the transcription into the another language comprises:

identify an unknown word in the recognized speech;

conduct an internet search for words in the another language based on the original presentation materials; and

determine a translation of the unknown word based on the internet search.