Enhanced speech-to-speech translation system and methods for adding a new word

• US 8,972,268 B2
• Filed: 01/18/2011
• Issued: 03/03/2015
• Est. Priority Date: 04/15/2008
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Waibel, Alexander, Lane, Ian R.
• Primary Examiner(s): ROBERTS, SHAUN A
• Application Number:
  US13/008,346
• Publication Number:
  US 20110307241A1
• Time in Patent Office: 1,505 Days
• Field of Search: 704/2, 704/277
• US Class Current: 704/277
• CPC Class Codes: G06F 40/44 Statistical methods, e.g. probability models
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 15/06 Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice G10L15/14 takes precedence
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/0631 Creating reference templates; Clustering

Abstract

A speech translation system and methods for cross-lingual communication that enable users to improve and modify content and usage of the system and easily abort or reset translation. The system includes a speech recognition module configured for accepting an utterance and adding a new word, a machine translation module, an interface configured to communicate the utterance and proposed translation, a correction module and an abort action unit that removes any hypotheses or partial hypotheses and terminates translation. The system also includes modules for storing favorites, changing language mode, automatically identifying language, providing language drills, viewing third party information relevant to conversation, among other things.

Claims

1. A method comprising:

receiving from a user, an utterance in a first language that is to be translated by a speech translation system from the first language to a second language;

receiving, an indication to add a new word in the first language to a first recognition lexicon of the first language of a first automatic speech recognition module of the speech translation system;

determining for the new word, by a processor, word class information, a pronunciation in the first language, a translation in the second language, and a pronunciation in the second language in response to receiving the indication to add the new word;

adding the new word the determined word class information and the determined pronunciation in the first language to the first recognition lexicon of the first language of the first automatic speech recognition module; and

adding the new word, the determined word class information, the determined translation in the second language and the pronunciation of the translation in the second language, to a first machine translation module associated with the first language of the speech translation system.

2. The method of claim 1, wherein determining word class information comprises estimating the word class information via a tagging model.

3. The method of claim 1, wherein determining a pronunciation in the first language comprises generating a pronunciation in the first language via either a rule-based model or a statistical model.

4. The method of claim 1, wherein determining a translation in the second language comprises generating a translation in the second language via either a rule-based model or a statistical model.

5. The method of claim 1, wherein the user is prompted to verify the word class information, pronunciation in the first language, and translation in the second language before the new word is added to the first recognition lexicon of the first language of the automatic speech recognition module.

6. The method of claim 1, wherein the user is prompted to verify the word class information, pronunciation in the first language, and translation in the second language before the new word is added to the first machine translation module associated with the first language of the speech translation system.

7. The method of claim 1, further comprising:

displaying simultaneously in text, on a user interface display of the speech translation system, at least (i) recognized speech in the utterance in the first language, and (ii) the translation into the second language of the speech in the utterance;

storing, by the speech translation system, a bilingual sentence-pair selected by the user via the user interface display, as a speech translation favorite, wherein the bilingual sentence pair comprises a sentence in the first language uttered by the user in a first dialog and a translation of the sentence from the first language into the second language; and

playing the translation of the sentence into the second language upon selection by the user to play the translation of the sentence into the second language in a second dialog that is after the first dialog, without the user having to speak the sentence in the second dialog.

8. A device-comprising:

at least one microphone for receiving an utterance in a first language from a user; and

a speech translation system in communication with the at least one microphone, wherein the speech translation system is for translating the utterance into a second language, and wherein the speech translation system comprises;

a first automatic speech recognition module for the first language for recognizing speech in the utterance in the first language;

a first machine translation module in communication with the first speech recognition module, wherein the first machine translation module is for translating the recognized speech in the first language, recognized by the first speech recognition module, into the second language; and

;

a user interface in communication with the speech translation system for outputting the translation of the utterance in the second language determined by the first machine translation module;

wherein the speech translation system is configured to;

receive from the user, the utterance in the first language that is to be translated by the speech translation system from the first language to the second language;

receive an indication to add a new word in the first language to the first recognition lexicon of the first automatic speech recognition module of the speech translation system;

determine for the new word, by a processor, word class information, a pronunciation in the first language, the translation in the second language, and a pronunciation in the second language;

add the new word, the determined word class information and the determined pronunciation in the first language to the first recognition lexicon of the first language of the first automatic speech recognition module; and

add the new word, the determined word class information, the determined translation in the second language and the pronunciation of the translation in the second language, to the first machine translation module.

9. The device of claim 8, wherein determining word class information Comprises estimating the word class information via a tagging model.

10. The device of claim 8, wherein determining a pronunciation in the first language comprises generating a pronunciation in the first language via either a rule-based model Or a statistical model.

11. The device of claim 8, wherein determining a translation in the second Language comprises generating a translation in the second language via either a rule-based model Or a statistical model.

12. The device of claim 8, wherein the user is prompted to verify the word class information, pronunciation in the first language, and translation in the second language before the new word is added to the first recognition lexicon of the first language of the automatic speech recognition module.

13. The device of claim 8, wherein the user is prompted to verify the word class information, pronunciation in the first language, and translation in the second language before the new word is added to the first machine translation module associated with the first language of the speech translation system.

14. The device of claim 8, further comprising a text processor configured to identify words in the utterance as being potentially inappropriate and replacing the inappropriate words with a sound.

15. The device of claim 8, further comprising a conditioning mode configured to use the first translation module and first language module in combination with prosodic parameters to render the pronunciation of the translation more appropriate based on language use.

16. The device of claim 8, wherein the speech translation system:

further comprises;

a second automatic speech recognition module for recognizing speech in the second language; and

a second machine translation module for translating recognized speech in the second language to the first language; and

is further configured to identify whether the utterance is in the first language or the second language.

17. The device of claim 16, wherein the speech translation system is for:

automatically using the first automatic speech recognition module when it is identified by the speech translation system that an input utterance is in the first language; and

automatically using the second automatic speech recognition module when it is identified by the speech translation system that the input utterance is in the second language, without having the user input to the device who is speaking in a dialog between two speakers.

18. The device of claim 16, wherein the speech translation system further comprises an information extraction module for:

generating usage statistics from system logs generated by the first and second automatic speech recognition modules, and the first and second machine translation modules;

retrieving relevant information based on keyword occurrences in the usage statistics; and

presenting the relevant information to the user via the user interface display.

19. The device of claim 8, wherein the device comprises a portable device.

20. The device of claim 8, wherein the user interface comprises a speaker.

21. The device of claim 20, wherein the user interface further comprises a display for displaying text of the translated utterance in the second language.

22. The device of claim 8, further comprising speech translation favorites module for:

storing a bilingual sentence-pair selected by the use via the user interface as a speech translation favorite of the user, wherein the bilingual sentence pair comprises a sentence in the first language uttered by the user in a first dialog and a translation of the sentence in the first language into the second language; and

playing the translation of the sentence into the second language upon selection by the user to play the translation of the sentence into the second language in a second dialog that is after the first dialog, without the user having to speak the sentence in the second dialog.

23. The device of claim 22, wherein the user selects to play the translation of the sentence into the second language in the second dialog by clicking on an identifier for the bilingual sentence-pair via the user interface.

24. The device of claim 8, wherein the speech translation system further comprises a language learning module that is for:

building a profile of language usage statistics for the user based on usage by the user of the speech translation user device over a period of time, wherein the usage comprises utterances by the user in the first language; and

in a language learning mode, constructs a vocabulary drill for the user in the second language based on the profile of language usage statistics for the user.

25. The device of claim 24, wherein the language learning module is further for, in the language learning mode, generate a syntax drill for the user in the second language based on the profile of language usage statistics for the user.