HYBRID, OFFLINE/ONLINE SPEECH TRANSLATION SYSTEM

• US 20140337007A1
• Filed: 06/12/2013
• Published: 11/13/2014
• Est. Priority Date: 05/13/2013
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Waibel, Alexander, Fuegen, Christian, Waibel, Naomi Aoki, Rottmann, Kay
• Granted Patent: US 9,430,465 B2
• Time in Patent Office: Days
• US Class Current: 704/3
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 13/02 Methods for producing synthetic speech; Speech synthesisers
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications

Abstract

A hybrid speech translation system whereby a wireless-enabled client computing device can, in an offline mode, translate input speech utterances from one language to another locally, and also, in an online mode when there is wireless network connectivity, have a remote computer perform the translation and transmit it back to the client computing device via the wireless network for audible outputting by client computing device. The user of the client computing device can transition between modes or the transition can be automatic based on user preferences or settings. The back-end speech translation server system can adapt the various recognition and translation models used by the client computing device in the offline mode based on analysis of user data over time, to thereby configure the client computing device with scaled-down, yet more efficient and faster, models than the back-end speech translation server system, while still be adapted for the user'"'"'s domain.

Claims

1. A speech translation system comprising:

a translation server; and

a client device that is configured for communicating with the translation server, wherein the client device comprises;

a microphone;

a processor connected to the microphone;

a memory connected to the processor that stores instructions to be executed by the processor; and

a speaker connected to the processor, wherein;

the client device is for outputting via the speaker a translation of verbally input phrases from a first language to a second language; and

the memory stores instructions such that;

the processor determines the second language for input phrases received at the client device from a user of the client device;

the processor receives from the user a translation mode setting for the client device for the translation of the verbally input phrase into the determined second language, the translation mode setting comprising one or more user-selected performance criteria indicating whether to perform translations at the client device or at the translation server;

in response to the one or more user-selected performance criteria indicating performance of a translation at the client device, when the processor executes the instructions, the processor translates the input word phrases to the second language for output to the user; and

in response to the one or more user-selected performance criteria indicating performance of a translation at the translation server;

the client device sends to the translation server information associated with the input word phrases in the first language received by the client device;

the translation server determines the translation to the second language of the input word phrases in the first language based on the data received via the wireless network from the client device; and

the translation server transmits data regarding the translation to the second language of the input word phrases in the first language to the client device such that the client device outputs the translation to the second language of the input word phrases in the first language.

2. The speech translation system of claim 1, wherein the client device has a user interface that permits a user to switch between translation at the client device or at the translation server.

5. The speech translation system of claim 1, wherein the input word phrases are input to the client device through one of:

input speech utterances captured by the microphone of the client device;

or

text input via a text input field on a user interface of the client device.

7. The speech translation system of claim 1, wherein:

the client device stores in memory a local acoustic model, a local language model, a local translation model and a local speech synthesis model for translation at the client device recognizing the speech utterances in the first language and translating the recognized speech utterances to the second language for output via the speaker of the client device;

the translation server comprises a back-end acoustic model, a back-end language model, a back-end translation model and a back-end speech synthesis model for translation at the translation server determining the translation to the second language of the speech utterances in the first language based on the data received via a wireless network from the client device;

the local acoustic model is different from the back-end acoustic model;

the local language model is different from the back-end language model;

the local translation model is different from the back-end translation model; and

the local speech synthesis model is different from the back-end speech synthesis model.

8. The speech translation system of claim 7, wherein the translation server is programmed to:

monitor over time speech utterances received by the client device for translation from the first language to the second language; and

update at least one of the local acoustic model, the local language model, the local translation model and the local speech synthesis model of the client device based on the monitoring over time of speech utterances received by the client device for translation from the first language to the second language, wherein updates to the at least one of the local acoustic model, the local language model, the local translation model and the local speech synthesis model of the client device are transmitted from the translation server to the client device via the wireless network.

10. The speech translation system of claim 7, wherein:

the client device comprises a GPS system for determining a location of the client device; and

the translation server is programmed to update at least one of the local acoustic model, the local language model, the local translation model and the local speech synthesis model of the client device based on the location of the client device, wherein updates to the at least one of local acoustic model, the local language model, the local translation model and the local speech synthesis model of the client device are transmitted from the back-end speech translation server system to the client device via the wireless network.

11. The speech translation system of claim 7, wherein:

the translation server is one of a plurality of translation servers, and the client device is configured for communicating with the each of the plurality of translation servers via the wireless network; and

in the second operating mode;

each of the plurality of translation servers is for determining a translation to the second language of the speech utterances in the first language based on the data received via the wireless network from the client device; and

one of the plurality of translation servers systems one of the translations from the plurality of translation servers for transmitting to the client device.

12. The speech translation system of claim 7, wherein:

the translation server is one of a plurality of translation servers, and the client device is configured for communicating with the each of the plurality of translation servers via the wireless network; and

in the second operating mode;

each of the plurality of translation servers is for determining a translation to the second language of the speech utterances in the first language based on the data received via the wireless network from the client device; and

one of the plurality of translation servers merges two or more of the translations from the plurality of translation servers to generate a merged translation for transmitting to the client device.

3-4. -4. (canceled)

6. (canceled)

9. (canceled)

13. A method comprising:

receiving at a client device, from a user of the client device, a verbally input phrase in a first language;

determining a second language for translation of the verbally input phrase;

receiving from the user a translation mode setting for the client device for the translation of the verbally input phrase into the determined second language, the translation mode setting comprising one or more user-selected performance criteria indicating whether to perform the translation at the client device or at a translation server;

in response to the one or more user-selected performance criteria indicating performance of a translation at the client device;

translating by the client device the verbally input phrase from the first language into the second language; and

outputting in the second language a local translation of the verbally input phrase; and

in response to the one or more user-selected performance criteria indicating performance of a translation at the translation server;

sending from the client device to the translation server information associated with the verbally input phrase;

receiving, at the client device, data associated with a translation of the verbally input phrase from the first language to the second language by the translation server; and

outputting in the second language the server translation of the verbally input phrase.

15. The speech translation method of claim 13, further comprising downloading by the client device application software for a language translation pair that comprises the first and second languages.

17. The speech translation system of claim 15, further comprising:

determining by the client device a location of the client device; and

downloading by the client device the application software for the language translation pair based on the determined location of the client device and when suitable connectivity between the client device and the translation server is available via a wireless network.

21. The method of claim 13, wherein the one or more user-selected performance criteria comprise a preference for a fastest available translation outcome.

22. The method of claim 13, wherein the one or more user-selected performance criteria comprise a preference for a highest available quality translation outcome.

23. The method of claim 13, wherein determining the second language for translation of the verbally input phrase is in response to receiving user input selecting the second language.

24. The method of claim 13, wherein determining the second language for translation of the verbally input phrase is automatically determined by the client device.

25. The method of claim 24, wherein the second language is automatically determined by the client device based on a location of the user.

26. The method of claim 24, wherein the second language is automatically determined by the client device based on a most recently used translation language.

27. The method of claim 24, wherein the second language is automatically determined by the client device based on language pairs available.

14. (canceled)

16. (canceled)

18-20. -20. (canceled)