Network based speech to speech translation

• US 10,025,781 B2
• Filed: 06/09/2014
• Issued: 07/17/2018
• Est. Priority Date: 08/05/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: LeBeau, Michael J., Jitkoff, John Nicholas
• Primary Examiner(s): He, Jialong
• Application Number:
  US14/299,327
• Publication Number:
  US 20140288919A1
• Time in Patent Office: 1,499 Days
• Field of Search: 704 2- 8
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: H04M 2203/2061 Language aspects
• CPC Class Codes: H04M 2242/12 Language recognition, selection or translation arrangements
• CPC Class Codes: H04M 2250/58 including a multilanguage function
• CPC Class Codes: H04M 3/42348 Location-based services which utilize the location information of a target

Abstract

A method, performed on a server, of translating between languages includes receiving first audio data for a first language from a mobile device, translating the first audio data to second audio data for a second language, receiving an indication that the mobile device has moved between two locations, and sending the second audio data to the mobile device in response to the indication.

Claims

1. A computer-implemented method comprising:

receiving, from a server and at a first client device, during a translation process, first audio data corresponding to a translation of first user input speech in a first language to a second language, wherein the first language is determined based at least on a determined geographical location of a second client device and the second language is determined based at least on a determined geographical location of the first client device;

providing, for output at the first client device and during the translation process, the first audio data corresponding to the translation of the first user input speech in the first language to the second language;

receiving, at the first client device and during the translation process, second audio data corresponding to second user input speech in the second language;

determining, by the first client device, to initiate translation of the second audio data corresponding to the second user input speech in the second language; and

based on the determination to initiate translation of the second audio data corresponding to the second user input speech in the second language, transmitting the second audio data corresponding to the second user input speech in the second language from the first client device to the server for translation from the second language to the first language, wherein the server is configured to;

generate third audio data corresponding to a translation of the second user input speech in the second language to the first language;

store the third audio data corresponding to the translation of the second user input speech; and

in response to receiving a request from the second client device to retrieve the third audio data corresponding to the translation of the second user input speech, transmit, to the second client device, the third audio data corresponding to the translation of the second user input speech.

2. The method of claim 1, wherein the second audio data corresponding to the second user input speech in the second language corresponds to user input speech in the second language received by a microphone associated with the first client device.

3. The method of claim 1, comprising:

receiving the second audio data corresponding to the second user input speech in the second language after providing the first audio data corresponding to the translation of the first user input speech in the first language to the second language for output.

4. The method of claim 1, wherein the first client device and the second client device are paired.

5. The method of claim 4, comprising:

performing a process to pair the first client device and the second client device, the process comprising;

transmitting, from the first client device and for receipt by the second client device, information identifying the first client device, and

receiving, by the first client device and from the second client device, information identifying the second client device.

6. The method of claim 1, wherein the first client device receives the first audio data corresponding to the translation of the first user input speech in the first language to the second language without receiving user input indicating a request for the first audio data corresponding to the translation of the first user input speech in the first language to the second language.

7. The method of claim 1, wherein the second audio data corresponding to the second user input speech in the second language is received at the first client device without receiving additional user input indicating an instruction to continue the translation process.

8. The method of claim 1, wherein:

the translation process occurs between the first client device and the second client device, and the second client device is remote from the first client device; and

the server is configured to generate the first audio data corresponding to the translation of the first user input speech in the first language to the second language.

9. The method of claim 1, wherein the first client device receives the first audio data corresponding to the translation of the first user input speech in the first language to the second language in response to a user input indicating a request for the first audio data corresponding to the translation of the first user input speech in the first language to the second language.

10. The method of claim 1, wherein the first client device is configured to transmit a target language identifier to the server each time the first client device transmits audio data corresponding to user input speech to the server for translation.

11. The method of claim 1, wherein the server is further configured to:

store the third audio data that corresponds to the translation of the second user input speech before receiving the request from the second client device to retrieve the third audio data; and

access the stored third audio data for transmission to the second device in response to receiving the request from the second client device.

12. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

receiving, from a server and at a first client device, during a translation process, first audio data corresponding to a translation of first user input speech in a first language to a second language, wherein the first language is determined based at least on a determined geographical location of a second client device and the second language is determined based at least on a determined geographical location of the first client device;

providing, for output at the first client device and during the translation process, the first audio data corresponding to the translation of the first user input speech in the first language to the second language;

receiving, at the first client device and during the translation process, second audio data corresponding to second user input speech in the second language;

determining, by the first client device, to initiate translation of the second audio data corresponding to the second user input speech in the second language; and

based on the determination to initiate translation of the second audio data corresponding to the second user input speech in the second language, transmitting the second audio data corresponding to the second user input speech in the second language from the first client device to the server for translation from the second language to the first language, wherein the server is configured to;

generate third audio data corresponding to a translation of the second user input speech in the second language to the first language;

store the third audio data corresponding to the translation of the second user input speech; and

in response to receiving a request from the second client device to retrieve the third audio data corresponding to the translation of the second user input speech, transmit, to the second client device, the third audio data corresponding to the translation of the second user input speech.

13. The system of claim 12, wherein the second audio data corresponding to the second user input speech in the second language corresponds to user input speech in the second language received by a microphone associated with the first client device.

14. The system of claim 12, wherein the operations comprise:

receiving the second audio data corresponding to the second user input speech in the second language after providing the first audio data corresponding to the translation of the first user input speech in the first language to the second language for output.

15. The system of claim 12, wherein the first client device and the second client device are paired.

16. A computer-readable storage device encoded with a computer program, the program comprising instructions that if executed by one or more computers cause the one or more computers to perform operations comprising:

receiving, from a server and at a first client device, during a translation process, first audio data corresponding to a translation of first user input speech in a first language to a second language, wherein the first language is determined based at least on a determined geographical location of a second client device and the second language is determined based at least on a determined geographical location of the first client device;

providing, for output at the first client device and during the translation process, the first audio data corresponding to the translation of the first user input speech in the first language to the second language;

receiving, at the first client device and during the translation process, second audio data corresponding to second user input speech in the second language;

determining, by the first client device, to initiate translation of the second audio data corresponding to the second user input speech in the second language; and

based on the determination to initiate translation of the second audio data corresponding to the second user input speech in the second language, transmitting the second audio data corresponding to the second user input speech in the second language from the first client device to the server for translation from the second language to the first language, wherein the server is configured to;

generate third audio data corresponding to a translation of the second user input speech in the second language to the first language;

store the third audio data corresponding to the translation of the second user input speech; and

in response to receiving a request from the second client device to retrieve the third audio data corresponding to the translation of the second user input speech, transmit, to the second client device, the third audio data corresponding to the translation of the second user input speech.

17. The device of claim 16, wherein the second audio data corresponding to the second user input speech in the second language corresponds to user input speech in the second language received by a microphone associated with the first client device.

18. The device of claim 16, wherein the operations comprise:

receiving the second audio data corresponding to the second user input speech in the second language after providing the first audio data corresponding to the translation of the first user input speech in the first language to the second language for output.

19. The device of claim 16, wherein the first client device and the second client device are paired.

20. A computer-implemented method comprising:

transmitting, from a server and to a first client device, during a translation process, first audio data corresponding to a translation of first user input speech in a first language to a second language, wherein the first language is determined based at least on a determined geographical location of a second client device and the second language is determined based at least on a determined geographical location of the first client device;

receiving, at the server and from the first client device, and during the translation process, second audio data corresponding to second user input speech in the second language;

generating, at the server, third audio data corresponding to a translation of the second user input speech in the second language to the first language;

storing, at the server, the third audio data corresponding to the translation of the second user input speech;

receiving, at the server and from the second client device after having stored the third audio data corresponding to the translation of the second user input speech, a request to provide the second client device with the third audio data; and

in response to receiving the request to provide the second client device with the third audio data corresponding to the translation of the second user input speech, transmitting, from the server to the second client device, the third audio data corresponding to the translation of the second user input speech.