Motion responsive user interface for realtime language translation

• US 9,355,094 B2
• Filed: 11/12/2013
• Issued: 05/31/2016
• Est. Priority Date: 08/14/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Cuthbert, Alexander J., Estelle, Joshua J., Hughes, Macduff Richard, Goyal, Sunny, Jiang, Minqi Sebastian
• Primary Examiner(s): Opsasnick, Michael N
• Application Number:
  US14/077,411
• Publication Number:
  US 20150051898A1
• Time in Patent Office: 931 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/263 Language identification
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G10L 13/086 Detection of language
• CPC Class Codes: G10L 15/005 Language recognition
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

A data processing apparatus receives data indicating a movement of a client device by a first user. The apparatus determines that the movement of the client device is a delimiter motion for switching between a first mode, in which the client device is configured to (i) provide a first interface for a first user speaking in a first language and (ii) perform speech recognition of the first language, and a second mode, in which the client device is configured to (i) provide a second interface for a second user speaking in a second language and (ii) perform speech recognition of the second language, the second interface being different from the first interface. Based on determining that the movement is a delimiter motion, the apparatus switches between the first mode and the second mode without the second user physically interacting with the client device.

Claims

1. A computer-implemented method comprising:

receiving data indicating a movement of a client device by a first user;

determining that the movement of the client device is classified as a delimiter motion for; and

based on determining that the movement is classified as a delimiter motion, switching between a first mode, in which the client device (i) configures a language setting of an automated speech recognition engine to transcribe the speech of the first user into text that is in the first language, and (ii) provides a first interface that includes the transcribed text of the first user that is in the first language, and a translation of the transcribed text of the first user that is in a second language, and a second mode, in which the client device (i) configures the language setting of the automated speech recognition engine to transcribe the speech of a second user into text that is in the second language, and (ii) provides a second interface that includes the transcribed text of the second user that is in the second language, and a translation of the transcribed text of the second user that is in the first language, wherein the second interface is different from the first interface, and wherein switching between the first mode and the second mode occurs without the second user physically interacting with the client device.

2. The method of claim 1, wherein the movement of the client device by the first user comprises rotation of the client device by the first user.

3. The method of claim 1, further comprising, while the client device is in the second mode,

receiving an utterance of the second user in the second language;

and

outputting an audio signal encoding the translation of the transcribed text of the second user that is in the first language.

4. The method of claim 1, wherein the client device configures the language setting of the automated speech recognition engine to transcribe the speech of the first user into text that is in the first language by providing an identifier indicating to the automated speech recognition engine that the speech is in the first language.

5. The method of claim 1, wherein the automated speech recognition engine executes on a server that is remote from the client device.

6. The method of claim 1, further comprising, while the client device is in the first mode,

receiving an utterance of the first user in the first language;

and

outputting an audio signal encoding the translation of the transcribed text of the first user that is in the second language.

7. The method of claim 6, further comprising, while the client device is in the first mode,

translating, at the client device, the utterance of the first user into the second language.

8. A client device comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

receiving data indicating a movement of a client device by a first user;

determining that the movement of the client device is classified as a delimiter motion for; and

based on determining that the movement is classified as a delimiter motion, switching between a first mode, in which the client device (i) configures a language setting of an automated speech recognition engine to transcribe the speech of the first user into text that is in the first language, and (ii) provides a first interface that includes the transcribed text of the first user that is in the first language, and a translation of the transcribed text of the first user that is in a second language, and a second mode, in which the client device (i) configures the language setting of the automated speech recognition engine to transcribe the speech of a second user into text that is in the second language, and (ii) provides a second interface that includes the transcribed text of the second user that is in the second language, and a translation of the transcribed text of the second user that is in the first language, wherein the second interface is different from the first interface, and wherein switching between the first mode and the second mode occurs without the second user physically interacting with the client device.

9. The client device of claim 8, wherein the movement of the client device by the first user comprises rotation of the client device by the first user.

10. The client device of claim 8, wherein the operations further comprise, while the client device is in the second mode,

receiving an utterance of the second user in the second language;

and

outputting an audio signal encoding the translation of the transcribed text of the second user that is in the first language.

11. The client device of claim 8, wherein the operations further comprise, while the client device is in the first mode,

receiving an utterance of the first user in the first language;

and

outputting an audio signal encoding the translation of the transcribed text of the first user that is in the second language.

12. The client device of claim 11, wherein the operations further comprise, while the client device is in the first mode,

translating, at the client device, the utterance of the first user into the second language.

13. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

receiving data indicating a movement of a client device by a first user;

determining that the movement of the client device is classified as a delimiter motion for; and

based on determining that the movement is classified as a delimiter motion, switching between a first mode, in which the client device (i) configures a language setting of an automated speech recognition engine to transcribe the speech of the first user into text that is in the first language, and (ii) provides a first interface that includes the transcribed text of the first user that is in the first language, and a translation of the transcribed text of the first user that is in a second language, and a second mode, in which the client device (i) configures the language setting of the automated speech recognition engine to transcribe the speech of a second user into text that is in the second language, and (ii) provides a second interface that includes the transcribed text of the second user that is in the second language, and a translation of the transcribed text of the second user that is in the first language, wherein the second interface is different from the first interface, and wherein switching between the first mode and the second mode occurs without the second user physically interacting with the client device.

14. The computer-readable medium of claim 13, wherein the movement of the client device by the first user comprises rotation of the client device by the first user.

15. The computer-readable medium of claim 13, wherein the operations further comprise, while the client device is in the first mode,

receiving an utterance of the first user in the first language;

and

outputting an audio signal encoding the translation of the transcribed text of the first user that is in the second language.

16. The computer-readable medium of claim 15, wherein the operations further comprise, while the client device is in the first mode,

translating, at the client device, the utterance of the first user into the second language.