In-call translation

• US 9,614,969 B2
• Filed: 02/13/2015
• Issued: 04/04/2017
• Est. Priority Date: 05/27/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Aue, Anthony, Menezes, Arul A., Lindblom, Jonas Nils, Furesj, Fredrik, Greborio, Pierre P. N.
• Primary Examiner(s): Nguyen, Khai N
• Application Number:
  US14/622,311
• Publication Number:
  US 20150350451A1
• Time in Patent Office: 781 Days
• Field of Search: 37920201
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: H04M 2201/50 Telephonic communication in combination with video communication
• CPC Class Codes: H04M 2203/2061 Language aspects
• CPC Class Codes: H04M 2242/12 Language recognition, selection or translation arrangements
• CPC Class Codes: H04M 2250/58 including a multilanguage function
• CPC Class Codes: H04M 3/42 Systems providing special services or facilities to subscribers specially adapted for wireless communication networks H04W4/00
• CPC Class Codes: H04M 3/567 Multimedia conference systems
• CPC Class Codes: H04M 3/568 audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

The disclosure pertains to a communication system for effecting a voice or video call between at least a source user speaking a source language and a target user speaking a target language. A translation procedure is performed on call audio of the call to generate an audio translation of the source user'"'"'s speech in the target language for outputting to the target user. A notification is outputted to the target user to notify the target user of a change in the behavior of the translation procedure, the change relating to the generation of the translation.

Claims

1. A computer-implemented method performed in a communication system, the communication system for effecting a voice or video call between at least a source user speaking a source language and a target user speaking a target language, the method comprising:

receiving call audio of the call, the call audio comprising speech of the source user in the source language;

performing, by a speech translator module, an automatic translation procedure on the call audio to generate an audio translation of the source user'"'"'s speech in the target language for outputting to the target user; and

signalling, by the speech translator module, a change in behaviour of the automatic translation procedure, the change relating to the generation of the automatic translation, and thereby causing a notification to be outputted to the target user to notify the target user of the change, the notification corresponding to the change in behaviour of the translation procedure and including a synthetic video embodied as a visual action by an animated avatar mimicking visual cues of a human.

2. The computer-implemented method according to claim 1 wherein the change in the behaviour is the translation procedure entering a listening state, in which it is currently awaiting future speech activity by the source user during a current interval of speech inactivity by the source user.

3. The computer-implemented method according to claim 1 wherein the change in the behaviour is the translation procedure entering a passive translation state responsive to the source user commencing a period of speech activity, in which the translation procedure is monitoring current speech activity by the source user in the call audio.

4. The computer-implemented method according to claim 1 wherein the change in behaviour is the translation procedure entering an active translation state responsive to the source user finishing an interval of speech activity, in which the translation procedure is currently generating an audio translation of the source user'"'"'s speech in that interval to be outputted when that generating is complete.

5. The computer-implemented method according to claim 1 wherein the change in the behaviour is the translation procedure entering an outputting state responsive to the translation procedure completing generation of an audio translation of the source user'"'"'s speech during a preceding interval of source user speech activity, in which that generated audio translation is currently being outputted by the translation procedure for outputting to the target user.

6. The computer-implemented method according to claim 1 wherein the change in the behaviour is the translation procedure entering an error state responsive to the procedure encountering an error in generating the translation.

7. The computer-implemented method according to claim 1 wherein the translated audio is transmitted via a communication network as it is generated to a target device of the target user for outputting via one or more audio output components of that device as it is received.

8. The computer-implemented method according to claim 1 wherein the notification comprises a visual notification for displaying at a target user device of the target user and/or an audio notification for playing out at the target user device and/or a tactile notification outputted by actuating a mechanical component of the target user device.

9. A computer system for use in a communication system, the communication system for effecting a voice or video call between at least a source user speaking a source language and a target user speaking a target language, the computer system comprising:

one or more audio output components available to the target user;

a translation output component configured to output an audio translation of the source user'"'"'s speech in the target language to the target user via the audio output components, the translation generated by performing an automatic translation procedure on call audio of the call which comprises speech of the source user in the source language; and

a notification output component configured to output a notification to the target user to notify the target user of a change in behaviour of the translation procedure, the change relating to the generation of the translation and the notification including a synthetic video embodied as a visual action by an animated avatar mimicking visual cues of a human.

10. The computer system according to claim 9 wherein the call audio comprises speech of the source user in the source language during intervals of source user speech activity interspersed with intervals of speech inactivity in which the source user is not speaking;

wherein, for at least one interval of source user speech activity, the translation output component is configured to output via the audio output components an audio translation of the source user'"'"'s speech during that interval, and wherein the notification output component is configured to output the notification when the outputting of that translation has substantially finished to indicate that the target user is free to respond to the source user.

11. The computer system according to claim 9 wherein the computer system is embodied by a target user device of the target user or by a combination the target user device and at least one other computer device to which the target user device is connected via a communication network.

12. The computer system according to claim 9 comprising:

an input configured to receive a signal signalling the change in the behaviour of translation procedure; and

a notification generation component configured to generate the notification in dependence on the received signal.

13. The computer system according to claim 12 wherein the notification output component is configured to generate output-related information defining the manner in which the notification is to be outputted to the target user;

and

wherein the notification generation component is configured to generate the notification in dependence on the output-related information.

14. The computer system according to claim 13 comprising a display available to the target user, wherein the synthetic video embodied as the visual action is displayed on the display and the output-related information comprises related layout information.

15. The computer system according to claim 14 wherein the notification generation component is configured to generate the synthetic video embodied as the visual action and embodying the notification, the synthetic video generated in dependence on the layout information.

16. The computer system according to claim 15 wherein the animated avatar mimicking visual cues of a human is controlled in dependence on the layout information.

17. A computer program product comprising computer code stored on a computer readable storage device configured when executed on a processor to cause operations of:

establishing a voice or video call between at least a source user speaking a source language and a target user speaking a target language;

outputting an audio translation of the source user'"'"'s speech in the target language to the target user, the translation generated by performing an automatic translation procedure on call audio of the call which comprises speech of the source user in the source language; and

outputting a notification to the target user to notify the target user of a change in behaviour of the translation procedure, the change relating to the generation of the translation and the notification including a synthetic video embodied as a visual action by an animated avatar mimicking visual cues of a human.

18. The computer program product according to claim 17 wherein the change in the behaviour is the translation procedure entering a listening state, in which it is currently awaiting future speech activity by the source user during a current interval of speech inactivity by the source user.

19. The computer program product according to claim 17 wherein the change in the behaviour is the translation procedure entering a passive translation state responsive to the source user commencing a period of speech activity, in which the translation procedure is monitoring current speech activity by the source user in the call audio.

20. The computer program product according to claim 17 wherein the change in behaviour is the translation procedure entering an active translation state responsive to the source user finishing an interval of speech activity, in which the translation procedure is currently generating an audio translation of the source user'"'"'s speech in that interval to be outputted when that generating is complete.