In-Call Translation

• US 20150350451A1
• Filed: 02/13/2015
• Published: 12/03/2015
• Est. Priority Date: 05/27/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Aue, Anthony, Menezes, Arul A., Greborio, Pierre P.N., Lindblom, Jonas Nils, Furesj, Fredrik
• Granted Patent: US 9,614,969 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: H04M 2201/50 Telephonic communication in combination with video communication
• CPC Class Codes: H04M 2203/2061 Language aspects
• CPC Class Codes: H04M 2242/12 Language recognition, selection or translation arrangements
• CPC Class Codes: H04M 2250/58 including a multilanguage function
• CPC Class Codes: H04M 3/42 Systems providing special services or facilities to subscribers specially adapted for wireless communication networks H04W4/00
• CPC Class Codes: H04M 3/567 Multimedia conference systems
• CPC Class Codes: H04M 3/568 audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

The disclosure pertains to a communication system for effecting a voice or video call between at least a source user speaking a source language and a target user speaking a target language. A translation procedure is performed on call audio of the call to generate an audio translation of the source user'"'"'s speech in the target language for outputting to the target user. A notification is outputted to the target user to notify the target user of a change in the behaviour of the translation procedure, the change relating to the generation of the translation.

Claims

1. A computer-implemented method performed in a communication system, the communication system for effecting a voice or video call between at least a source user speaking a source language and a target user speaking a target language, the method comprising:

receiving call audio of the call, the call audio comprising speech of the source user in the source language;

performing a translation procedure on the call audio to generate an audio translation of the source user'"'"'s speech in the target language for outputting to the target user; and

signalling a change in the behaviour of the translation procedure, the change relating to the generation of the translation, and thereby causing a notification to be outputted to the target user to notify the target user of the change.

2. A method according to claim 1 wherein the change in the behaviour is the translation procedure entering a listening state, in which it is currently awaiting future speech activity by the source user during a current interval of speech inactivity by the source user.

3. A method according to claim 1 wherein the change in the behaviour is the translation procedure entering a passive translation state responsive to the source user commencing a period of speech activity, in which the translation procedure is monitoring current speech activity by the source user in the call audio.

4. A method according to claim 1 wherein the change in behaviour is the translation procedure entering an active translation state responsive to the source user finishing an interval of speech activity, in which the translation procedure is currently generating an audio translation of the source user'"'"'s speech in that interval to be outputted when that generating is complete.

5. A method according to claim 1 wherein the change in the behaviour is the translation procedure entering an outputting state responsive to the translation procedure completing generation of an audio translation of the source user'"'"'s speech during a preceding interval of source user speech activity, in which that generated audio translation is currently being outputted by the translation procedure for outputting to the target user.

6. A method according to claim 1 wherein the change in the behaviour is the translation procedure entering an error state responsive to the procedure encountering an error in generating the translation.

7. A method according to claim 1 wherein the translated audio is transmitted via a communication network as it is generated to a target device of the target user for outputting via one or more audio output components of that device as it is received.

8. A method according to claim 1 comprising generating synthetic video in dependence on the signalled change in the behaviour, the synthetic video for displaying at a target user device of the target user and embodying the notification.

9. A method according to claim 8 wherein the synthetic video is of an animated avatar performing visual actions, the notification embodied as a visual action by the avatar.

10. A method according to claim 9 wherein the embodying action approximates an action expected to be performed by a human translator or interpreter when communicating a change in their state-of-mind in a real-life translation or interpretation scenario, that change corresponding to the change in the behaviour of the translation procedure.

11. A method according to claim 1 wherein the notification comprises a visual notification for displaying at a target user device of the target user and/or an audio notification for playing out at the target user device and/or a tactile notification outputted by actuating a mechanical component of the target user device.

12. A computer system for use in a communication system, the communication system for effecting a voice or video call between at least a source user speaking a source language and a target user speaking a target language, the computer system comprising:

one or more audio output components available to the target user;

a translation output component configured to output an audio translation of the source user'"'"'s speech in the target language to the target user via the audio output components, the translation generated by performing an automatic translation procedure on call audio of the call which comprises speech of the source user in the source language; and

a notification output component configured to output a notification to the target user to notify the target user of a change in the behaviour of the translation procedure, the change relating to the generation of the translation.

13. A computer system according to claim 12 wherein the call audio comprises speech of the source user in the source language during intervals of source user speech activity interspersed with intervals of speech inactivity in which the source user is not speaking;

wherein, for at least one interval of source user speech activity, the translation output component is configured to output via the audio output components an audio translation of the source user'"'"'s speech during that interval, and wherein the notification output component is configured to output the notification when the outputting of that translation has substantially finished to indicate that the target user is free to respond to the source user.

14. A computer system according to claim 12 wherein the computer system is embodied by a target user device of the target user or by a combination the target user device and at least one other computer device to which the target user device is connected via a communication network.

15. A computer system according to claim 12 comprising:

an input configured to receive a signal signalling the change in the behaviour of translation procedure; and

a notification generation component configured to generate the notification in dependence on the received signal.

16. A computer system according to claim 15 wherein the notification output component is configured to generate output-related information defining the manner in which the notification is to be outputted to the target user;

and

wherein the notification generation component is configured to generate the notification in dependence on the output-related information.

17. A computer system according to claim 16 comprising a display available to the target user, wherein the notification comprises a visual notification to be displayed on the display and the output-related information comprises related layout information.

18. A computer system according to claim 17 wherein the notification generation component is configured to generate synthetic video embodying the notification, the synthetic video generated in dependence on the layout information.

19. A computer system according to claim 18 wherein the synthetic video is of an animated avatar which performs visual actions, the notification embodied as a visual avatar action that is controlled in dependence on the layout information.

20. A computer program product comprising computer code stored on a computer readable storage configured when executed on a processor to cause operations of:

establishing a voice or video call between at least a source user speaking a source language and a target user speaking a target language;

outputting an audio translation of the source user'"'"'s speech in the target language to the target user, the translation generated by performing an automatic translation procedure on call audio of the call which comprises speech of the source user in the source language; and

outputting a notification to the target user to notify the target user of a change in the behaviour of the translation procedure, the change relating to the generation of the translation.