Language translation based on speaker-related information

• US 9,053,096 B2
• Filed: 12/29/2011
• Issued: 06/09/2015
• Est. Priority Date: 12/01/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Elwha LLC (Intellectual Ventures LLC)
• Inventors: Lord, Richard T., Lord, Robert W., Myhrvold, Nathan P., Tegreene, Clarence T., Hyde, Roderick A., Wood, Lowell L. Jr., Ishikawa, Muriel Y., Wood, Victoria Y. H., Whitmer, Charles, Bahl, Paramvir, Burger, Douglas C., Chandra, Ranveer, Gates, William H. III, Holman, Paul, Kare, Jordin T., Mundie, Craig J., Paek, Tim, Tan, Desney S., Zhong, Lin, Dyor, Matthew G.
• Primary Examiner(s): SAINT CYR, LEONARD
• Application Number:
  US13/340,143
• Publication Number:
  US 20130144595A1
• Time in Patent Office: 1,258 Days
• Field of Search: 704 2- 10, 704/246, 704/247, 704/251, 704/252
• US Class Current: 1/1
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation

Abstract

Techniques for ability enhancement are described. Some embodiments provide an ability enhancement facilitator system (“AEFS”) configured to automatically translate utterances from a first to a second language, based on speaker-related information determined from speaker utterances and/or other sources of information. In one embodiment, the AEFS receives data that represents an utterance of a speaker in a first language, the utterance obtained by a hearing device of the user, such as a hearing aid, smart phone, media player/device, or the like. The AEFS then determines speaker-related information associated with the identified speaker, such as by determining demographic information (e.g., gender, language, country/region of origin) and/or identifying information (e.g., name or title) of the speaker. The AEFS translates the utterance in the first language into a message in a second language, based on the determined speaker-related information. The AEFS then presents the message in the second language to the user.

Claims

1. A method for ability enhancement, the method comprising:

receiving data representing a speech signal obtained at a hearing device associated with a user, the speech signal representing an utterance of a speaker in a first language;

determining speaker-related information associated with the speaker, based on the data representing the speech signal;

translating the utterance in the first language into a message in a second language, based on the speaker-related information;

presenting the message in the second language; and

distributing processing tasks among available computing resources including the hearing device, a computing device of the user and/or the speaker, and a remote computing system, by determining where to perform the processing tasks by identifying one of the computing resources that has available capacity, the processing tasks including determining speaker-related information and translating the utterance in the first language into a message in a second language.

2. The method of claim 1, wherein the determining speaker-related information includes:

determining the first language.

3. The method of claim 2, wherein the determining the first language includes:

concurrently processing the received data with multiple speech recognizers that are each configured to recognize speech in a different corresponding language; and

selecting as the first language the language corresponding to a speech recognizer of the multiple speech recognizer that produces a result that has a higher confidence level than other of the multiple speech recognizers.

4. The method of claim 2, wherein the determining the first language includes:

identifying signal characteristics in the received data that are correlated with the first language.

5. The method of claim 2, wherein the determining the first language includes:

receiving an indication of a current location of the user;

determining one or more languages that are commonly spoken at the current location; and

selecting one of the one or more languages as the first language.

6. The method of claim 2, wherein the determining the first language includes:

presenting indications of multiple languages to the user; and

receiving from the user an indication of one of the multiple languages.

7. The method of claim 2, further comprising:

selecting a speech recognizer configured to recognize speech in the first language.

8. The method of claim 1, wherein the translating the utterance in the first language into a message in a second language includes:

performing speech recognition, based on the speaker-related information, on the data representing the speech signal to convert the utterance in the first language into text representing the utterance in the first language; and

translating, based on the speaker-related information, the text representing the utterance in the first language into text representing the message in the second language.

9. The method of claim 8, wherein the presenting the message in the second language includes:

performing speech synthesis to convert the text representing the utterance in the second language into audio data representing the message in the second language; and

causing the audio data representing the message in the second language to be played to the user.

10. The method of claim 1, wherein the translating the utterance in the first language into a message in a second language includes:

translating the utterance based on speaker-related information including an identity of the speaker.

11. The method of claim 1, wherein the translating the utterance in the first language into a message in a second language includes:

translating the utterance based on speaker-related information including a language model that is specific to the speaker.

12. The method of claim 11, wherein the translating the utterance based on speaker-related information including a language model that is specific to the speaker includes:

translating the utterance based on a language model that is tailored to a group of people of which the speaker is a member.

13. The method of claim 11, wherein the translating the utterance based on speaker-related information including a language model that is specific to the speaker includes:

generating the language model based on communications generated by the speaker.

14. The method of claim 13, wherein the generating the language model based on communications generated by the speaker includes:

generating the language model based on emails transmitted by the speaker.

15. The method of claim 13, wherein the generating the language model based on communications generated by the speaker includes:

generating the language model based on documents authored by the speaker.

16. The method of claim 13, wherein the generating the language model based on communications generated by the speaker includes:

generating the language model based on social network messages transmitted by the speaker.

17. The method of claim 1, wherein the translating the utterance in the first language into a message in a second language includes:

translating the utterance based on speaker-related information including a speech model that is tailored to the speaker.

18. The method of claim 17, wherein the translating the utterance based on speaker-related information including a speech model that is tailored to the speaker includes:

translating the utterance based on a speech model that is tailored to a group of people of which the speaker is a member.

19. The method of claim 1, wherein the translating the utterance in the first language into a message in a second language includes:

translating the utterance based on speaker-related information including an information item that references the speaker.

20. The method of claim 1, wherein the translating the utterance in the first language into a message in a second language includes:

translating the utterance based on speaker-related information including a document that references the speaker.

21. The method of claim 1, wherein the translating the utterance in the first language into a message in a second language includes:

translating the utterance based on speaker-related information including a message that references the speaker.

22. The method of claim 1, wherein the translating the utterance in the first language into a message in a second language includes:

translating the utterance based on speaker-related information including a calendar event that references the speaker.

23. The method of claim 1, wherein the translating the utterance in the first language into a message in a second language includes:

translating the utterance based on speaker-related information including an indication of gender of the speaker.

24. The method of claim 1, wherein the translating the utterance in the first language into a message in a second language includes:

translating the utterance based on speaker-related information including an organization to which the speaker belongs.

25. The method of claim 1, wherein the determining speaker-related information includes:

performing voice identification based on the received data to identify the speaker.

26. The method of claim 25, further comprising:

determining that the speaker cannot be identified.

27. The method of claim 26, further comprising:

when it is determined that the speaker cannot be identified, storing the received data for system training.

28. The method of claim 26, further comprising:

when it is determined that the speaker cannot be identified, notifying the user.

29. The method of claim 1, further comprising:

receiving data representing a speech signal that represents an utterance of the user; and

determining the speaker-related information based on the data representing a speech signal that represents an utterance of the user.

30. The method of claim 29, wherein the determining the speaker-related information based on the data representing a speech signal that represents an utterance of the user includes:

determining whether the utterance of the user includes a name of the speaker.

31. The method of claim 1, wherein the determining speaker-related information includes:

identifying a plurality of candidate speakers; and

presenting indications of the plurality of candidate speakers.

32. The method of claim 31, further comprising:

receiving from the user a selection of one of the plurality of candidate speakers that is the speaker; and

determining the speaker-related information based on the selection received from the user.

33. The method of claim 31, further comprising:

receiving from the user an indication that none of the plurality of candidate speakers are the speaker; and

training a speaker identification system based on the received indication.

34. The method of claim 31, further comprising:

training a speaker identification system based on a selection regarding the plurality of candidate speakers received from a user.

35. The method of claim 1, wherein the presenting the message in the second language includes:

transmitting the message in the second language from a first device to a second device.

36. The method of claim 35, wherein the transmitting the message in the second language from a first device to a second device includes:

transmitting the message in the second language from a smart phone or portable media device to the second device.

37. The method of claim 35, wherein the transmitting the message in the second language from a first device to a second device includes:

transmitting the message in the second language from a server system to the second device.

38. The method of claim 37, wherein the transmitting the message in the second language from a server system includes:

transmitting the message in the second language from a server system to a mobile device of the user.

39. The method of claim 1, further comprising:

performing the receiving data representing a speech signal, the determining speaker-related information, the translating the utterance in the first language into a message in a second language, and/or the presenting the message in the second language on a mobile device that is operated by the user.

40. The method of claim 1, further comprising:

performing the receiving data representing a speech signal, the determining speaker-related information, the translating the utterance in the first language into a message in a second language, and/or the presenting the message in the second language on a desktop computer that is operated by the user.

41. The method of claim 1, further comprising:

receiving at least some of speaker-related information from the identified computing resource.

42. The method of claim 1, further comprising:

informing the user of the speaker-related information.

43. The method of claim 42, further comprising:

receiving feedback from the user regarding correctness of the speaker-related information; and

refining the speaker-related information based on the received feedback.

44. The method of claim 43, wherein the refining the speaker-related information based on the received feedback includes:

presenting speaker-related information corresponding to each of multiple likely speakers; and

receiving from the user an indication that the speaker is one of the multiple likely speakers.

45. The method of claim 42, wherein the informing the user of the speaker-related information includes:

presenting the speaker-related information on a display of the hearing device.

46. The method of claim 42, wherein the informing the user of the speaker-related information includes:

presenting the speaker-related information on a display of a computing device that is distinct from the hearing device.

47. A non-transitory computer-readable medium including instructions that are configured, when executed, to cause a computing system to perform a method for ability enhancement, the method comprising:

receiving data representing a speech signal obtained at a hearing device associated with a user, the speech signal representing an utterance of a speaker in a first language;

determining speaker-related information associated with the speaker, based on the data representing the speech signal;

translating the utterance in the first language into a message in a second language, based on the speaker-related information;

presenting the message in the second language; and

distributing processing tasks among available computing resources including the hearing device, a computing device of the user and/or the speaker, and a remote computing system, by determining where to perform the processing tasks by identifying one of the computing resources that has available capacity, the processing tasks including determining speaker-related information and translating the utterance in the first language into a message in a second language.

48. A computing system for ability enhancement, the computing system comprising:

a processor;

a memory; and

a module that is stored in the memory and that is configured, when executed by the processor, to perform a method comprising;

receiving data representing a speech signal obtained at a hearing device associated with a user, the speech signal representing an utterance of a speaker in a first language;

determining speaker-related information associated with the speaker, based on the data representing the speech signal;

translating the utterance in the first language into a message in a second language, based on the speaker-related information;

presenting the message in the second language; and

distributing processing tasks among available computing resources including the hearing device, a computing device of the user and/or the speaker, and a remote computing system, by determining where to perform the processing tasks by identifying one of the computing resources that has available capacity, the processing tasks including determining speaker-related information and translating the utterance in the first language into a message in a second language.