Visual presentation of speaker-related information

• US 8,934,652 B2
• Filed: 12/13/2011
• Issued: 01/13/2015
• Est. Priority Date: 12/01/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Elwha LLC (Intellectual Ventures LLC)
• Inventors: Lord, Richard T., Lord, Robert W., Myhrvold, Nathan P., Tegreene, Clarence T., Hyde, Roderick A., Wood, Lowell L. Jr., Ishikawa, Muriel Y., Wood, Victoria Y. H., Whitmer, Charles, Bahl, Paramvir, Burger, Douglas C., Chandra, Ranveer, Gates, William H. III, Holman, Paul, Kare, Jordin T., Mundie, Craig J., Paek, Tim, Tan, Desney S., Zhong, Lin, Dyor, Matthew G.
• Primary Examiner(s): Kuntz, Curtis
• Assistant Examiner(s): Joshi, Sunita
• Application Number:
  US13/324,232
• Publication Number:
  US 20130144623A1
• Time in Patent Office: 1,127 Days
• Field of Search: 381/312, 381/315, 381/320, 381/60, 455/556.1, 704/249
• US Class Current: 381/312
• CPC Class Codes: G09B 21/006 using audible presentation of the information
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques

Abstract

Techniques for ability enhancement are described. Some embodiments provide an ability enhancement facilitator system (“AEFS”) configured to determine and present speaker-related information based on speaker utterances. In one embodiment, the AEFS receives data that represents an utterance of a speaker received by a hearing device of the user, such as a hearing aid, smart phone, media player/device, or the like. The AEFS identifies the speaker based on the received data, such as by performing speaker recognition. The AEFS determines speaker-related information associated with the identified speaker, such as by determining an identifier (e.g., name or title) of the speaker, by locating an information item (e.g., an email message, document) associated with the speaker, or the like. The AEFS then informs the user of the speaker-related information, such as by presenting the speaker-related information on a display of the hearing device or some other device accessible to the user.

Claims

1. A method for ability enhancement, the method comprising:

receiving data representing a speech signal obtained at a hearing device associated with a user, the speech signal representing an utterance of a speaker who is distinct from the user;

identifying the speaker based on the data representing the speech signal, wherein identifying the speaker includes;

performing speech recognition to convert the received data into text data; and

identifying the speaker based on the text data, by finding a document that references the speaker and that includes one or more words of the text data, thereby inferring that the speaker is the author of the document, a recipient of the document, or a person described in the document;

determining speaker-related information associated with the identified speaker; and

visually presenting the speaker-related information to the user.

2. The method of claim 1, wherein the visually presenting the speaker-related information includes:

presenting the speaker-related information on a display of the hearing device.

3. The method of claim 1, wherein the visually presenting the speaker-related information includes:

presenting the speaker-related information on a display of a computing device that is distinct from the hearing device.

4. The method of claim 1, wherein the visually presenting the speaker-related information includes:

determining a display to serve as a destination for the speaker-related information.

5. The method of claim 4, wherein the determining a display includes:

selecting from one of multiple displays, based at least in part on whether each of the multiple displays is capable of displaying all of the speaker-related information.

6. The method of claim 4, wherein the determining a display includes:

selecting from one of multiple displays, based at least in part on a size of each of the multiple displays.

7. The method of claim 4, wherein the determining a display includes:

selecting from one of multiple displays, based at least in part on whether each of the multiple displays is suitable for displaying the speaker-related information, the speaker-related information being at least one of text information, a communication, a document, an image, and/or a calendar event.

8. The method of claim 1, further comprising:

audibly notifying the user to view the speaker-related information on a display device.

9. The method of claim 8, wherein the audibly notifying the user includes:

playing a tone via an audio speaker of the hearing device.

10. The method of claim 8, wherein the audibly notifying the user includes:

playing synthesized speech via an audio speaker of the hearing device, the synthesized speech telling the user to view the display device, such that the speaker does not hear the notification.

11. The method of claim 8, wherein the audibly notifying the user includes:

telling the user that at least one of a document, a calendar event, and/or a communication is available for viewing on the display device.

12. The method of claim 8, wherein the audibly notifying the user includes:

audibly notifying the user in a manner that is not audible to the speaker.

13. The method of claim 1, wherein the visually presenting the speaker-related information includes:

informing the user of an identifier of the speaker.

14. The method of claim 1, wherein the visually presenting the speaker-related information includes:

informing the user of information aside from identifying information related to the speaker.

15. The method of claim 1, wherein the visually presenting the speaker-related information includes:

informing the user of an organization to which the speaker belongs.

16. The method of claim 1, wherein the visually presenting the speaker-related information includes:

informing the user of a previously transmitted communication referencing the speaker.

17. The method of claim 1, wherein the visually presenting the speaker-related information includes:

informing the user of an event involving the user and the speaker.

18. The method of claim 1, wherein the determining speaker-related information includes:

accessing information items associated with the speaker.

19. The method of claim 18, wherein the accessing information items associated with the speaker includes:

accessing a social networking service to find messages or status updates that reference the speaker.

20. The method of claim 18, wherein the accessing information items associated with the speaker includes:

accessing a calendar to find information about appointments with the speaker.

21. The method of claim 1, further comprising:

retrieving information items that reference the text data; and

informing the user of the retrieved information items.

22. The method of claim 1, wherein the performing speech recognition includes:

performing speech recognition based at least in part on a language model that represents word transition likelihoods and that is associated with the speaker.

23. The method of claim 22, wherein the performing speech recognition based at least in part on a language model associated with the speaker includes:

generating the language model based on communications generated by the speaker.

24. The method of claim 23, wherein the generating the language model based on communications generated by the speaker includes:

generating the language model based on emails transmitted by the speaker.

25. The method of claim 23, wherein the generating the language model based on communications generated by the speaker includes:

generating the language model based on documents authored by the speaker.

26. The method of claim 23, wherein the generating the language model based on communications generated by the speaker includes:

generating the language model based on social network messages transmitted by the speaker.

27. The method of claim 1, further comprising:

receiving data representing a speech signal that represents an utterance of the user; and

identifying the speaker based on the data representing a speech signal that represents an utterance of the user.

28. The method of claim 27, wherein the identifying the speaker based on the data representing a speech signal that represents an utterance of the user includes:

determining whether the utterance of the user includes a name of the speaker.

29. The method of claim 1, further comprising:

receiving data representing an ongoing conversation amongst multiple speakers;

identifying the multiple speakers based on the data representing the ongoing conversation; and

as each of the multiple speakers takes a turn speaking during the ongoing conversation, informing the user of a name or other speaker-related information associated with the speaker.

30. The method of claim 29, wherein the receiving data representing an ongoing conversation amongst multiple speakers includes:

receiving audio data from a telephonic conference call, the received audio data representing utterances made by at least one of the multiple speakers.

31. The method of claim 29, further comprising:

presenting, while a current speaker is speaking, speaker-related information on a display device of the user, the displayed speaker-related information identifying the current speaker.

32. The method of claim 1, wherein the visually presenting the speaker-related information includes:

transmitting the speaker-related information from a first device to a second device having a display.

33. The method of claim 32, wherein the transmitting the speaker-related information from a first device to a second device includes:

wirelessly transmitting the speaker-related information.

34. The method of claim 32, wherein the transmitting the speaker-related information from a first device to a second device includes:

transmitting the speaker-related information from a smart phone or portable media player to the second device.

35. The method of claim 32, wherein the transmitting the speaker-related information from a first device to a second device includes:

transmitting the speaker-related information from a server system to the second device.

36. The method of claim 1, further comprising:

performing the receiving data representing a speech signal, the identifying the speaker, and/or the determining speaker-related information on a mobile device that is operated by the user.

37. The method of claim 1, further comprising:

performing the receiving data representing a speech signal, the identifying the speaker, and/or the determining speaker-related information on a desktop computer that is operated by the user.

38. The method of claim 1, further comprising:

determining to perform at least some of identifying the speaker or determining speaker-related information on another computing device that has available processing capacity.

39. The method of claim 38, further comprising:

receiving at least some of speaker-related information from the another computing device.

40. The method of claim 1, further comprising:

determining whether or not the user can name the speaker, by determining whether the user has named the speaker; and

when it is determined that the user cannot name the speaker, visually presenting the speaker-related information by informing the user of an identifier of the speaker.

41. The method of claim 40, wherein the determining whether the user has named the speaker includes:

determining whether the speaker has uttered a given name or surname of the speaker.

42. The method of claim 40, wherein the determining whether the user has named the speaker includes:

determining whether the speaker has uttered a nickname of the speaker.

43. The method of claim 40, wherein the determining whether the user has named the speaker includes:

determining whether the speaker has uttered a name of a relationship between the user and the speaker.