Audible assistance

• US 8,811,638 B2
• Filed: 12/01/2011
• Issued: 08/19/2014
• Est. Priority Date: 12/01/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Elwha LLC (Intellectual Ventures LLC)
• Inventors: Lord, Richard T., Lord, Robert W., Myhrvold, Nathan P., Tegreene, Clarence T., Hyde, Roderick A., Wood, Lowell L. Jr., Ishikawa, Muriel Y., Wood, Victoria Y. H., Whitmer, Charles, Bahl, Paramvir, Burger, Douglas C., Chandra, Ranveer, Gates, William H. III, Holman, Paul, Kare, Jordin T., Mundie, Craig J., Paek, Tim, Tan, Desney S., Zhong, Lin, Dyor, Matthew G.
• Primary Examiner(s): Kuntz, Curtis
• Assistant Examiner(s): Joshi, Sunita
• Application Number:
  US13/309,248
• Publication Number:
  US 20130142365A1
• Time in Patent Office: 992 Days
• Field of Search: 381/312, 381/315, 381/320, 381/60, 455/556.1, 700/94, 704/246
• US Class Current: 381/312
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: H04M 2201/41 using speaker recognition speaker recognition per se G10L17/00
• CPC Class Codes: H04M 3/42391 where the subscribers are hearing-impaired persons, e.g. telephone devices for the deaf
• CPC Class Codes: H04R 2225/43 Signal processing in hearing aids to enhance the speech intelligibility
• CPC Class Codes: H04R 2225/55 Communication between hearing aids and external devices via a network for data exchange
• CPC Class Codes: H04R 25/554 using a wireless connection, e.g. between microphone and amplifier or using Tcoils

Abstract

Techniques for sensory enhancement and augmentation are described. Some embodiments provide an audible assistance facilitator system (“AAFS”) configured to provide audible assistance to a user via a hearing device. In one embodiment, the AAFS receives data that represents an utterance of a speaker received by a hearing device of the user, such as a hearing aid, smart phone, media device, or the like. The AAFS identifies the speaker based on the received data, such as by performing speaker recognition. The AAFS determines speaker-related information associated with the identified speaker, such as by determining an identifier (e.g., name or title) of the speaker, by locating an information item (e.g., an email message, document) associated with the speaker, or the like. The AAFS then informs the user of the speaker-related information, such as by causing an audio representation of the speaker-related information to be output via the hearing device.

Claims

1. A method for providing audible assistance via a hearing device, the method comprising:

receiving data representing a speech signal obtained at a hearing device associated with a user, the speech signal representing an utterance of a speaker;

identifying the speaker based on the data representing the speech signal;

determining speaker-related information associated with the identified speaker;

determining whether or not the user can name the speaker, by determining whether the user has named the speaker; and

when it is determined that the user cannot name the speaker, informing the user of the speaker-related information via the hearing device, wherein the informing the user of the speaker-related information via the hearing device includes;

informing the user of an identifier of the speaker.

2. The method of claim 1, wherein the informing the user of the speaker-related information via the hearing device includes:

informing the user of information aside from identifying information related to the speaker.

3. The method of claim 1, wherein the informing the user of the speaker-related information via the hearing device includes:

informing the user of an organization to which the speaker belongs.

4. The method of claim 3, wherein the informing the user of an organization includes:

informing the user of a company associated with the speaker.

5. The method of claim 1, wherein the informing the user of the speaker-related information via the hearing device includes:

informing the user of a previously transmitted communication referencing the speaker, wherein the previously transmitted communication is an email or text message transmitted between the speaker and the user.

6. The method of claim 1, wherein the informing the user of the speaker-related information via the hearing device includes:

informing the user of an event involving the user and the speaker.

7. The method of claim 6, wherein the informing the user of an event includes:

informing the user of a previously occurring event, a future event, a project, a meeting, and/or a deadline.

8. The method of claim 1, wherein the determining speaker-related information includes:

accessing information items associated with the speaker.

9. The method of claim 8, wherein the accessing information items associated with the speaker includes:

searching for information items that reference the speaker, searching stored emails to find emails that reference the speaker, searching stored text messages to find text messages that reference the speaker, accessing a social networking service to find messages or status updates that reference the speaker, accessing a calendar to find information about appointments with the speaker, and/or accessing a document store to find documents that reference the speaker.

10. The method of claim 1, wherein the identifying the speaker includes:

performing voice identification based on the received data to identify the speaker.

11. The method of claim 10, wherein the performing voice identification includes:

comparing properties of the speech signal with properties of previously recorded speech signals from multiple distinct speakers.

12. The method of claim 11, further comprising:

processing voice messages from the multiple distinct speakers to generate voice print data for each of the multiple distinct speakers.

13. The method of claim 10, wherein the performing voice identification includes:

processing telephone voice messages stored by a voice mail service.

14. The method of claim 1, wherein the identifying the speaker includes:

performing speech recognition to convert the received data into text data; and

identifying the speaker based on the text data.

15. The method of claim 14, wherein the identifying the speaker based on the text data includes:

finding a document that references the speaker and that includes one or more words in the text data.

16. The method of claim 14, further comprising:

retrieving information items that reference the text data; and

informing the user of the retrieved information items.

17. The method of claim 14, further comprising:

converting the text data into audio data that represents a voice of a different speaker; and

causing the audio data to be played through the hearing device.

18. The method of claim 14, wherein the performing speech recognition includes:

performing speech recognition based at least in part on a language model associated with the speaker.

19. The method of claim 18, wherein the performing speech recognition based at least in part on a language model associated with the speaker includes:

generating the language model based on communications generated by the speaker.

20. The method of claim 19, wherein the generating the language model based on communications generated by the speaker includes:

generating the language model based on emails transmitted by the speaker, documents authored by the speaker, and/or social network messages transmitted by the speaker.

21. The method of claim 1, further comprising:

receiving data representing a speech signal that represents an utterance of the user; and

identifying the speaker based on the data representing a speech signal that represents an utterance of the user.

22. The method of claim 21, wherein the identifying the speaker based on the data representing a speech signal that represents an utterance of the user includes:

determining whether the utterance of the user includes a name of the speaker.

23. The method of claim 1, wherein the identifying the speaker includes:

receiving context information related to the user; and

identifying the speaker, based on the context information.

24. The method of claim 23, wherein the receiving context information related to the user includes:

receiving an indication of a location of the user; and

determining a plurality of persons with whom the user commonly interacts at the location.

25. The method of claim 24, wherein the receiving an indication of a location of the user includes:

receiving a GPS location from a mobile device of the user, receiving a network identifier that is associated with the location, receiving an indication that the user is at a workplace, and/or receiving an indication that the user is at a residence.

26. The method of claim 23, wherein the receiving context information related to the user includes:

receiving information about a communication that references the speaker, the context information including a message and/or a document that references the speaker.

27. The method of claim 1, further comprising:

receiving data representing an ongoing conversation amongst multiple speakers;

identifying the multiple speakers based on the data representing the ongoing conversation; and

as each of the multiple speakers takes a turn speaking during the ongoing conversation, informing the user of a name or other speaker-related information associated with the speaker.

28. The method of claim 1, further comprising:

developing a corpus of speaker data by recording speech from a plurality of speakers; and

identifying the speaker based at least in part on the corpus of speaker data.

29. The method of claim 28, further comprising:

generating a speech model associated with each of the plurality of speakers, based on the recorded speech.

30. The method of claim 28, further comprising:

receiving feedback regarding accuracy of the speaker-related information; and

training a speech processor based at least in part on the received feedback.

31. The method of claim 1, wherein the informing the user of the speaker-related information via the hearing device includes:

transmitting the speaker-related information to a hearing device configured to amplify speech for the user.

32. The method of claim 1, wherein the informing the user of the speaker-related information via the hearing device includes:

transmitting the speaker-related information to the hearing device from a computing system that is remote from the hearing device.

33. The method of claim 32, wherein the transmitting the speaker-related information to the hearing device from a computing system includes:

transmitting the speaker-related information from a mobile device that is operated by the user, that is in communication with the hearing device, and that is a smart phone and/or a portable media player.

34. The method of claim 32, wherein the transmitting the speaker-related information to the hearing device from a computing system includes:

transmitting the speaker-related information from a server system.

35. The method of claim 34, wherein the transmitting the speaker-related information from a server system includes:

transmitting the speaker-related information from a server system that resides in a data center.

36. The method of claim 1, wherein the informing the user of the speaker-related information via the hearing device includes:

transmitting the speaker-related information to earphones in communication with a mobile device that is operating as the hearing device, transmitting the speaker-related information to earbuds in communication with a mobile device that is operating as the hearing device, transmitting the speaker-related information to a headset in communication with a mobile device that is operating as the hearing device, and/or transmitting the speaker-related information to a pillow speaker in communication with a mobile device that is operating as the hearing device.

37. The method of claim 1, wherein the identifying the speaker includes:

identifying the speaker, performed on a mobile device that is operated by the user and that is a smart phone and/or a personal media device.

38. The method of claim 1, wherein the determining speaker-related information includes:

determining speaker-related information, performed on a mobile device that is operated by the user and that is a smart phone and/or a personal media device.

39. The method of claim 1, wherein the determining whether the user has named the speaker includes:

determining whether the speaker has uttered a given name or surname of the speaker, determining whether the speaker has uttered a nickname of the speaker, and/or determining whether the speaker has uttered a name of a relationship between the user and the speaker.

40. The method of claim 1, wherein the determining whether or not the user can name the speaker includes:

determining whether the user has uttered information that is related to both the speaker and the user.

41. The method of claim 1, wherein the determining whether the user has named the speaker includes:

determining whether the user has named a person, place, thing, or event that the speaker and the user have in common.

42. The method of claim 1, wherein the determining whether or not the user can name the speaker includes:

performing speech recognition to convert an utterance of the user into text data; and

determining whether or not the user can name the speaker based at least in part on the text data.

43. The method of claim 1, wherein the determining whether or not the user can name the speaker includes:

when the user does not name the speaker within a predetermined time interval, determining that the user cannot name the speaker.