Enhanced voice conferencing with history, language translation and identification

• US 9,245,254 B2
• Filed: 02/15/2012
• Issued: 01/26/2016
• Est. Priority Date: 12/01/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Elwha LLC (Intellectual Ventures LLC)
• Inventors: Lord, Richard T., Lord, Robert W., Myhrvold, Nathan P., Tegreene, Clarence T., Hyde, Roderick A., Wood, Lowell L. Jr., Ishikawa, Muriel Y., Wood, Victoria Y. H., Whitmer, Charles, Bahl, Paramvir, Burger, Douglas C., Chandra, Ranveer, Gates, William H. III, Holman, Paul, Kare, Jordin T., Mundie, Craig J., Paek, Tim, Tan, Desney S., Zhong, Lin, Dyor, Matthew G.
• Primary Examiner(s): Kazeminezhad, Farzad
• Application Number:
  US13/397,289
• Publication Number:
  US 20130144603A1
• Time in Patent Office: 1,441 Days
• Field of Search: 704/9, 704/235, 715/716
• US Class Current: 1/1
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G10L 13/02 Methods for producing synthetic speech; Speech synthesisers
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: H04L 12/1831 Tracking arrangements for later retrieval, e.g. recording contents, participants activities or behavior, network status
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 67/306 User profiles
• CPC Class Codes: H04M 2201/405 involving speaker-dependent recognition adaptation to speaker for speech recognition G10L15/07
• CPC Class Codes: H04M 2201/41 using speaker recognition speaker recognition per se G10L17/00
• CPC Class Codes: H04M 3/42221 Conversation recording systems at the subscriber's set H04M1/656
• CPC Class Codes: H04M 3/56 Arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities video conference systems H04N7/15
• CPC Class Codes: H04N 21/47 End-user applications

Abstract

Techniques for ability enhancement are described. Some embodiments provide an ability enhancement facilitator system (“AEFS”) configured to enhance voice conferencing among multiple speakers. Some embodiments of the AEFS enhance voice conferencing by recording, translating and presenting voice conference history information based on speaker-related information, wherein the translation is based on language identification using multiple speech recognizers and GPS information. The AEFS receives data that represents utterances of multiple speakers who are engaging in a voice conference with one another. The AEFS then determines speaker-related information, such as by identifying a current speaker, locating an information item (e.g., an email message, document) associated with the speaker, or the like. The AEFS records conference history information (e.g., a transcript) based on the determined speaker-related information. The AEFS then informs a user of the conference history information, such as by presenting a transcript of the voice conference and/or related information items on a display of a conferencing device associated with the user.

Claims

1. A method for ability enhancement, the method comprising:

by a computer system,

receiving data representing speech signals from a voice conference amongst multiple speakers, wherein the multiple speakers are remotely located from one another, wherein each of the multiple speakers uses a separate conferencing device to participate in the voice conference;

determining speaker-related information associated with the multiple speakers, based on the data representing speech signals from the voice conference;

recording conference history information based on the speaker-related information, by recording indications of topics discussed during the voice conference by;

performing speech recognition to convert the data representing speech signals into text;

analyzing the text to identify frequently used terms or phrases; and

determining the topics discussed during the voice conference based on the frequently used terms or phrases;

audibly notifying a user to view the conference history information on a display device, wherein the user is notified in a manner that is not audible to at least some of the multiple speakers; and

presenting, on the display device, at least some of the conference history information to the user;

translating an utterance of one of the multiple speakers in a first language into a message in a second language, based on the speaker-related information,

wherein the speaker related information is determined by automatically determining the second and the first language comprising steps of;

concurrently or simultaneously applying multiple speech recognizers and using GPS information indicating the speakers'"'"' locations; and

recording the message in the second language as part of the conference history information.

2. The method of claim 1, wherein the recording conference history information based on the speaker-related information includes:

recording a transcription of utterances made by speakers during the voice conference.

3. The method of claim 2, wherein the recording a transcription includes:

performing speech recognition to convert data representing a speech signal from one of the multiple speakers into text; and

storing the text in association with an indicator of the one speaker.

4. The method of claim 1, further comprising:

performing voice identification based on the data representing the speech signals from the voice conference.

5. The method of claim 4, wherein the performing voice identification includes:

in a conference call system, matching a portion of the data representing the speech signals with an identity of one of the multiple speakers, based on a communication channel that is associated with the one speaker and over which the portion of the data is transmitted.

6. The method of claim 4, the semantic content including a name, event, or entity mentioned by a speaker.

7. The method of claim 6, further comprising:

processing voice messages from the multiple persons to generate voice print data for each of the multiple persons, wherein each of the voice messages is a telephone voice mail message stored by a voice mail service in association with a sender telephone number; and

performing reverse directory lookups using the sender telephone numbers to determine names of speakers associated with the voice messages.

8. The method of claim 1, wherein the recording conference history information based on the speaker-related information includes:

recording indications of information items related to subject matter of the voice conference.

9. The method of claim 8, wherein the recording indications of information items related to subject matter of the voice conference includes:

performing speech recognition to convert the data representing speech signals into text; and

analyzing the text to identify information items mentioned by the speakers.

10. The method of claim 1, wherein the presenting at least some of the conference history information includes:

presenting the conference history information to a participant in the voice conference, the participant having rejoined the voice conference after having not participated in the voice conference for a period of time.

11. The method of claim 10, wherein the participant rejoins the voice conference after at least one of:

pausing the voice conference, muting the voice conference, holding the voice conference, voluntarily leaving the voice conference, and/or involuntarily leaving the voice conference.

12. The method of claim 1, wherein the recording conference history information based on the speaker-related information includes:

recording the data representing speech signals from the voice conference.

13. The method of claim 1, wherein the recording conference history information based on the speaker-related information includes:

as each of the multiple speakers takes a turn speaking during the voice conference, recording speaker-related information associated with the speaker.

14. The method of claim 1, wherein the recording conference history information based on the speaker-related information includes:

recording conference history information based on the speaker-related information during a telephone conference call amongst the multiple speakers.

15. The method of claim 1, wherein the presenting at least some of the conference history information includes:

presenting the conference history information to a new participant in the voice conference, the new participant having joined the voice conference while the voice conference was already in progress.

16. The method of claim 1, wherein the presenting at least some of the conference history information includes:

presenting the conference history information to a user after conclusion of the voice conference.

17. The method of claim 1, wherein the presenting at least some of the conference history information includes:

providing a user interface configured to access the conference history information by scrolling through a temporal record of the voice conference.

18. The method of claim 1, wherein the presenting at least some of the conference history information includes:

presenting a transcription of utterances made by speakers during the voice conference.

19. The method of claim 1, wherein the presenting at least some of the conference history information includes:

presenting indications of topics discussed during the voice conference.

20. The method of claim 1, wherein the presenting at least some of the conference history information includes:

presenting indications of information items related to subject matter of the voice conference.

21. The method of claim 1, wherein the presenting at least some of the conference history information includes:

presenting, while a current speaker is speaking, conference history information on a display device of the user, the displayed conference history information providing information related to previous statements made by the current speaker.

22. The method of claim 1, further comprising:

retrieving information items that reference the text data; and

informing the user of the retrieved information items.

23. The method of claim 1, wherein the performing speech recognition includes:

performing speech recognition based at least in part on a language model associated with the one speaker, wherein the language model represents word transition likelihoods; and

generating the language model based on information items generated by or referencing any of the multiple speakers, the information items including emails, documents, and/or social network messages.

24. The method of claim 1, wherein the determining speaker-related information associated with the multiple speakers includes:

determining which one of the multiple speakers is speaking during a time interval.

25. The method of claim 1, wherein the determining speaker-related information associated with the multiple speakers includes:

developing a corpus of speaker data by recording speech from multiple persons;

generating a speech model associated with each of the multiple persons, based on the recorded speech;

determining the speaker-related information based at least in part on the corpus of speaker data;

receiving feedback regarding accuracy of the conference history information; and

training a speech processor based at least in part on the received feedback.

26. The method of claim 1, wherein the presenting at least some of the conference history information includes:

presenting the conference history information on a display of a conferencing device of the user.

27. The method of claim 1, wherein audibly notifying the user to view the conference history information on a display device includes:

playing, via an earpiece speaker of the user, synthesized speech telling the user that a document is available for viewing on the display device, such that other parties to the conference do not hear the notification.

28. The method of claim 1, wherein the presenting at least some of the conference history information includes all of:

informing the user of an identifier of each of the multiple speakers;

informing the user of an identifier of a speaker along with a transcription of a previous utterance made by the speaker;

informing the user of an organization to which each of the multiple speakers belongs;

informing the user of a previously transmitted communication referencing one of the multiple speakers; and

informing the user of an event involving the user and one of the multiple speakers.

29. The method of claim 1, wherein the determining speaker-related information associated with the multiple speakers includes:

accessing information items associated with one of the multiple speakers, the accessing including all of;

searching for information items that reference the one speaker, the information items including at least one of a document, an email, and/or a text message;

accessing a social networking service to find messages or status updates that reference the one speaker;

accessing a calendar to find information about appointments with the one speaker; and

accessing a document store to find documents that reference the one speaker.

30. The method of claim 1, wherein the receiving data representing speech signals from a voice conference amongst multiple speakers includes:

receiving audio data from at least one of a telephone, a conference call, an online audio chat, a video conference, and/or a face-to-face conference that includes the multiple speakers, the received audio data representing utterances made by at least one of the multiple speakers.

31. The method of claim 1, wherein the presenting at least some of the conference history information includes:

transmitting the conference history information from a first device to a second device having a display.

32. The method of claim 1, further comprising:

performing the receiving data representing speech signals from a voice conference amongst multiple speakers, the determining speaker-related information associated with the multiple speakers, the recording conference history information based on the speaker-related information, and/or the presenting at least some of the conference history information on a mobile device that is operated by the user.

33. The method of claim 1, further comprising:

performing the receiving data representing speech signals from a voice conference amongst multiple speakers, the determining speaker-related information associated with the multiple speakers, the recording conference history information based on the speaker-related information, and/or the presenting at least some of the conference history information on a general purpose computing device that is operated by the user.

34. The method of claim 1, further comprising:

performing one or more of the receiving data representing speech signals from a voice conference amongst multiple speakers, the determining speaker-related information associated with the multiple speakers, the recording conference history information based on the speaker-related information, and/or the presenting at least some of the conference history information on each of multiple computing systems, wherein each of the multiple systems is associated with one of the multiple speakers.

35. The method of claim 1, further comprising:

performing one or more of the receiving data representing speech signals from a voice conference amongst multiple speakers, the determining speaker-related information associated with the multiple speakers, the recording conference history information based on the speaker-related information, and/or the presenting at least some of the conference history information within a conference call provider system.

36. The method of claim 1, further comprising:

determining to perform at least some of the receiving data representing speech signals from a voice conference amongst multiple speakers, the determining speaker-related information associated with the multiple speakers, the recording conference history information based on the speaker-related information, and/or the presenting at least some of the conference history information on another computing device that has available processing capacity.

37. The method of claim 1, further comprising:

selecting a portion of the conference history information based on capabilities of a device operated by the user; and

transmitting the selected portion for presentation on the device operated by the user.

38. The method of claim 1, further comprising:

performing speech recognition to convert an utterance of one of the multiple speakers into text, the speech recognition performed at a mobile device of the one speaker; and

transmitting the text along with an audio representation of the utterance and an identifier of the speaker to a remote conferencing device and/or a conference call system.

39. The method of claim 1, wherein the user is not one of the multiple speakers.

40. The method of claim 1, wherein the speaker is not a human.

41. The method of claim 1, further comprising:

determining to perform one or more of archiving, indexing, searching, removing, redacting, duplicating, or deleting some of the conference history information based on a data retention policy.

42. A non-transitory computer-readable medium having contents that are configured, when executed, to cause a computing system to perform a method for ability enhancement, the method comprising:

by the computer system,

receiving data representing speech signals from a voice conference amongst multiple speakers, wherein the multiple speakers are remotely located from one another, wherein each of the multiple speakers uses a separate conferencing device to participate in the voice conference;

determining speaker-related information associated with the multiple speakers, based on the data representing speech signals from the voice conference;

recording conference history information based on the speaker-related information, by recording indications of topics discussed during the voice conference by;

performing speech recognition to convert the data representing speech signals into text;

analyzing the text to identify frequently used terms or phrases; and

determining the topics discussed during the voice conference based on the frequently used terms or phrases;

audibly notifying a user to view the conference history information on a display device,

wherein the user is notified in a manner that is not audible to at least some of the multiple speakers; and

presenting, on the display device, at least some of the conference history information to the user;

translating an utterance of one of the multiple speakers in a first language into a message in a second language, based on the speaker-related information,

wherein the speaker related information is determined by automatically determining the second and the first language comprising steps of;

concurrently or simultaneously applying multiple speech recognizers and using GPS information indicating the speakers'"'"' locations; and

recording the message in the second language as part of the conference history information.

43. A computing system for ability enhancement, the computing system comprising:

a processor;

a memory; and

a module that is stored in the memory and that is configured, when executed by the processor, to perform a method comprising;

by the computer system,

receiving data representing speech signals from a voice conference amongst multiple speakers, wherein the multiple speakers are remotely located from one another, wherein each of the multiple speakers uses a separate conferencing device to participate in the voice conference;

determining speaker-related information associated with the multiple speakers, based on the data representing speech signals from the voice conference;

recording conference history information based on the speaker-related information, by recording indications of topics discussed during the voice conference by;

performing speech recognition to convert the data representing speech signals into text;

analyzing the text to identify frequently used terms or phrases; and

determining the topics discussed during the voice conference based on the frequently used terms or phrases;

audibly notifying a user to view the conference history information on a display device, wherein the user is notified in a manner that is not audible to at least some of the multiple speakers; and

presenting, on the display device, at least some of the conference history information to the user;

translating an utterance of one of the multiple speakers in a first language into a message in a second language, based on the speaker-related information,

wherein the speaker related information is determined by automatically determining the second and the first language comprising steps of;

concurrently or simultaneously applying multiple speech recognizers and using GPS information indicating the speakers'"'"' locations; and

recording the message in the second language as part of the conference history information.