Multimodal teleconferencing

• US 8,416,714 B2
• Filed: 08/05/2009
• Issued: 04/09/2013
• Est. Priority Date: 08/05/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Agapi, Ciprian, Bodin, William K., Cross, Charles W. Jr.
• Primary Examiner(s): Ton, Dang T
• Assistant Examiner(s): LEE, KEVIN H
• Application Number:
  US12/535,923
• Publication Number:
  US 20110032845A1
• Time in Patent Office: 1,343 Days
• Field of Search: 370259-261
• US Class Current: 370/260
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: H04L 12/413 with random access, e.g. carrier-sense multiple-access with collision detection [CSMA-CD]

Abstract

Multimodal teleconferencing including receiving, by a multimodal teleconferencing module, a speech utterance from one of a plurality of participants in the multimodal teleconference; identifying the participant making the speech utterance as a current speaker; retrieving, by the multimodal teleconferencing module from accounts for the current speaker, content for display to the current speaker; retrieving, by the multimodal teleconferencing module from accounts for the current speaker, content for display to one or more other participants in the multimodal teleconference; providing, by the multimodal teleconferencing module to a multimodal teleconferencing client for display to the current speaker, an identification of the speaker and the content retrieved for the speaker; and providing, by the multimodal teleconferencing module to one or more of multimodal teleconferencing clients for display to the other participants, an identification of the current speaker with the content retrieved for the one or more other participants in the multimodal teleconference.

Claims

1. A method of multimodal teleconferencing, the method implemented with a multimodal application and a multimodal teleconferencing module, a module of automated computing machinery operating on a server supporting a voice mode including accepting speech input from participants of the multimodal teleconference through a telephony interface, digitizing the speech, and providing digitized speech to a speech engine, the method comprising:

receiving, by the multimodal teleconferencing module, a speech utterance from one of a plurality of participants in the multimodal teleconference;

identifying, by the multimodal teleconferencing module, the participant making the speech utterance as a current speaker;

retrieving, by the multimodal teleconferencing module from accounts for the current speaker, first content for display to the current speaker;

retrieving, by the multimodal teleconferencing module from accounts for the current speaker, second content for display to one or more other participants in the multimodal teleconference, wherein at least a part of the first content is different from the second content;

providing, by the multimodal teleconferencing module to a multimodal teleconferencing client for display to the current speaker, an identification of the speaker and the first content retrieved for the speaker;

providing, by the multimodal teleconferencing module to one or more of multimodal teleconferencing clients for display to the other participants, an identification of the current speaker with the second content retrieved for the one or more other participants in the multimodal teleconferences;

receiving, by the multimodal teleconferencing client of listening participant other than the current speaker, identification of another particular participant of interest;

requesting, by the multimodal teleconferencing client from the multimodal teleconferencing module;

third content of the another particular participant of interest;

retrieving, by the multimodal teleconferencing module from accounts for the another particular participant of interest, third content of the another particular participant of interest;

providing, by the multimodal teleconferencing module to a multimodal teleconferencing client for display to the listening participant, the third content of the another particular participant of interest; and

displaying, by the multimodal teleconferencing client to the listening participant, the third content of the another particular participant of interest.

2. The method of claim 1 further comprising:

receiving, by the multimodal teleconferencing module, speech from the current speaker;

creating, by the multimodal teleconferencing module, text from the speech; and

providing, by the multimodal teleconferencing module to the multimodal teleconferencing clients for all participants in the multimodal teleconference, the text as a transcript of the multimodal teleconference.

3. The method of claim 1 wherein identifying, by the multimodal teleconferencing module, the participant making the speech utterance as a current speaker further comprises:

receiving speech from the current speaker; and

determining whether the speech matches the voiceprint of a participant of the multimodal teleconference.

4. The method of claim 1 wherein identifying, by the multimodal teleconferencing module, the participant making the speech utterance as a current speaker further comprises:

receiving speech from the current speaker;

identifying the telephone number of the current speaker; and

determining whether the phone number of the current speaker matches a known number of a participant of the multimodal teleconference.

5. The method of claim 1, wherein receiving, by the multimodal teleconferencing client of listening participant other than the current speaker, the identification of another particular participant of interest comprises receiving, by the multimodal teleconferencing client of listening participant other than the current speaker, a speech utterance identifying the another particular participant of interest.

6. The method of claim 1, wherein receiving, by the multimodal teleconferencing client of listening participant other than the current speaker the identification of the another particular participant of interest comprises receiving, by the multimodal teleconferencing client of listening participant other than the current speaker, through a non voice mode user input device, the identification of the another particular participant of interest.

7. The method of claim 1, wherein the first content comprises at least one of confidential material and speaker notes that are defined as not to be viewed by the one or more participants that are not the current speaker.

8. An apparatus of multimodal teleconferencing, the apparatus including a multimodal application and a multimodal teleconferencing module, a module of automated computing machinery operating on a server supporting a voice mode including accepting speech input from participants of the multimodal teleconference through a telephony interface, digitizing the speech, and providing digitized speech to a speech engine, the apparatus comprising a computer memory operatively coupled to a computer processor, the memory having disposed within it computer program instructions for:

receiving, by the multimodal teleconferencing module, a speech utterance from one of a plurality of participants in the multimodal teleconference;

identifying, by the multimodal teleconferencing module, the participant making the speech utterance as a current speaker;

retrieving, by the multimodal teleconferencing module from accounts for the current speaker, first content for display to the current speaker;

retrieving, by the multimodal teleconferencing module from accounts for the current speaker, second content for display to one or more other participants in the multimodal teleconference, wherein at least a part of the first content is different from the second content;

providing, by the multimodal teleconferencing module to a multimodal teleconferencing client for display to the current speaker, an identification of the speaker and the first content retrieved for the speaker;

providing, by the multimodal teleconferencing module to one or more of multimodal teleconferencing clients for display to the other participants, an identification of the current speaker with the second content retrieved for the one or more other participants in the multimodal teleconferences;

receiving, by the multimodal teleconferencing client of listening participant other than the current speaker, identification of another particular participant of interest;

requesting, by the multimodal teleconferencing client from the multimodal teleconferencing module;

third content of the another particular participant of interest;

retrieving, by the multimodal teleconferencing module from accounts for the another particular participant of interest, third content of the another particular participant of interest;

providing, by the multimodal teleconferencing module to a multimodal teleconferencing client for display to the listening participant, the third content of the another particular participant of interest; and

displaying, by the multimodal teleconferencing client to the listening participant, the third content of the another particular participant of interest.

9. The apparatus of claim 8 wherein the computer memory also has disposed within it computer program instructions for:

receiving, by the multimodal teleconferencing module, speech from the current speaker;

creating, by the multimodal teleconferencing module, text from the speech; and

providing, by the multimodal teleconferencing module to the multimodal teleconferencing clients for all participants in the multimodal teleconference, the text as a transcript of the multimodal teleconference.

10. The apparatus of claim 8 wherein computer program instructions for identifying, by the multimodal teleconferencing module, the participant making the speech utterance as a current speaker further comprise computer program instructions for:

receiving speech from the current speaker; and

determining whether the speech matches the voiceprint of a participant of the multimodal teleconference.

11. The apparatus of claim 8 wherein computer program instructions for identifying, by the multimodal teleconferencing module, the participant making the speech utterance as a current speaker further comprise computer program instructions for:

receiving speech from the current speaker;

identifying the telephone number of the current speaker; and

determining whether the phone number of the current speaker matches a known number of a participant of the multimodal teleconference.

12. The apparatus of claim 8 wherein computer program instructions for retrieving by the multimodal teleconferencing module from accounts for the current speaker, the first content for display to the current speaker include computer program instructions for retrieving email for the current speaker concerning the multimodal teleconference, biographical information about the current speaker, and documents of the current speaker related to the multimodal teleconference.

13. The apparatus of claim 8 wherein computer program instructions for retrieving by the multimodal teleconferencing module from accounts for the current speaker, the second content for display to one or more other participants in the multimodal teleconference include computer program instructions for retrieving email for the current speaker concerning the multimodal teleconference, biographical information about the current speaker, and documents of the current speaker related to the multimodal teleconference.

14. The apparatus of claim 8, wherein the first content comprises at least one of confidential material and speaker notes that are defined as not to be viewed by the one or more participants that are not the current speaker.

15. A computer program product of multimodal teleconferencing, the computer program product including a module of automated computing machinery operating on a server supporting a voice mode including accepting speech input from participants of the multimodal teleconference through a telephony interface, digitizing the speech, and providing digitized speech to a speech engine, the computer program product disposed on a non-transitory recording medium, the computer program product comprising computer program instructions for:

receiving, by the multimodal teleconferencing module, a speech utterance from one of a plurality of participants in the multimodal teleconference;

identifying, by the multimodal teleconferencing module, the participant making the speech utterance as a current speaker;

retrieving, by the multimodal teleconferencing module from accounts for the current speaker, first content for display to the current speaker;

retrieving, by the multimodal teleconferencing module from accounts for the current speaker, second content for display to one or more other participants in the multimodal teleconference, wherein at least a part of the first content is different from the second content;

providing, by the multimodal teleconferencing module to a multimodal teleconferencing client for display to the current speaker, an identification of the speaker and the first content retrieved for the speaker;

providing, by the multimodal teleconferencing module to one or more of multimodal teleconferencing clients for display to the other participants, an identification of the current speaker with the second content retrieved for the one or more other participants in the multimodal teleconference;

receiving, by the multimodal teleconferencing client of listening participant other than the current speaker, identification of another particular participant of interest;

requesting, by the multimodal teleconferencing client from the multimodal teleconferencing module;

third content of the another particular participant of interest;

retrieving, by the multimodal teleconferencing module from accounts for the another particular participant of interest, third content of the another particular participant of interest;

providing, by the multimodal teleconferencing module to a multimodal teleconferencing client for display to the listening participant, the third content of the another particular participant of interest; and

displaying, by the multimodal teleconferencing client to the listening participant, the third content of the another particular participant of interest.

16. The computer program product of claim 15 wherein the computer memory also has disposed within it computer program instructions for:

receiving, by the multimodal teleconferencing module, speech from the current speaker;

creating, by the multimodal teleconferencing module, text from the speech; and

providing, by the multimodal teleconferencing module to the multimodal teleconferencing clients for all participants in the multimodal teleconference, the text as a transcript of the multimodal teleconference.

17. The computer program product of claim 15 wherein computer program instructions for identifying, by the multimodal teleconferencing module, the participant making the speech utterance as a current speaker further comprise computer program instructions for:

receiving speech from the current speaker; and

determining whether the speech matches the voiceprint of a participant of the multimodal teleconference.

18. The computer program product of claim 15 wherein computer program instructions for identifying, by the multimodal teleconferencing module, the participant making the speech utterance as a current speaker further comprise computer program instructions for:

receiving speech from the current speaker;

identifying the telephone number of the current speaker; and

determining whether the phone number of the current speaker matches known number of a participant of the multimodal teleconference.

19. The computer program product of claim 15 wherein computer program instructions for retrieving by the multimodal teleconferencing module from accounts for the current speaker, the first content for display to the current speaker include computer program instructions for retrieving email for the current speaker concerning the multimodal teleconference, biographical information about the current speaker, and documents of the current speaker related to the multimodal teleconference.

20. The computer program product of claim 15 wherein computer program instructions for retrieving by the multimodal teleconferencing module from accounts for the current speaker, the second content for display to one or more other participants in the multimodal teleconference include computer program instructions for retrieving email for the current speaker concerning the multimodal teleconference, biographical information about the current speaker, and documents of the current speaker related to the multimodal teleconference.

21. The computer program product of claim 15, wherein the first content comprises at least one of confidential material and speaker notes that are defined as not to be viewed by the one or more participants that are not the current speaker.