Methods, apparatus, and products for automatically managing conversational floors in computer-mediated communications

• US 20040172255A1
• Filed: 04/16/2003
• Published: 09/02/2004
• Est. Priority Date: 02/28/2003
• Status: Active Grant

• Current Assignee: III Holdings 6, LLC
• Original Assignee: Palo Alto Research Center, Inc. (Xerox Holdings Corp.)
• Inventors: Thornton, James D., Woodruff, Allison G., Szymanski, Margaret H., Aoki, Paul M., Wilson, Daniel H.
• Granted Patent: US 7,698,141 B2
• Time in Patent Office: Days
• US Class Current: 704/275
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: H04M 3/564 whereby the feature is a sub-conference
• CPC Class Codes: H04M 3/566 relating to a participants right to speak arrangements for multi-party communication with floor control, e.g. for conferences, H04L65/4038, H04L65/4046, H04L65/4053
• CPC Class Codes: H04M 3/568 audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30
• CPC Class Codes: H04M 3/569 using the instant speaker's algorithm speech detection per se G10L25/78

Abstract

One embodiment of the invention is a computer controlled method for use with a communication system. The method includes a step of receiving a plurality of communications, where each one of the plurality of communications is from one of a plurality of communication sources; includes a step of mixing (that is responsive to a plurality of floor controls) the plurality of communications for a plurality of outputs associated with plurality of communication sources; and includes a step of analyzing, for a plurality of users associated with the plurality of communication sources, one or more conversational characteristics of two or more of the plurality of users. The method also includes a step of automatically adjusting the plurality of floor controls responsive to the step of analyzing. Other embodiments include systems and devices that use the method as well as program products that cause a computer to execute the method.

Claims

1. A computer controlled method for an audio communication system, said method comprising steps of:

receiving a plurality of audio streams, where each one of said plurality of audio streams is from one of a plurality of audio sources;

mixing, responsive to a plurality of floor controls, said plurality of audio streams for a plurality of outputs associated with said plurality of audio sources;

analyzing, for a plurality of users associated with said plurality of audio sources, one or more conversational characteristics of two or more of said plurality of users; and

automatically adjusting said plurality of floor controls responsive to the step of analyzing.

2. The computer controlled method of claim 1, wherein said plurality of audio streams includes a first audio stream and a second audio stream, corresponding to a first source and a second source respectively, from said plurality of audio sources, said first source and said second source corresponding with a first user and a second user respectively, wherein the step of analyzing comprises steps of:

temporally aligning a portion of said first audio stream with a portion of said second audio stream;

determining a feature value by comparing said portion of said first audio stream with said portion of said second audio stream; and

determining a probability that said first user and said second user are members of a floor.

3. The computer controlled method of claim 1, wherein said plurality of audio streams includes a first audio stream and a second audio stream, corresponding to a first source and a second source respectively, from said plurality of audio sources, said first source and said second source corresponding with a first user and a second user respectively, wherein the step of analyzing further comprises determining a probability that said first user has made a response to said second user.

4. The computer controlled method of claim 3, wherein the step of determining said probability further comprises steps of:

determining whether said first user has started a vocalization;

determining whether said second user vocalized within a vocalization window; and

incorporating within said probability that said vocalization is a backchannel word.

5. The computer controlled method of claim 1, wherein said plurality of audio streams includes a first audio stream and a second audio stream, corresponding to a first source and a second source respectively, from said plurality of audio sources, said first source and said second source corresponding with a first user and a second user respectively, wherein the step of analyzing further comprises determining a probability that said first audio stream and said second audio stream share a common element.

6. The computer controlled method of claim 5, wherein the step of determining said probability further comprises steps of:

creating a first temporal energy profile from a portion of said first audio stream;

creating a second temporal energy profile from a portion of said second audio stream;

comparing said first temporal energy profile with said second temporal energy profile for said common element; and

incorporating within said probability the result of the step of comparing.

7. The computer controlled method of claim 5, wherein the step of determining said probability further comprises steps of:

generating a first text using speech recognition on a portion of said first audio stream;

generating a second text using speech recognition on a portion of said second audio stream;

comparing said first text and said second text for said common element; and

incorporating within said probability the result of the step of comparing.

8. The computer controlled method of claim 5, wherein the step of determining said probability further comprises steps of:

recognizing a first set of phonemes from said first audio stream;

determining a first phoneme distribution from said first set of phonemes;

recognizing a second set of phonemes from said second audio stream;

determining a second phoneme distribution from said second set of phonemes;

comparing said first phoneme distribution with said second phoneme distribution for said common element; and

incorporating within said probability the result of the step of comparing.

9. The computer controlled method of claim 5, wherein the step of determining said probability further comprises steps of:

generating a first set of prosodic profiles from said first audio stream;

generating a second set of prosodic profiles from said second audio stream;

comparing said first set of prosodic profiles and said second set of prosodic profiles for said common element; and

incorporating within said probability the result of the step of comparing.

10. The computer controlled method of claim 1, wherein said plurality of audio streams includes a first audio stream corresponding to a first source, from said plurality of audio sources, said first source corresponding with a first user, wherein the step of analyzing further comprises determining a probability that said first user has made a reference to a second user.

11. The computer controlled method of claim 10, wherein the step of determining said probability further comprises steps of:

determining whether said first user has started a vocalization within a vocalization window; and

incorporating within said probability that said vocalization contains a name variant of said second user.

12. The computer controlled method of claim 10, wherein the step of determining said probability further comprises detecting a change of volume of said vocalization.

13. The computer controlled method of claim 1, wherein the step of analyzing further comprises steps of:

receiving information from a physiological measurement device configured to monitor one of said plurality of users; and

correlating said information with one or more of said plurality of audio streams.

14. The computer controlled method of claim 1, wherein the step of analyzing establishes at least one floor among said plurality of users and where the method further comprises indicating said at least one floor.

15. The computer controlled method of claim 14, wherein the step of indicating further comprises visually indicating which of said plurality of users are assigned to said at least one floor.

16. The computer controlled method of claim 14, wherein the step of indicating further comprises audibly indicating which of said plurality of users are not assigned to said at least one floor.

17. A computer controlled method for computer mediated communication comprising steps of:

receiving a plurality of communication streams, where each one of said plurality of communication streams is from one of a plurality of communication sources;

mixing, responsive to a plurality of floor controls, said plurality of communication streams for a plurality of outputs associated with said plurality of communication sources;

analyzing, an action by a first user to determine whether said action is responsive to an action by a second user;

wherein said first user and said second user are associated with different of said plurality of communication sources; and

automatically adjusting said plurality of floor controls responsive to the step of analyzing.

18. The computer controlled method of claim 17, wherein said action by said first user is one of a backchannel communication or a communication having similar content to that of said action of said second user.

19. A computer controlled method for computer mediated communication comprising steps of:

receiving a plurality of communication streams, where each one of said plurality of communication streams is from one of a plurality of communication sources;

mixing, responsive to a plurality of floor controls, said plurality of communication streams for a plurality of outputs associated with said plurality of communication sources;

analyzing, an action by a first user to determine whether said action refers to a second user; and

automatically adjusting said plurality of floor controls responsive to the step of analyzing.

20. The computer controlled method of claim 19, wherein said action by said first user is a reference to an identifier of said second user.

21. An apparatus for an audio communication system, said apparatus comprises:

a receiver configured to receive a plurality of audio streams, where each one of said plurality of audio streams is from one of a plurality of audio sources;

a mixer configured to mix, responsive to a plurality of floor controls, said plurality of audio streams received from the receiver for a plurality of outputs associated with said plurality of audio sources;

an analysis mechanism configured to analyze, for a plurality of users associated with said plurality of audio sources, one or more conversational characteristics of two or more of said plurality of users; and

a floor control mechanism configured to automatically adjust said plurality of floor controls responsive to the analysis mechanism.

22. The apparatus of claim 21, wherein said plurality of audio streams includes a first audio stream and a second audio stream, corresponding to a first source and a second source respectively, from said plurality of audio sources, said first source and said second source corresponding with a first user and a second user respectively, wherein the analysis mechanism further comprises:

a synchronization mechanism configured to temporally align a portion of said first audio stream with a portion of said second audio stream;

a feature determination mechanism configured to determine a feature value by comparing said portion of said first audio stream with said portion of said second audio stream as aligned by the synchronization mechanism; and

an evaluation mechanism, responsive to the feature determination mechanism, configured to determine a probability that said first user and said second user are members of a floor.

23. The apparatus of claim 21, wherein said plurality of audio streams includes a first audio stream and a second audio stream, corresponding to a first source and a second source respectively, from said plurality of audio sources, said first source and said second source corresponding with a first user and a second user respectively, wherein the analysis mechanism further comprises an evaluation mechanism configured to determine a probability that said first user has made a response to said second user.

24. The apparatus of claim 21, wherein said plurality of audio streams includes a first audio stream and a second audio stream, corresponding to a first source and a second source respectively, from said plurality of audio sources, said first source and said second source corresponding with a first user and a second user respectively, wherein the analysis mechanism further comprises an evaluation mechanism configured to determine a probability that said first audio stream and said second audio stream share a common element.

25. The apparatus of claim 21, wherein said plurality of audio streams includes a first audio stream corresponding to a first source, from said plurality of audio sources, said first source corresponding with a first user, wherein the analysis mechanism further comprises a referential determination mechanism configured to determine a probability that said first user has made a reference to a second user.

26. The apparatus of claim 21, wherein the analysis mechanism further comprises:

a biometric receiver configured to receive information from a physiological measurement device configured to monitor one of said plurality of users; and

a correlation mechanism configured to correlate said information with one or more of said plurality of audio streams.

27. An apparatus for computer mediated communication comprising:

a receiver configured to receive a plurality of communication streams, where each one of said plurality of communication streams is from one of a plurality of communication sources;

a mixer configured to mix, responsive to a plurality of floor controls, said plurality of communication streams for a plurality of outputs associated with said plurality of communication sources;

an analysis mechanism configured to analyze, an action by a first user to determine whether said action is responsive to an action by a second user;

wherein said first user and said second user are associated with different of said plurality of communication sources; and

a floor control mechanism configured to automatically adjust said plurality of floor controls responsive to the analysis mechanism.

28. The apparatus of claim 27, wherein said action by said first user is one of a backchannel communication or a communication having similar content to that of said action of said second user.

29. An apparatus for computer mediated communication comprising:

a receiver configured to receive a plurality of communication streams, where each one of said plurality of communication streams is from one of a plurality of communication sources;

a mixer configured to mix, responsive to a plurality of floor controls, said plurality of communication streams for a plurality of outputs associated with said plurality of communication sources;

an analysis mechanism configured to analyze, an action by a first user to determine whether said action refers to a second user; and

a floor control mechanism configured to automatically adjust said plurality of floor controls responsive to the analysis mechanism.

30. The apparatus of claim 29, wherein said action by said first user is a reference to an identifier of said second user.

31. A computer program product comprising:

a computer usable data carrier having computer readable code embodied therein for causing a computer to provide a group communication service, the computer program product comprising;

computer readable program code configured to cause said computer to effect a receiver configured to receive a plurality of audio streams, where each one of said plurality of audio streams is from one of a plurality of audio sources;

computer readable program code configured to cause said computer to effect a mixer configured to mix, responsive to a plurality of floor controls, said plurality of audio streams received from the receiver for a plurality of outputs associated with said plurality of audio sources;

computer readable program code configured to cause said computer to effect an analysis mechanism configured to analyze, for a plurality of users associated with said plurality of audio sources, one or more conversational characteristics of two or more of said plurality of users; and

computer readable program code configured to cause said computer to effect a floor control mechanism configured to automatically adjust said plurality of floor controls responsive to the analysis mechanism.

32. A computer program product comprising:

computer readable program code configured to cause said computer to effect a computer usable data carrier having computer readable code embodied therein for causing a computer to provide a group communication service, the computer program product comprising;

computer readable program code configured to cause said computer to effect a receiver configured to receive a plurality of communication streams, where each one of said plurality of communication streams is from one of a plurality of communication sources;

computer readable program code configured to cause said computer to effect a mixer configured to mix, responsive to a plurality of floor controls, said plurality of communication streams for a plurality of outputs associated with said plurality of communication sources;

computer readable program code configured to cause said computer to effect an analysis mechanism configured to analyze, an action by a first user to determine whether said action is responsive to an action by a second user;

wherein said first user and said second user are associated with different of said plurality of communication sources; and

computer readable program code configured to cause said computer to effect a floor control mechanism configured to automatically adjust said plurality of floor controls responsive to the analysis mechanism.

33. A computer program product comprising:

a computer usable data carrier having computer readable code embodied therein for causing a computer to provide a group communication service, the computer program product comprising;

computer readable program code configured to cause said computer to effect a receiver configured to receive a plurality of communication streams, where each one of said plurality of communication streams is from one of a plurality of communication sources;

computer readable program code configured to cause said computer to effect a mixer configured to mix, responsive to a plurality of floor controls, said plurality of communication streams for a plurality of outputs associated with said plurality of communication sources;

computer readable program code configured to cause said computer to effect an analysis mechanism configured to analyze, an action by a first user to determine whether said action refers to a second user; and

computer readable program code configured to cause said computer to effect a floor control mechanism configured to automatically adjust said plurality of floor controls responsive to the analysis mechanism.