Methods, apparatus, and products for automatically managing conversational floors in computer-mediated communications

• US 7,698,141 B2
• Filed: 04/16/2003
• Issued: 04/13/2010
• Est. Priority Date: 02/28/2003
• Status: Active Grant

• Current Assignee: III Holdings 6, LLC
• Original Assignee: Palo Alto Research Center, Inc. (Xerox Holdings Corp.)
• Inventors: Aoki, Paul M., Thornton, James D., Woodruff, Allison G., Szymanski, Margaret H., Wilson, Daniel H.
• Primary Examiner(s): Hudspeth, David R
• Assistant Examiner(s): Serrou, Abdelali
• Application Number:
  US10/414,923
• Publication Number:
  US 20040172255A1
• Time in Patent Office: 2,554 Days
• Field of Search: 704/236, 704/246, 704/251, 704270-275, 704/201, 704/206, 704/210, 704/215
• US Class Current: 704/270
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: H04M 3/564 whereby the feature is a sub-conference
• CPC Class Codes: H04M 3/566 relating to a participants right to speak arrangements for multi-party communication with floor control, e.g. for conferences, H04L65/4038, H04L65/4046, H04L65/4053
• CPC Class Codes: H04M 3/568 audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30
• CPC Class Codes: H04M 3/569 using the instant speaker's algorithm speech detection per se G10L25/78

Abstract

One embodiment of the invention is a computer controlled method for use with a communication system. The method includes a step of receiving a plurality of communications, where each one of the plurality of communications is from one of a plurality of communication sources; includes a step of mixing (that is responsive to a plurality of floor controls) the plurality of communications for a plurality of outputs associated with plurality of communication sources; and includes a step of analyzing, for a plurality of users associated with the plurality of communication sources, one or more conversational characteristics of two or more of the plurality of users. The method also includes a step of automatically adjusting the plurality of floor controls responsive to the step of analyzing. Other embodiments include systems and devices that use the method as well as program products that cause a computer to execute the method.

Claims

1. A computer controlled method for an audio communication system, comprising:

receiving a plurality of audio streams, which each originates from one of a plurality of audio sources;

mixing, as controlled by a plurality of floor controls, said audio streams into a plurality of outputs that are each associated with said audio sources;

establishing a conversational floor between two or more users, comprising;

analyzing, for a plurality of users associated with said audio sources, conversational characteristics comprising audio content of said audio streams of said users for different combinations of said users in possible conversation floors;

identifying one or more conversational characteristics audio content shared between the two or more users in the possible conversational floors; and

assigning the two or more users to the conversational floor based on the shared conversational characteristics audio content; and

maintaining the conversational floor by automatically adjusting said floor controls based upon the conversational characteristics audio content.

2. The computer controlled method of claim 1, wherein said audio streams comprise a first audio stream and a second audio stream, which correspond to a first audio source and a second audio source that are each associated with a first user and a second user respectively, further comprising:

temporally aligning a portion of said first audio stream with a portion of said second audio stream;

determining a feature value by comparing said portion of said first audio stream with said portion of said second audio stream; and

determining a probability that said first user and said second user are members of the conversational floor.

3. The computer controlled method of claim 1, wherein said audio streams comprise a first audio stream and a second audio stream, which correspond to a first audio source and a second audio source that are each associated with a first user and a second user respectively, further comprising determining a probability that said first user has made a response to said second user.

4. The computer controlled method of claim 3, further comprising:

determining whether said first user has started a vocalization;

determining whether said second user vocalized within a vocalization window; and

incorporating within said probability that said vocalization is a backchannel word.

5. The computer controlled method of claim 1, wherein said audio streams comprise a first audio stream and a second audio stream, which correspond to a first audio source and a second audio source that are each associated with a first user and a second user respectively, further comprising determining a probability that said first audio stream and said second audio stream share a common element.

6. The computer controlled method of claim 5, further comprising:

creating a first temporal energy profile from a portion of said first audio stream;

creating a second temporal energy profile from a portion of said second audio stream;

comparing said first temporal energy profile with said second temporal energy profile for said common element to form a result; and

incorporating the result within said probability.

7. The computer controlled method of claim 5, further comprising:

generating a first text using speech recognition on a portion of said first audio stream;

generating a second text using speech recognition on a portion of said second audio stream;

comparing said first text and said second text for said common element to form a result; and

incorporating the result within said probability.

8. The computer controlled method of claim 5, further comprising:

recognizing a first set of phonemes from said first audio stream;

determining a first phoneme distribution from said first set of phonemes;

recognizing a second set of phonemes from said second audio stream;

determining a second phoneme distribution from said second set of phonemes;

comparing said first phoneme distribution with said second phoneme a distribution for said common element to form a result; and

incorporating the result within said probability.

9. The computer controlled method of claim 5, wherein the step of determining said probability further comprises steps of further comprising:

generating a first set of prosodic profiles from said first audio stream;

generating a second set of prosodic profiles from said second audio stream;

comparing said first set of prosodic profiles and said second set of prosodic profiles for said common element to form a result; and

incorporating the result within said probability the result of the step of comparing.

10. The computer controlled method of claim 1, wherein said audio streams comprise a first audio stream, which corresponds to a first audio source that is associated with a first user, further comprising determining a probability that said first user has made a reference to a second user.

11. The computer controlled method of claim 10, further comprising:

determining whether said first user has started a vocalization within a vocalization window; and

incorporating within said probability that said vocalization contains a name variant of said second user.

12. The computer controlled method of claim 10, further comprising detecting a change of volume of said vocalization.

13. The computer controlled method of claim 1, further comprising:

establishing at least one conversational floor among said users; and

indicating said at least one conversational floor.

14. The computer controlled method of claim 13, further comprising visually indicating which of said users are assigned to said at least one conversational floor.

15. The computer controlled method of claim 13, further comprising audibly indicating which of said users are not assigned to said at least one floor.

30. The computer controlled method of claim 1, wherein the conversational characteristics further comprise physiological characteristics of said users.

16. A computer controlled method for computer mediated communication, comprising:

receiving a plurality of communication streams, which each originates from one of a plurality of communication sources;

mixing, as controlled by a plurality of floor controls, said communication streams into a plurality of outputs that are each associated with said communication sources;

determining a plurality of users associated with the communication streams;

identifying an action comprising a vocalization by a first such user, wherein the first such user is associated with a first of said communication sources;

identifying a different action comprising a different vocalization by a second such user, wherein the second such user is associated with a second of said communication sources;

analyzing one or more conversational characteristics comprising audio content simultaneously shared by the first and second of such users for an indication in the shared audio content based on a probabilistic inference that said vocalization is in response to said different vocalization; and

automatically adjusting said floor controls based upon the indication,

wherein the action is selected from the group comprising backchannel/continuers, common content, prosody, and temporal vocalization adjacency.

17. A computer controlled method for computer mediated communication, comprising:

receiving a plurality of communication streams, which each originates from one of a plurality of communication sources;

mixing, as controlled by a plurality of floor controls, said communication streams into a plurality of outputs that are each associated with said communication sources;

determining a plurality of users associated with the communication streams;

identifying an action comprising a vocalization by a first such user;

analyzing one or more conversational characteristics comprising audio content simultaneously shared by the first and a second of such users for an indication in the shared audio content based on a probabilistic inference that said vocalization refers to the second of such users; and

automatically adjusting said floor controls based upon the indication,

wherein the action is selected from the group comprising backchannel/continuers, common content, prosody, and temporal vocalization adjacency.

18. The computer controlled method of claim 17, wherein said action by the first user is a reference to an identifier of the second user.

19. An apparatus for an audio communication system, comprising:

a receiver configured to receive a plurality of audio streams, which each originates from one of a plurality of audio sources;

a mixer configured to mix, as controlled by a plurality of floor controls, said audio streams received from the receiver into a plurality of outputs that are each associated with said audio sources;

a floor control mechanism configured to establish a conversational floor between two or more users, comprising;

an analysis mechanism configured to analyze, for a plurality of users associated with said audio sources, one or more conversational characteristics comprising content of said audio streams of said users for different combinations of said users in possible conversation floors;

an identification mechanism configured to identify one or more conversational characteristics audio content shared between the two or more users in the possible conversational floors; and

a floor assignment mechanism configured to assign the two or more users to the conversational floor based on the shared conversational characteristics audio content,

wherein the floor control mechanism is further configured to maintain the conversational floor by automatically adjusting said floor controls based upon the conversational characteristics audio content.

20. The apparatus of claim 19, wherein said audio streams comprise a first audio stream and a second audio stream, which correspond to a first audio source and a second audio source that are each associated with a first user and a second user respectively, further comprising:

a synchronization mechanism configured to temporally align a portion of said first audio stream with a portion of said second audio stream;

a feature determination mechanism configured to determine a feature value by comparing said portion of said first audio stream with said portion of said second audio stream as aligned by the synchronization mechanism; and

an evaluation mechanism, responsive to the feature determination mechanism, configured to determine a probability that said first user and said second user are members of the conversational floor.

21. The apparatus of claim 19, wherein said audio streams comprise a first audio stream and a second audio stream, which correspond to a first audio source and a second audio source that are each associated with a first user and a second user respectively, further comprising an evaluation mechanism configured to determine a probability that said first user has made a response to said second user.

22. The apparatus of claim 19, wherein said audio streams comprise a first audio stream and a second audio stream, which correspond to a first audio source and a second audio source that are each associated with a first user and a second user respectively, further comprising an evaluation mechanism configured to determine a probability that said first audio stream and said second audio stream share a common element.

23. The apparatus of claim 19, wherein said audio streams comprise a first audio stream which correspond to a first audio source that are each associated with a first user, further comprising a referential determination mechanism configured to determine a probability that said first user has made a reference to a second user.

31. The apparatus of claim 19, wherein the conversational characteristics further comprise physiological characteristics of said users.

24. An apparatus for computer mediated communication, comprising:

a receiver configured to receive a plurality of communication streams, which each originates from one of a plurality of communication sources;

a mixer configured to mix, as controlled by a plurality of floor controls, said communication streams into a plurality of outputs that are each associated with said communication sources;

a user determination module to determine a plurality of users associated with the communication streams;

a first user action module to identify an action comprising a vocalization by a first such user, wherein the first such user is associated with a first of said communication sources;

a second user action module to identify a different action comprising a different vocalization by a second such user, wherein the second such user is associated with a second of said communication sources;

an analysis mechanism configured to analyze one or more conversational characteristics comprising audio content simultaneously shared by the first and second of such users for an indication in the shared audio content based on a probabilistic inference that said vocalization is in response to said different vocalization; and

a floor control mechanism configured to automatically adjust said floor controls based upon the indication,

wherein the action is selected from the group comprising backchannel/continuers, common content, prosody, and temporal vocalization adjacency.

25. An apparatus for computer mediated communication, comprising:

a receiver configured to receive a plurality of communication streams, which each originates from one of a plurality of communication sources;

a mixer configured to mix, as controlled by a plurality of floor controls, said communication streams into a plurality of outputs that are each associated with said communication sources;

a user determination module to determine a plurality of users associated with the communication streams;

an action module to identify an action comprising a vocalization by a first such user;

an analysis mechanism configured to analyze one or more conversational characteristics comprising audio content simultaneously shared by the first and a second of such users for an indication in the shared audio content based on a probabilistic inference that said vocalization refers to the second of such users; and

a floor control mechanism configured to automatically adjust said floor controls based upon the indication,

wherein the action is selected from the group comprising backchannel/continuers, common content, prosody, and temporal vocalization adjacency.

26. The apparatus of claim 25, wherein said action by the first user is a reference to an identifier of the second user.

27. A computer-readable storage medium storing computer program code that when executed by a computer, causes the computer to perform:

receiving a plurality of audio streams, which each originates from one of a plurality of audio sources;

mixing, as controlled by a plurality of floor controls, said audio streams received from the receiver into a plurality of outputs that are each associated with said audio sources;

establishing a conversational floor between two or more users, comprising;

analyzing, for a plurality of users associated with said audio sources, conversational characteristics comprising content of said audio streams of said users for different combinations of said users in possible conversation floors;

identifying one or more conversational characteristics audio content shared between the two or more users in the possible conversational floors; and

assigning the two or more users to the conversational floor based on the shared conversational characteristics audio content; and

maintaining the conversational floor by automatically adjusting said floor controls based upon the conversational characteristics audio content.

28. A computer-readable storage medium storing computer program code that when executed by a computer, causes the computer to perform:

receiving a plurality of communication streams, which each originates from one of a plurality of communication sources;

mixing, as controlled by a plurality of floor controls, said communication streams into a plurality of outputs that are each associated with said communication sources;

determining a plurality of users associated with the communication streams;

identifying an action comprising a vocalization by a first such user, wherein the first such user is associated with a first of said communication sources;

identifying a different action comprising a different vocalization by a second such user, wherein the second such user is associated with a second of said communication sources;

analyzing one or more conversational characteristics comprising audio content simultaneously shared by the first and second of such users for an indication in the shared audio content based on a probabilistic inference that said vocalization is in response to said different vocalization; and

automatically adjusting said floor controls based upon the indication,

wherein the action is selected from the group comprising backchannel/continuers, common content, prosody, and temporal vocalization adjacency.

29. A computer-readable storage medium storing computer program code that when executed by a computer, causes the computer to perform:

receiving a plurality of communication streams, which each originates from one of a plurality of communication sources;

mixing, as controlled by a plurality of floor controls, said communication streams into a plurality of outputs that are each associated with said communication sources;

determining a plurality of users associated with the communication streams;

identifying an action comprising a vocalization by a first such user;

analyzing one or more conversational characteristics comprising audio content simultaneously shared by the first and a second of such users for an indication in the shared audio content based on a probabilistic inference that said vocalization refers to the second of such users; and

automatically adjusting said floor controls based upon the indication,

wherein the action is selected from the group comprising backchannel/continuers, common content, prosody, and temporal vocalization adjacency.