System and method for adjusting floor controls based on conversational characteristics of participants

• US 8,463,600 B2
• Filed: 02/27/2012
• Issued: 06/11/2013
• Est. Priority Date: 02/28/2003
• Status: Expired due to Term

• Current Assignee: III Holdings 6, LLC (Intellectual Ventures LLC)
• Original Assignee: Palo Alto Research Center, Inc. (Xerox Holdings Corp.)
• Inventors: Aoki, Paul Masami, Szymanski, Margaret H., Thornton, James D., Wilson, Daniel H., Woodruff, Allison Gyle
• Primary Examiner(s): Desir, Pierre-Louis
• Assistant Examiner(s): BAKER, MATTHEW H
• Application Number:
  US13/406,365
• Publication Number:
  US 20120158402A1
• Time in Patent Office: 470 Days
• Field of Search: 704/201, 704/235, 704/255, 704/275, 379/52, 379/202.01, 370/260, 370/263, 370/338, 370/353, 455/518
• US Class Current: 704/206
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 21/0208 Noise filtering
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: H04M 3/564 whereby the feature is a sub-conference
• CPC Class Codes: H04M 3/566 relating to a participants right to speak arrangements for multi-party communication with floor control, e.g. for conferences, H04L65/4038, H04L65/4046, H04L65/4053
• CPC Class Codes: H04M 3/568 audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30
• CPC Class Codes: H04M 3/569 using the instant speaker's algorithm speech detection per se G10L25/78

Abstract

A system and method for automatically adjusting floor controls based on conversational characteristics is provided. Audio streams are received, which each originate from an audio source. Floor controls for a current configuration including at least a portion of the audio streams are maintained. Conversational characteristics shared by two or more of the audio sources are determined. Possible configurations for the audio streams are identified based on the conversational characteristics. An analysis of the current configuration and the possible configurations is performed. A change threshold comprising a minimum number of timeslices for at least one of the current configuration and one of the possible configurations is applied to the analysis. When the analysis satisfies the change threshold, the floor controls are automatically adjusted. The audio streams are mixed into one or more outputs based on the adjusted floor controls.

Claims

1. A system for adjusting floor controls based on conversational characteristics, comprising:

input modules to receive audio streams, which each originate from an audio source;

a floor analysis module to maintain floor controls for a current configuration of at least a portion of the audio streams from the audio sources that are participants to an interaction represented by the current configuration, to determine conversational characteristics shared by two or more of the audio sources, to identify different possible configurations of interactions between participants of the audio streams based on the conversational characteristics, to apply a predetermined number of timeslices to each one of the possible configurations, and to automatically adjust the floor controls when the predetermined number of timeslices of one of the possible configurations is satisfied;

an audio mixer module to mix the audio streams into one or more outputs based on the adjusted floor controls; and

a processor to execute the modules.

2. A system according to claim 1, wherein the conversational characteristics are determined by at least one of a turn taking analysis, a referential action analysis, and a responsive action analysis.

3. A system according to claim 2, further comprising:

a turn taking analysis module to perform the turn taking analysis by aligning the audio streams from two or more of the sources, monitoring the aligned audio stream, and identifying the conversational characteristics from the aligned audio stream.

4. A system according to claim 2, further comprising:

a referential action analysis module to perform the referential action analysis by determining that at least one of the audio streams occurs during a time window and identifying at least one conversational characteristic comprising one or more of a name and a name variant for at least one of the other audio sources during the time window.

5. A system according to claim 2, further comprising:

a responsive action analysis module to perform the responsive action analysis by determining that at least one of the audio streams occurs during a time window, by identifying one or more conversational characteristics comprising a backchannel response during the time window, and by matching the back channel response to vocalizations of another audio stream.

6. A system according to claim 1, wherein the conversational characteristics comprise at least one of sustained periods of overlapping speech, a lack of correlation for a beginning of speech, speech beginning at a transition relevance place, communication identifiers, change in speech volume, speech energy amplitude, common content, prosody, and backchannel responses.

7. A system according to claim 1, wherein the predetermined number of timeslices comprises consecutive timeslices.

8. A system according to claim 7, wherein the floor controls are automatically adjusted when the current conversational configuration is maintained for the predetermined number of timeslices and a most probable conversational configuration of the possible configurations is different from the current conversational configuration.

9. A system according to claim 7, wherein the floor controls are automatically adjusted when the current conversational configuration is different from a most probable conversational configuration of the possible conversational configurations for the predetermined number of consecutive timeslices.

10. A system according to claim 7, wherein the floor controls arc automatically adjusted when the current conversational configuration is different from a most probable conversational configuration of the possible conversational configurations for more than the predetermined net number of timeslices.

11. A computer-implemented method for adjusting floor controls based on conversational characteristics, comprising:

receiving audio streams, which each originate from an audio source and maintaining floor controls for a current configuration of at least a portion of the audio streams from the audio sources who are participants to an interaction represented by the current configuration;

determining conversational characteristics shared by two or more of the audio sources;

identifying different possible configurations of interactions between participants of the audio streams based on the conversational characteristics;

applying a predetermined number of timeslices to each one of the possible configurations and automatically adjusting the floor controls when the predetermined number of timeslices of one of the possible configurations is satisfied; and

mixing the audio streams into one or more outputs based on the adjusted floor controls.

12. A computer-implemented method according to claim 11, wherein the conversational characteristics are determined by at least one of a turn taking analysis, a referential action analysis, and a responsive action analysis.

13. A computer-implemented method according to claim 12, further comprising:

performing the turn taking analysis, comprising;

aligning the audio streams from two or more of the sources;

monitoring the aligned audio streams; and

identifying the conversational characteristics from the aligned audio stream.

14. A computer-implemented method according to claim 12, further comprising:

performing the referential action analysis, comprising;

determining that at least one of the audio streams occurs during a time window; and

identifying at least one conversational characteristic comprising one or more of a name and a name variant for at least one of the other audio sources during the time window.

15. A computer-implemented method according to claim 12, further comprising:

performing the responsive action analysis, comprising;

determining that at least one of the audio streams occurs during a time window;

identifying one or more conversational characteristics comprising a backchannel response during the time window; and

matching the back channel response to vocalizations of another audio stream.

16. A computer-implemented method according to claim 11, wherein the conversational characteristics comprise at least one of sustained periods of overlapping speech, a lack of correlation for a beginning of speech, speech beginning at a transition relevance place, communication identifiers, change in speech volume, speech energy amplitude, common content, prosody, and backchannel responses.

17. A computer-implemented method according to claim 11, wherein the predetermined number of timeslices comprises consecutive timeslices.

18. A computer-implemented method according to claim 17, wherein the floor controls are automatically adjusted when the current conversational configuration is maintained for the predetermined number of timeslices and a most probable conversational configuration of the possible configurations is different from the current conversational configuration.

19. A computer-implemented method according to claim 17, wherein the floor controls are automatically adjusted when the current conversational configuration is different from a most probable conversational configuration of the possible conversational configurations for the predetermined number of consecutive timeslices.

20. A computer-implemented method according to claim 17, wherein the floor controls are automatically adjusted when the current conversational configuration is different from a most probable conversational configuration of the possible conversational configurations for more than the predetermined net number of timeslices.