Speech-selective audio mixing for conference

• US 9,237,238 B2
• Filed: 07/23/2014
• Issued: 01/12/2016
• Est. Priority Date: 07/26/2013
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Polycom Incorporated (HP Inc.)
• Inventors: Chu, Peter L., Liu, Yibo, Avni, Uri, Noy, Ami
• Primary Examiner(s): Nguyen, Khai N
• Application Number:
  US14/339,244
• Publication Number:
  US 20150030149A1
• Time in Patent Office: 538 Days
• Field of Search: 379/90.01, 379/93.01, 379/93.21, 379/157, 379/158, 379/201.01, 379/202.01, 379/207.01, 370/259, 370/260, 370/261, 370/262, 455/414.1, 455/416
• US Class Current: 1/1
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: H04M 2201/14 Delay circuits; Timers
• CPC Class Codes: H04M 3/56 Arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities video conference systems H04N7/15
• CPC Class Codes: H04M 3/568 audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30

Abstract

A conference apparatus reduces or eliminates noise in audio for endpoints in a conference. Endpoints in the conference are designated as a primary talker and as secondary talkers. Audio for the endpoints is processed with speech detectors to characterize the audio as speech or not and to determine energy levels of the audio. As the audio is written to buffers and then read from the buffers, decisions for the gain settings of faders for read audio of the endpoints being combined in the speech selective mix. In addition, the conference apparatus can mitigate the effects of a possible speech collision that may occur during the conference between endpoints.

Claims

1. A method for conducting a conference, comprising:

buffering audio of each of a plurality of endpoints in the conference with an audio delay, whereby the endpoints have respective delay values for the audio delay;

leveling the audio of each of the endpoints in the conference with a fader, whereby the endpoints have respective gain values for the fader;

detecting speech in the audio of any one of the endpoints in the conference;

controlling the respective delay value for the audio delay and the respective gain value for the fader for each of the endpoints based on the detection of the speech; and

outputting a mix of the audio of the endpoints in the conference based on the control.

2. The method of claim 1, wherein the method comprises performing the acts of buffering, leveling, detecting, controlling, and outputting with a conferencing bridge in communication with the endpoints.

3. The method of claim 2, wherein the method comprises performing the acts of buffering, leveling, detecting, controlling, and outputting with one or more of the endpoints along with the conferencing bridge in communication with the endpoints.

4. The method of claim 1, wherein the method comprises performing the acts of buffering, leveling, detecting, controlling, and outputting with each of the endpoints.

5. The method of claim 1, wherein buffering the audio of each of the endpoints in the conference with the audio delay comprises designating write and read pointers of buffers for each of the endpoints.

6. The method of claim 1, wherein detecting speech in the audio of any one of the endpoints in the conference comprises determining that at least one of the endpoints is a primary endpoint having a primary talker.

7. The method of claim 6, wherein determining that the at least one of the endpoints is the primary endpoint having the primary talker comprises:

determining energy levels of the detected speech of the any of the endpoints;

determining which of the endpoints with the detected speech has a maximum of the energy levels; and

designating the determined endpoint as the primary endpoint.

8. The method of claim 7, wherein determining which of the endpoints with the detected speech has the maximum energy level comprises maintaining a previous designation of the primary endpoint when the determined maximum does not at least meet a threshold.

9. The method of claim 6, wherein the step of controlling the respective delay value for the audio delay and the respective gain value for the fader comprises controlling the respective gain value of the fader for the primary endpoint in relation to the respective delay value of the audio delay for buffering of the primary endpoint.

10. The method of claim 9, wherein the step of controlling the respective gain value of the fader in relation to the respective delay value of the audio delay for the primary endpoint comprises:

increasing the fader for the primary endpoint toward a maximum gain while setting the audio delay for buffering of the primary endpoint toward a maximum value.

11. The method of claim 10, wherein increasing the fader for the primary endpoint toward the maximum gain is done over a time interval.

12. The method of claim 10, wherein the step of controlling the respective gain value of the fader in relation to the respective delay value of the audio delay for the primary endpoint comprises:

decreasing the audio delay for the primary endpoint toward a minimum value as long as the fader for the primary endpoint is toward a maximum gain and an energy level of the audio for the primary endpoint is below a threshold.

13. The method of claim 6, further comprising designating at least one of the other endpoints as a secondary endpoint.

14. The method of claim 13, wherein the step of controlling the respective delay value for the audio delay and the respective gain value for the fader comprises controlling the respective gain value of the fader for the secondary endpoint in relation to the respective delay value of the audio delay for buffering of the secondary endpoint.

15. The method of claim 14, wherein the step of controlling the respective gain value of the fader in relation to the respective delay value of the audio delay for the secondary endpoint comprises:

decreasing the fader of the secondary endpoint toward a minimum gain as long as the audio of the secondary endpoint is not detected speech.

16. The method of claim 15, wherein decreasing the fader for the second endpoint toward the minimum gain is done over a time interval.

17. The method of claim 14, wherein the step of controlling the respective gain value of the fader in relation to the respective delay value of the audio delay for the secondary endpoint comprises:

decreasing the fader of the secondary endpoint toward a minimum gain as long as the audio of the secondary endpoint is detected speech having an energy level below a threshold.

18. The method of claim 14, wherein the step of controlling the respective gain value of the fader in relation to the respective delay value of the audio delay for the secondary endpoint comprises:

increasing the fader of the secondary endpoint toward a maximum gain as long as the audio of the secondary endpoint is detected speech having an energy level above a threshold.

19. The method of claim 14, wherein the step of controlling the respective gain value of the fader in relation to the respective delay value of the audio delay for the secondary endpoint comprises:

decreasing the audio delay toward a minimum value for the secondary endpoint as long as the audio of the secondary endpoint is detected speech, the fader for the secondary endpoint is toward a maximum gain, and an energy level of the detected speech for the secondary endpoint is below a threshold.

20. The method of claim 1, wherein outputting the mix of the audio of the endpoints in the conference comprises mixing at an intermediate node between the endpoints, at the endpoints, or at a combination of the intermediate node and the endpoints.

21. The method of claim 1, wherein detecting the speech in the audio of any one of the endpoints in the conference comprises identifying a speech collision between at least two of the endpoints.

22. The method of claim 21, wherein controlling the audio delay and the fader for each of the endpoints based on the detection of the speech comprises managing the speech collision with minimal interruption to the conference.

23. The method of claim 21, wherein identifying the speech collision between the at least one endpoints comprises:

determining audio energy of the audio for each of the endpoints; and

monitoring the determined audio energy for the speech collision between the at least one endpoints.

24. The method of claim 23, wherein monitoring the determined audio energy for the speech collision between the at least one endpoints comprises determining a first increase in the audio energy for a first of the at least two endpoints occurring within a time interval of a second increase in the audio energy for a second of the at least two endpoints.

25. The method of claim 21, wherein identifying the speech collision between at least two of the endpoints comprises informing at least one of the at least two endpoints about the speech collision.

26. The method of claim 25, wherein informing the at least one endpoint about the speech collision comprises adding at least one of an audio indication and a visual indication of the speech collision to information for the at least one of the at least two endpoints.

27. The method of claim 25, further comprising adding the audio of the at least one endpoint to the mix for the conference.

28. The method of claim 27, wherein adding the audio of the at least one endpoint comprises increasing a gain of the fader for the at least one endpoint over a time interval.

29. The method of claim 21, wherein identifying the speech collision between the at least one endpoints comprises ignoring a break in the detected speech for a first of the at least two endpoints when interrupted by a second of the at least two endpoints.

30. A programmable storage device having program instructions for causing a programmable control device to perform a method for conducting a conference, the method comprising:

buffering audio of each of a plurality of endpoints in the conference with an audio delay, whereby the endpoints have respective delay values for the audio delay;

leveling the audio of each of the endpoints in the conference with a fader, whereby the endpoints have respective gains values for the fader;

detecting speech in the audio of any one of the endpoints in the conference;

controlling the respective delay value for the audio delay and the respective gain value for the fader for each of the endpoints based on the detection of the speech; and

outputting a mix of the audio of the endpoints in the conference based on the control.

31. An apparatus for conducting a conference, comprising:

communication equipment connecting a plurality of endpoints in the conference, the communication equipment sending and receiving audio; and

memory having buffers for buffering the audio;

processing equipment in communication with the communication equipment and the memory, the processing equipment configured to;

buffer the audio in the buffers of each of the endpoints with an audio delay, whereby the endpoints have respective delay values for the audio delay;

level the audio of each of the endpoints in the conference with a fader, whereby the endpoints have respective fade values for the fader;

detect speech in the audio of any one of the endpoints;

control the respective delay value for the audio delay and the respective fade value for the fader for each of the endpoints based on the detection of the speech; and

output a mix of the audio of each of the endpoints in the conference based on the control.

32. The apparatus of claim 31, wherein one or more of the communication equipment, the memory, and the processing equipment comprise a part of a conferencing bridge in communication with the endpoints.

33. The apparatus of claim 31, wherein one or more of the communication equipment, the memory, and the processing equipment comprise a part of at least one of the endpoints.