Audio processing in a multi-participant conference

• US 8,456,508 B2
• Filed: 11/29/2010
• Issued: 06/04/2013
• Est. Priority Date: 04/28/2005
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Jeong, Hyeonkuk, Salsbury, Ryan
• Primary Examiner(s): WOO, STELLA L
• Application Number:
  US12/955,902
• Publication Number:
  US 20110074914A1
• Time in Patent Office: 918 Days
• Field of Search: 348/14.08, 348/14.09, 379/202.01, 379/206.01, 370/260, 370266-269
• US Class Current: 348/14.08
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Some embodiments provide an architecture for establishing multi-participant audio conferences over a computer network. This architecture has a central distributor that receives audio signals from one or more participants. The central distributor mixes the received signals and transmits them back to participants. In some embodiments, the central distributor eliminates echo by removing each participant'"'"'s audio signal from the mixed signal that the central distributor sends to the particular participant.

Claims

1. A non-transitory computer readable medium of a device of a first participant of a multi-participant conference, the computer readable medium storing a computer program which when executed by at least one processor of the device distributes audio content in a multi-participant conference, the computer program comprising sets of instructions to:

receive audio signals from at least second and third participants of the conference;

generate data representative of strengths of the received audio signals;

generate mixed audio signals from the received audio signals;

to each mixed audio signal, appending a set of the generated strength data for the audio signals that are mixed to produce each mixed audio signal; and

transmit the mixed audio signals with the strength data to the second and third participants of the conference.

2. The non-transitory computer readable medium of claim 1, wherein the computer program further comprises a set of instructions to determine a strength of each received audio signal, wherein the set of instructions to determine the strength comprise a set of instructions to calculate the strength of each received audio signal as a root mean square (RMS) power of the received audio signal.

3. The non-transitory computer readable medium of claim 1, wherein the set of instructions to generate a mixed audio signal for a participant of the multi-participant conference comprises a set of instructions to remove the audio signal of the participant.

4. The non-transitory computer readable medium of claim 1, wherein the appended strength data is configured to be used by devices of the second and third participants to pan the mixed audio signal across audio loudspeakers.

5. The non-transitory computer readable medium of claim 1, wherein the set of instructions to generate the mixed audio signals comprises a set of instructions to generate a first mixed audio signal for the second participant and generating a second, different mixed audio signal for the third participant.

6. The non-transitory computer readable medium of claim 1, wherein the mixed audio signals are transmitted using real time protocol (RTP) packets comprising the strength data for each of the audio signals in the mixed audio signals.

7. The non-transitory computer readable medium of claim 1, wherein the computer program comprises a set of instructions to locally output the mixed audio signals at the device of the first participant.

8. A method of distributing audio content in a multi-participant conference, the method comprising:

at a device of a first participant of the conference;

receiving audio signals from at least second and third participants of the conference;

generating a single mixed audio signal comprising an audio signal locally captured by the device of the first participant and the received audio signals;

transmitting the mixed audio signal to the second and third participants; and

removing the audio signal of the device of the first participant from the mixed audio signal in order to locally output the mixed audio signal with audio signal locally captured by the device of the first participant removed.

9. The method of claim 8 comprising generating strength data that represents strengths of (i) the audio signal captured by the device of the first participant and (ii) each of the received audio signals.

10. The method of claim 9, wherein the strength data comprises the strength of each of the audio signals as a root mean square (RMS) of the audio signals.

11. The method of claim 8, wherein the single mixed audio signal is generated by the device of the first participant using the audio signals of the first, second, and third participants.

12. The method of claim 8 further comprising locally outputting the mixed audio signal without the audio signal captured by the device of the first participant.

13. The method of claim 9, wherein the strength data is for panning the mixed audio signal across audio loudspeakers of the device of the first participant and devices associated with the second and third participants.

14. The method of claim 8, wherein the device generates, during the conference, the mixed audio signal, a second mixed audio signal to be provided to the second participant with the received audio signal of the second participant removed, a third mixed audio signal to be provided to the third participant with the received audio signal of the third participant removed, wherein devices associated with the second and third participants do not generate any mixed audio signals during the conference.

15. A non-transitory computer readable medium of a first device of a first participant of a multi-participant conference, the computer readable medium storing a computer program which when executed by at least one processor of the device receives audio content in a multi-participant conference, the computer program comprising sets of instructions to:

receive a mixed audio signal from a second device of a second participant of the conference, the mixed audio signal comprising (i) an audio signal sent from the first device captured by the first device and sent to the second device, (ii) audio captured by the second device, and (iii) an audio signal captured by a third device and sent to the second device;

remove the audio signal of the first device from the mixed audio signal; and

output the mixed audio signal at the first device with the audio signal of the first device removed.

16. The non-transitory computer readable medium of claim 15, wherein the computer program comprises a set of instructions to send the audio signal of the first device to the second device of the second participant.

17. The non-transitory computer readable medium of claim 15, wherein the computer program comprises a set of instructions to receive strength data that represents strengths of audio signals received at the second device of the second participant.

18. The non-transitory computer readable medium of claim 17, wherein the set of instructions to output the mixed audio signal comprises a set of instructions to use the strength data is to pan the mixed audio signal across audio loudspeakers of the first device.

19. The non-transitory computer readable medium of claim 17, wherein the strength data is generated by the second device by calculating a strength of each audio signal of the mixed audio signal as a root mean square (RMS) power of the audio signal.

20. The non-transitory computer readable medium of claim 15, wherein the second device generates mixed audio signals during the conference and the first and third devices do not generate mixed audio signals during the conference.