Audio processing in a multi-participant conference

• US 7,864,209 B2
• Filed: 04/28/2005
• Issued: 01/04/2011
• Est. Priority Date: 04/28/2005
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Jeong, Hyeonkuk, Salsbury, Ryan
• Primary Examiner(s): Woo; Stella L
• Application Number:
  US11/118,555
• Publication Number:
  US 20060247045A1
• Time in Patent Office: 2,077 Days
• Field of Search: 348/14.08, 348/14.09, 348/14.01, 370/352, 379/202.01, 379/203.01, 379/204.01, 379/205.01, 379/206.01
• US Class Current: 348/14.08
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Some embodiments provide an architecture for establishing multi-participant audio conferences over a computer network. This architecture has a central distributor that receives audio signals from one or more participants. The central distributor mixes the received signals and transmits them back to participants. In some embodiments, the central distributor eliminates echo by removing each participant'"'"'s audio signal from the mixed signal that the central distributor sends to the particular participant.

Claims

1. A method of distributing audio content in a multi-participant audio/video conference, the method comprising:

at a device of a first participant of said conference;

receiving audio signals from at least second and third participants of said conference;

determining a strength of each received audio signal;

generating indicia representative of said strengths for the received audio signals;

generating at least first and second mixed audio signals from the received audio signals, said first mixed audio signal different from said second mixed audio signal;

to each particular mixed audio signal, appending a set of generated strength indicia of the audio signals that are mixed to produce the particular mixed audio signal;

transmitting said first mixed audio signal to said second participant; and

transmitting said second mixed audio signal to said third participant.

2. The method of claim 1, wherein determining said strength comprises calculating the strength of each received audio signal as a root mean square (RMS) power of said received audio signal.

3. The method of claim 1, wherein the received audio signals are in real time protocol (RTP) packets.

4. The method of claim 1 further comprising locally outputting the mixed audio signals at the device of the first participant.

5. The method of claim 1 further comprising using the determined strength of a particular received audio signal to eliminate noise by not including the particular received audio signal in the mixed audio signal when said audio signal is weaker than other participants'"'"' audio signals.

6. The method of claim 1, wherein the determined strength of each audio signal is used to assign a weight to each audio signal, wherein the assigned weights are used to prevent one participant'"'"'s audio signal from overpowering other participants'"'"' audio signals.

7. The method of claim 1, wherein the conference is an audio-only conference.

8. The method of claim 1, wherein said generating comprises generating a third mixed audio signal that includes the audio signals of the second and third participants, and said method further comprises locally playing said third mixed audio signal on the device of the first participant of said conference.

9. A method of distributing audio content in a multi-participant audio/video conference the method comprising:

at a device of a first participant of said conference;

receiving audio signals from at least second and third participants of said conference;

generating at least first and second mixed audio signals from the received audio signals, said first mixed audio signal different from said second mixed audio signal;

transmitting said first mixed audio signal to said second participant; and

transmitting said second mixed audio signal to said third participant,

wherein generating said mixed audio signals comprises removing the audio signal of the second participant from the first mixed audio signal and removing the audio signal of the third participant from the second mixed audio signal.

10. The method of claim 9 further comprising:

determining a strength of each received audio signal;

generating a set of indicia representative of said strengths for the received audio signals; and

appending said set of signal strength indicia to said mixed audio signals.

11. The method of claim 9, wherein the received audio signals are in real time protocol (RTP) packets.

12. A method of creating a stereo panning effect in a multi-participant audio/video conference, the method comprising:

displaying representations of at least two different participants in a display area, said displaying comprising displaying each of the representations in a distinct location in the display area;

receiving a mixed audio signal that comprises a set of indicia indicative of a signal strength for each of the different participants; and

panning the mixed audio signal across audio loudspeakers using said set of signal strength indicia in order to create an effect that a perceived location of an audio signal of a particular participant matches the location of the particular participant in the display area.

13. The method of claim 12, wherein said panning comprises:

identifying the location of the representation of the particular participant displayed in said display area, and

creating a delay on at least one of the audio loudspeakers to account for the identified location of the representation of the particular participant.

14. The method of claim 12, wherein at least one of the representations displayed in said display area is at least one of an icon, an image, and a video.

15. A method of creating a stereo panning effect in a multi-participant audio/video conference, the method comprising:

determining that a first participant in said conference performed a particular action;

identifying a location of a video presentation of the first participant on a display device of a second participant, said display device of the second participant further displaying a video presentation of at least a third participant at another location;

determining a sound effect for the particular action; and

based on said identified location, panning said sound effect across audio loudspeakers of the second participant to cause the second participant to perceive sound associated with said action to originate from the location of the video presentation of the first participant on the display device.

16. The method of claim 15, wherein said panning comprises creating a delay on at least one of the audio loudspeakers of the second participant to account for the identified location of the first participant.

17. The method of claim 15, wherein said action is one of joining and leaving the conference.

18. The method of claim 15, wherein said panning comprises using a signal strength indicia to account for the identified location of the first participant.

19. A computer readable medium storing a computer program for distributing audio content in a multi-participant audio/video conference, the computer program comprising sets of instructions for:

at a device of a first participant of said conference;

receiving audio signals from at least second and third participants of said conference;

determining a strength of each received audio signal;

generating indicia representative of said strengths for the received audio signals;

generating at least first and second mixed audio signals from the received audio signals, said first mixed audio signal different from said second mixed audio signal;

to each particular mixed audio signal, appending the set of generated strength indicia of the audio signals that are mixed to produce the particular mixed audio signal;

transmitting said first mixed audio signal to said second participant; and

transmitting said second mixed audio signal to said third participant.

20. The computer readable medium of claim 19, wherein said set of instructions for determining said strength comprises a set of instructions for calculating the strength of each received audio signal as a root mean square (RMS) power of said received audio signal.

21. The computer readable medium of claim 19, wherein said set of instructions for generating comprises a set of instructions for generating a third mixed audio signal that includes the audio signals of the second and third participants, and said program further comprises a set of instructions for locally playing said third mixed audio signal on the device of the first participant of said conference.

22. A computer readable medium storing a computer program for distributing audio content in a multi-participant audio/video conference, the computer program comprising sets of instructions for:

at a device of a first participant of said conference;

receiving audio signals from at least second and third participants of said conference;

generating at least first and second mixed audio signals from the received audio signals, said first mixed audio signal different from said second mixed audio signal;

transmitting said first mixed audio signal to said second participant; and

transmitting said second mixed audio signal to said third participant;

wherein the set of instructions for generating the mixed audio signals comprises a set of instructions for removing the audio signal of the second participant from the first mixed audio signal and removing the audio signal of the third participant from the second mixed audio signal.

23. The computer readable medium of claim 22, wherein said generating comprises generating a third mixed audio signal that includes the audio signals of the second and third participants, said computer program further comprising a set of instructions for locally playing said third mixed audio signal on the device of the first participant of said conference.

24. A computer readable medium storing a computer program for creating a stereo panning effect in a multi-participant audio/video conference, the computer program comprising sets of instructions for:

displaying representations of at least two different participants in a display area, said displaying comprising displaying each of the representations in a distinct location;

receiving a mixed audio signal that comprises a set of indicia indicative of a signal strength for each of the different participants; and

panning the mixed audio signal across audio loudspeakers using said set of signal strength indicia in order to create an effect that a perceived location of an audio signal of a particular participant matches the location of the particular participant in the display area.

25. The computer readable medium of claim 24, wherein said set of instructions for said panning comprises sets of instructions for:

identifying the location of one of the representations that is displayed in said display area, and

creating a delay on at least one of the audio loudspeakers to account for the identified location of the representation.

26. The computer readable medium of claim 24, wherein at least one of the representations displayed in said display area is at least one of an icon, an image, and a video.

27. A method of distributing audio content in a multi-participant audio/video conference the method comprising:

at a device of a first participant of said conference;

receiving audio signals from at least second and third participants of said conference;

generating at least first and second mixed audio signals from the received audio signals, said first mixed audio signal being different from said second mixed audio signal;

transmitting said first mixed audio signal to said second participant; and

transmitting said second mixed audio signal to said third participant,

wherein a plurality of the mixed audio signals are transmitted using real time protocol (RTP) packets comprising an indicia of strength of each audio signal in the mixed audio signal.

28. The method of claim 27 further comprising:

determining a strength of each received audio signal;

generating a set of indicia representative of said strengths for the received audio signals; and

appending said set of signal strength indicia to said mixed audio signals.

29. The method of claim 28, wherein the determined strength of each audio signal is used to assign a weight to each audio signal, wherein the assigned weights are used to prevent one participant'"'"'s audio signal from overpowering other participants'"'"' audio signals.

30. For an audio/video conference comprising a plurality of participants, a method comprising:

providing a graphical user interface (GUI) comprising a display area for displaying graphical representations of each of the plurality of participants in a particular location; and

providing a controller for (i) receiving a mixed audio signal comprising audio signals corresponding to each participant of the plurality of participants, (ii) specifying at least one playback parameter for playing back the mixed audio signal in order to create a panning effect that a perceived location of an audio signal of a particular participant matches the particular location of the particular participant in the display area, and (iii) receiving a set of signal strength indicia.

31. The method of claim 30, wherein a first participant has a set of stereo loudspeakers, wherein the creating the panning effect comprises playing the audio signal such that said audio signal is perceived as originating from a loudspeaker that is closer to the particular location of the particular participant in the display area than a loudspeaker that is farther from said particular location.

32. The method of claim 30, wherein a first participant has a set of stereo loudspeakers, wherein a first loudspeaker is closer to the particular location of the particular participant in the display area than a second loudspeaker, wherein the specified playback parameter causes the mixed audio signal to be played through the first loudspeaker before the second loudspeaker when the particular participant speaks.

33. The method of claim 30, wherein a first participant has a set of stereo loudspeakers, wherein the controller is further for using said particular location as a factor to define a difference in playback parameters of the mixed audio signal through each loudspeaker.

34. The method of claim 30, wherein a first participant has a set of stereo loudspeakers, wherein the specified playback parameter causes the audio signal to be played through one of said loudspeakers with an altered amplitude.

35. The method of claim 30, wherein the graphical representation of a particular participant is one of a picture, a video, and an icon of the particular participant.

36. The method of claim 30, wherein the audio signal comprises a sound associated with an action event that is one of joining and leaving the conference.

37. The method of claim 30, wherein the controller is further for determining that the particular participant is a most active participant by comparing signal strengths of the audio signals of each participant and determining that the particular participant has the strongest audio signal.

38. For an audio conference having a plurality of participants, a method comprising:

providing an audio capture module for capturing a first audio signal of a first participant speaking during said conference;

providing an audio signal strength calculator for calculating a signal strength of the received audio signals; and

providing an audio mixer for (i) receiving the first audio signal and at least a second audio signal of a second participant, (ii) generating a mixed audio signal for the second participant, said generating comprising removing the audio signal of the second participant from said mixed audio signal, and (iii) transmitting said mixed audio signal to said second participant with data regarding said calculated audio signal strength, wherein said audio capture module and said audio mixer are provided as parts of one audio conference application.

39. The method of claim 38 further comprising providing a focus point module comprising:

a set of decoders for receiving the audio signals;

said audio mixer; and

said audio capture module.

40. The method of claim 38 further comprising providing a set of buffers for storing the audio signals received from the participants of said conference.

41. For an audio conference having a plurality of participants, a method comprising:

providing an audio capture module for capturing a first audio signal of a first participant speaking during said conference;

providing an audio mixer for (i) receiving the first audio signal and at least a second audio signal of a second participant, (ii) generating a mixed audio signal for the second participant, said generating comprising removing the audio signal of the second participant from said mixed audio signal, and (iii) transmitting said mixed audio signal to said second participant, wherein said audio capture module and said audio mixer are provided as parts of one audio conference application; and

providing a display area for displaying signal strengths of the received audio signals.

42. The method of claim 41 further comprising providing a set of buffers for storing the audio signals received from the participants of said conference.

43. The method of claim 41 further comprising providing an audio signal strength calculator for calculating signal strength of the received audio signals.

44. For an audio conference having a plurality of participants, a method comprising:

providing an audio capture module for capturing a first audio signal of a first participant speaking during said conference; and

providing an audio mixer for (i) receiving the first audio signal and at least a second audio signal of a second participant, (ii) generating a mixed audio signal for the second participant, said generating comprising removing the audio signal of the second participant from said mixed audio signal, and (iii) transmitting said mixed audio signal to said second participant, wherein said audio capture module and said audio mixer are provided as parts of one audio conference application,

wherein the mixed audio signal comprises indicia that express signal strengths of the received audio signals.

45. The method of claim 44 further comprising providing an encoder for (i) encoding the mixed audio signal and (ii) outputting said encoded mixed audio signal to at least one particular participant.

46. The method of claim 45, wherein said mixed audio signal does not include an audio signal corresponding to the particular participant.

47. For a multi-participant audio/video conference, a method comprising:

providing a multi-participant audio/video conference application, wherein said providing said multi-participant audio/video conference application comprises;

providing an audio capture module for locally capturing an audio signal of a first participant speaking during said multi-participant audio/video conference; and

providing an audio mixer for (i) receiving at least one other audio signal from at least a second participant of said conference, (ii) generating a mixed audio signal from the audio signals of the first and second participants, and (iii) transmitting said mixed audio signal to a third participant of said conference.

48. The method of claim 47, wherein providing the multi-participant audio/video conference application further comprises providing a graphical user interface that comprises a display area for displaying representations of at least the second and third participants of said conference.

49. The method of claim 48 further comprising providing a focus point module, wherein said focus point module is for determining a set of indicia representative of signal strengths of the received audio signals, wherein providing the multi-participant audio/video conference application further comprises providing a level meter module for displaying level meters in said display area based on the set of signal strength indicia.

50. The method of claim 48, wherein providing the multi-participant audio/video conference application further comprises providing a panning module for panning the received mixed audio signals across loudspeakers in accord with locations of the representations displayed in said display area.

51. A method of distributing audio content in a multi-participant audio/video conference, said method comprising:

receiving audio signals from at least first and second participants of said conference;

for the first participant, generating a mixed audio signal from the received audio signals, said generating comprising removing the audio signal of the first participant from said mixed audio signal; and

transmitting said mixed audio signal to said first participant of said conference,

wherein said receiving, generating, and transmitting are operations performed on a device of a participant of said conference other than the first participant, said operations performed by an audio/video conference application executing on said device.

52. The method of claim 51 further comprising:

determining a strength of each received audio signal;

generating a set of indicia representative of said strengths for the received audio signals; and

appending said set of signal strength indicia to said mixed audio signals.

53. The method of claim 51, wherein the received audio signals are in real time protocol (RTP) packets.