Audio processing optimization in a multi-participant conference

• US 8,433,813 B2
• Filed: 07/07/2010
• Issued: 04/30/2013
• Est. Priority Date: 04/07/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Garcia, Roberto Jr., Jeong, Hyeonkuk, Patel, Nirav, Tung, Berkat S., Abuan, Joe S.
• Primary Examiner(s): Chan, Wing F
• Assistant Examiner(s): PATEL, HITESHKUMAR R
• Application Number:
  US12/832,046
• Publication Number:
  US 20110252157A1
• Time in Patent Office: 1,028 Days
• Field of Search: 709/204, 709/223, 709/231, 348/14.02, 348/14.08, 348/14.09, 370/260
• US Class Current: 709/231
• CPC Class Codes: H04L 12/1827 Network arrangements for conference optimisation or adaptation
• CPC Class Codes: H04L 43/08 Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 61/2514 between local and global IP addresses
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

A first computing device distributes audio streams to several computing devices of participants in a communication session. Some embodiments establishes a star network with the first computing device as a central network hub for receiving audio streams from other computing devices, compositing the audio streams and distributing the composited audio streams to the other computing devices. Through the star network, the first computing device receives audio streams from the other computing devices. The first computing device generates at least two different composite audio streams for at least two different computing devices by (i) identifying a set of silent participants in the communication session, and (ii) eliminating redundant audio processing operations that produce the same composite audio streams for different computing devices because of the identified set of silent participants. The first computing device sends each computing device the composited audio stream for the device.

Claims

1. A non-transitory computer readable medium of a first computing device that participates in a communication session among a plurality of computing devices, the computer readable medium storing a computer program that is executable by at least one processing unit, the computer program comprising:

a set of instructions for establishing a star network among the plurality of computing devices with the first computing device as a central network hub;

a set of instructions for receiving, through the star network, an audio stream from each of the plurality of computing devices;

a set of instructions for compositing at least a subset of the audio streams, wherein the set of instructions for compositing comprises a set of instructions for generating at least two different composite audio streams for at least two different computing devices, wherein the set of instructions for the generating comprises sets of instructions for (i) identifying a set of silent participants in the communication session by identifying audio data packets that do not contain audio from silent participants based on identified sizes of audio data packets of each received audio stream, (ii) identifying computing devices to which to send a same composite audio stream based on the identified set of silent participants; and

(iii) generating the same composite audio stream for the identified computing devices, wherein the same composite audio stream does not include audio streams from the set of silent participants; and

a set of instructions for sending each computing device the composite audio stream for the device, wherein the same composite audio stream is sent for the identified computing devices.

2. The non-transitory computer readable medium of claim 1, wherein the computer program further comprises sets of instructions for:

receiving audio metadata from a second computing device, the audio metadata comprising a list of other computing devices that the second computing device has muted; and

for the second computing device, specifying the computing devices on the list as silent participants.

3. The non-transitory computer readable medium of claim 1,

wherein the computer program further comprises a set of instructions for examining at different times each received audio stream to determine whether a participant who is using the computing device from which the first computing device receives the audio stream is speaking at different times;

wherein when the determination is made that the participant is not speaking at a particular time, the participant is added to the set of silent participants.

4. The non-transitory computer readable medium of claim 3, wherein the participant is added to and removed from the set of silent participants as the participant become silent during the communication session and starts talking during the communication session.

5. The non-transitory computer readable medium of claim 1, wherein the sets of instructions for identifying the set of silent participants, identifying the computing devices, and generating the same audio stream execute periodically in order to account for participants who are silent and talking at different instances in the communication session.

6. The non-transitory computer readable medium of claim 1, wherein said audio processing operations comprises decoding a received audio stream and encoding the composited audio stream.

7. A method for a first computing device to distribute audio streams to a plurality of computing devices of participants in a communication session conducted through a network connecting the computing devices, the method comprising:

establishing a star network with the first computing device as a central network hub;

through the star network, receiving an audio stream from each of the plurality of computing devices;

compositing at least a subset of the audio streams and distributing the composited audio streams to the other ones of the plurality of computing devices, wherein said compositing and said distributing comprises;

identifying a received audio stream that for at least one particular computing device does not need to be composited with other received audio stream, wherein said identifying the audio stream comprises determining that other audio streams that are to be routed to the particular computing device are from participants in the audio conference who are silent, said determining comprising identifying audio data packets that do not contain audio based on identified sizes of audio data packets of each of the other audio streams; and

routing the identified received audio stream to the particular computing device without compositing the identified audio stream with other audio streams.

8. The method of claim 7 further comprising:

receiving audio metadata from the particular computing device, the audio metadata comprising a list of other computing devices that the particular computing device has muted;

wherein a silent participant is a participant whose audio is muted by the particular computing device.

9. The method of claim 7 further comprising:

at different time intervals, examining each received audio stream to determine whether a participant who is using the computing device from which the first computing device received the audio stream is speaking at that time interval.

10. A non-transitory computer readable medium of a first computing device that participates in a communication session among a plurality of computing devices, the computer readable medium storing a computer program that is executable by at least one processing unit, the computer program comprising:

a set of instructions for establishing a star network among the plurality of computing devices with the first computing device as a central network hub for compositing a plurality of audio streams and distributing the composited audio streams to the other computing devices;

a set of instructions for receiving, through the star network, the plurality of audio streams, wherein one of the plurality of audio streams is received from each of the plurality of computing devices;

a set of instructions for identifying one of the received audio streams that for at least one particular computing device does not need to be composited with any of the other received audio streams, wherein the set of instructions for identifying one of the received audio streams comprises a set of instructions for determining that all audio streams that are to be routed to the particular computing device, other than the identified audio stream, are from participants in the communication session who are silent, by identifying audio data packets that do not contain audio based on identified sizes of audio data packets of each of the other received audio streams; and

a set of instructions for routing the identified audio stream to the particular computing device without compositing the identified audio stream with other audio streams.

11. The non-transitory computer readable medium of claim 10, wherein the computer program further comprises a set of instructions for receiving audio metadata from the particular computing device, the audio metadata comprising a list of other computing devices that the particular computing device has muted, wherein each of the silent participants is a participant whose audio is muted by the particular computing device.

12. The non-transitory computer readable medium of claim 10, wherein the computer program further comprises a set of instructions for, at different time intervals, examining each received audio stream to determine whether a participant who is using the computing device from which the first computing device received the audio stream is speaking at that time interval, wherein when the determination is made that the participant is not speaking at a particular time, the participant is added to the set of silent participants.

13. A method for a first computing device to distribute audio streams to a plurality of computing devices of participants in a communication session conducted through a network connecting the computing devices, the method comprising:

establishing a star network among the plurality of computing devices with the first computing device as a central network hub;

through the star network, receiving an audio stream from each of the plurality of computing devices;

compositing at least a subset of the audio streams, wherein the compositing comprises generating at least two different composite audio streams for at least two different computing devices, wherein the generating comprises (i) identifying a set of silent participants in the communication session by identifying audio data packets that do not contain audio from silent participants based on identified sizes of audio data packets of each received audio stream, (ii) identifying computing devices to which to send a same composite audio stream based on the identified set of silent participants; and

(iii) generating the same composite audio stream for the identified computing devices, wherein the same composite audio stream does not include audio streams from the set of silent participants; and

sending each computing device the composite audio stream for the device, wherein the same composite audio stream is sent for the identified computing devices.

14. The method of claim 13 further comprising:

receiving audio metadata from a second computing device, the audio metadata comprising a list of other computing devices that the second computing device has muted; and

for the second computing device, specifying the computing devices on the list as silent participants.

15. The method of claim 13 further comprising examining at different times each received audio stream to determine whether a participant who is using the computing device from which the first computing device receives the audio stream is speaking at different times, wherein when the determination is made that the participant is not speaking at a particular time, the participant is added to the set of silent participants.