Audio processing in a multi-participant conference

• US 8,711,736 B2
• Filed: 09/16/2010
• Issued: 04/29/2014
• Est. Priority Date: 09/16/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Garcia, Roberto Jr., Jeong, Hyeonkuk, Patel, Nirav, Tung, Berkat S., Abuan, Joe S.
• Primary Examiner(s): Sheikh, Ayaz
• Assistant Examiner(s): Moutaouakil, Mounir
• Application Number:
  US12/884,129
• Publication Number:
  US 20120069134A1
• Time in Patent Office: 1,321 Days
• Field of Search: 370259-271, 379157-158, 37920201-20501
• US Class Current: 370/260
• CPC Class Codes: H04M 3/561 by multiplexing
• CPC Class Codes: H04M 3/568 audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

A first computing device distributes audio signals to several computing devices of participants in a communication session. In some embodiments, the first computing device serves as a central distributor for receiving audio signals from other computing devices, compositing the audio signals and distributing the composited audio signals to the other computing devices. The first computing device prioritizes the received audio signals based on a set of criteria and selects several highly prioritized audio signals. The first computing device generates composite audio signals using only the selected audio signals. The first computing device sends each computing device the composited audio signal for the device. In some cases, the first computing device sends a selected audio signal to another computing device without mixing the signal with any other audio signal.

Claims

1. A focus point module for execution on a focus computing device that engages in a multi-participant communication session with a plurality of non-focus computing devices, the focus point module receiving audio streams from the plurality of non-focus computing devices and from the focus computing device, the focus point module comprising:

a device assessor for performing an operation to identify, based on a first assessment of computational resources of the focus computing device, a first particular number that is less than a total number of received audio streams, wherein the first particular number represents a maximum number of audio streams for compositing by the focus computing device;

a signal assessment module for selecting the first particular number of the received audio streams for compositing in order to reduce an amount of the computational resources of the focus computing device that is used for the compositing; and

an audio processing module for generating a set of composite streams from the selected audio streams, wherein each participant computing device receives a composite stream from the set of composite streams during the communication session,

wherein the device assessor performs the operation during the communication session to identify, based on a second assessment of the computational resources of the focus computing device, a second particular number that represents the maximum number of audio streams for compositing by the focus computing device, wherein the signal assessment module selects the second particular number of the received audio streams for compositing, wherein the second particular number is different from the first particular number and the first and second particular numbers can vary from session to session.

2. The focus point module of claim 1, wherein a number of generated composite streams is less than the total number of participant computing devices in the communication session and at least two participant computing devices receive a same composite stream.

3. The focus point module of claim 1, wherein a number of generated composite streams is different from the first particular number.

4. The focus point module of claim 1, wherein the signal assessment module determines which received audio streams to select for compositing based on a set of criteria.

5. The focus point module of claim 4, wherein the set of criteria relates to attributes of audio content in the received audio streams.

6. The focus point module of claim 5, wherein the attributes of the audio content comprise volume levels of the audio content in the received audio streams.

7. The focus point module of claim 5, wherein the attributes of the audio content comprise volume levels of the audio content in the received audio streams and durations that the volume levels exceed a threshold.

8. The focus point module of claim 5, wherein the attributes of the audio content are received as a form of metadata along the received audio streams from the participant computing devices in the communication session.

9. The focus point module of claim 3, wherein the set of criteria relates to non-audible inputs provided by the participant computing devices in the communication session.

10. The focus point module of claim 9, wherein the non-audible inputs comprise a particular device that is muted by the rest of the participant computing devices.

11. The focus point module of claim 10, wherein the number of generated composite streams is less than the total number of participant computing devices in the communication session because of non-audible inputs provided by the participant computing devices in the communication session.

12. The focus point module of claim 9, wherein the non-audible inputs are received as a form of metadata along the received audio streams from the participant computing devices in the communication session.

13. The focus point module of claim 4, wherein the signal assessment module changes the selected set of received audio streams for compositing when behavior of the participants during the communication session changes.

14. A method for a focus computing device that engages in a multi-participant communication session with a plurality of non-focus computing devices, the method comprising:

receiving audio streams from the plurality of non-focus computing devices and from the focus computing device;

based on a first assessment of computational resources of the focus computing device, identifying a first number that represents a maximum number of audio streams for compositing by the focus computing device;

selecting the first number of received audio streams for the focus computing device to process, the first number being less than a total number of participant computing devices;

generating at least two different composite audio streams for at least two different participant computing devices using no more than the selected number of received audio streams;

sending each participant computing device the composite audio stream for that device;

based on a second assessment of the computational resources of the focus computing device during the communication session, identifying a second number that represents the maximum number of audio streams for compositing by the focus computing device, wherein the second number is different from the first number and the first and second numbers can vary from session to session; and

selecting the second number of received audio streams for the focus computing device to process.

15. The method of claim 14 further comprising changing the set of received audio streams that are selected for compositing based on changing behavior of participants during the communication session.

16. The method of claim 14, wherein said selecting is further based on a set of criteria.

17. The method of claim 16, wherein the set of criteria relates to attributes of audio content in the received audio streams.

18. The method of claim 17, wherein the attributes of audio content comprise volume levels of the audio content in the received audio streams.

19. The method of claim 17, wherein the attributes of the audio content comprise durations that the volume levels exceed a threshold.

20. The method of claim 16, wherein the set of criteria relates to non-audible inputs provided by the participant computing devices in the communication session.

21. The method of claim 20, wherein the non-audible inputs comprise a participant computing device that is muted by the rest of the participant computing devices.

22. A non-transitory computer readable medium of a focus computing device that engages in a multi-participant communication session with a plurality of non-focus computing devices, the focus computing device receiving audio streams from the plurality of non-focus computing devices and from a participant using the focus computing device, the computer readable medium storing a computer program that is executable by at least one processing unit, the computer program comprising sets of instructions for:

based on a first assessment of computational resources of the focus computing device, identifying a first number that represents a maximum number of audio streams for compositing by the focus computing device;

selecting a first subset of the received audio streams for compositing, wherein the first subset contains the first number of the received audio streams, wherein the first number is less than a total number of participant computing devices in order to reduce an amount of the computational resources of the focus computing device that is used for compositing;

generating a set of composite streams from the selected audio streams, wherein each participant computing device receives a composite stream from the set of composite streams during the communication session;

based on a second assessment of the computational resources of the focus computing device during the communication session, identifying a second number that represents the maximum number of audio streams for compositing by the focus computing device, wherein the second number is different than the first number and the first and second numbers can vary from session to session; and

selecting a second subset of the received audio streams for compositing, wherein the second subset contains the second number of the received audio streams.

23. The non-transitory computer-readable medium of claim 22, wherein the computer program further comprises a set of instructions for determining which received audio streams to select for compositing based on a set of criteria.

24. The non-transitory computer-readable medium of claim 23, wherein the set of criteria relates to attributes of audio content in the received audio streams.

25. The non-transitory computer-readable medium of claim 23, wherein the set of criteria relates to non-audible inputs provided by the participant computing devices in the communication session.