Distributed audio mixing and forwarding

• US 9,591,037 B2
• Filed: 10/30/2013
• Issued: 03/07/2017
• Est. Priority Date: 10/30/2013
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Wu, Duanpei, Liu, Yanghua
• Primary Examiner(s): Hussain, Tauqir
• Assistant Examiner(s): Mohammadi, Kamran
• Application Number:
  US14/066,841
• Publication Number:
  US 20150120830A1
• Time in Patent Office: 1,224 Days
• Field of Search: 381/119, 381/66
• US Class Current: 1/1
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 65/80 Responding to QoS

Abstract

A scalable media conferencing system minimizes delay while maintaining a consistent session across all of the distributed media servers handling the conference session. Each of the media servers sends a list of the local top streams and their importance. A global list of top streams is created and disseminated to all of the media servers. Each of the media servers then forwards any local stream that is on the list of global top streams to all of the other media servers. Once a media server receives all of the streams on the list of global top streams, it locally mixes the streams together and outputs the mixed stream to the endpoints that are locally connected to that media server.

Claims

1. A method comprising:

at a first server among a plurality of servers in communication with each other and each of which processes multimedia signals associated with a conference session, receiving a plurality of local multimedia streams, each local multimedia stream having a corresponding indication of local importance, wherein each local multimedia stream in the plurality of local multimedia streams comprises a local video stream and a local audio stream;

selecting a plurality of local top multimedia streams among the plurality of local multimedia streams based on the corresponding indications of local importance;

creating a list of the local top multimedia streams with the corresponding indications of local importance;

receiving lists of remote top multimedia streams from one or more other servers among the plurality of servers, each list of remote top multimedia streams being based on indications of local importance corresponding to one of the other servers;

determining whether the first server is a primary server responsible for generating a list of global top multimedia streams;

responsive to a determination that the first server is the primary server, generating the list of global top multimedia streams by selecting from the list of local top multimedia streams and at least one of the lists of remote top multimedia streams based on the indications of local importance;

forwarding the list of global top multimedia streams to the one or more other servers;

determining which of the global top multimedia streams are selected from the local multimedia streams and which of the global top multimedia streams are selected from the remote top multimedia streams to be included in the list of global top multimedia streams;

receiving at the first server, from one or more other servers different from the first server, any remote top multimedia streams that are determined to be in the list of global top multimedia streams, wherein each of the remote top multimedia streams comprises a remote video stream and a remote audio stream;

at the first server, mixing the local audio streams corresponding to the local multimedia streams that are determined to be on the list of global top multimedia streams with the remote audio streams corresponding to the received remote top multimedia streams that are determined to be on the list of global top multimedia streams to create a mixed multimedia stream comprising a mixed audio stream and a plurality of global video streams; and

transmitting the mixed multimedia stream to at least one endpoint device associated with the first server.

2. The method of claim 1, further comprising transmitting, from the first server to one or more of the other servers, the list of local top multimedia streams with the corresponding indications of local importance.

3. The method of claim 1, further comprising forwarding, from the first server to the other servers, the local multimedia streams that are determined to be on the list of global top multimedia streams, and not forwarding, from the first server to the other servers, the local multimedia streams that are determined not to be on the list of top global multimedia streams.

4. The method of claim 1, further comprising the first server mixing the local multimedia streams that are determined to be on the list of global top multimedia streams with the received remote top multimedia streams that are determined to be on the list of global top multimedia streams, to create a fully mixed multimedia stream, and transmitting the fully mixed multimedia stream to the at least one endpoint device associated with the first server.

5. The method of claim 1, further comprising forwarding, from the first server to the one or more other servers, the local multimedia streams that are determined to be on the list of global top multimedia streams and the received remote top multimedia streams that are determined to be on the list of global top multimedia streams without mixing.

6. The method of claim 1, wherein each of the indications of local importance comprises a loudness of the local audio stream associated with the corresponding local multimedia stream, and wherein each of the indications of remote importance comprises a loudness of the remote audio stream associated with the corresponding remote multimedia stream.

7. An apparatus comprising:

a network interface unit configured to enable communication over a network in order to transmit and receive a plurality of local multimedia streams, each local multimedia stream having a corresponding indication of local importance, and to communicate with a plurality of servers, wherein each local multimedia stream in the plurality of local multimedia streams comprises a local video stream and a local audio stream; and

a processor coupled to the network interface unit, and configured to;

select a plurality of local top multimedia streams among the plurality of local multimedia streams based on the corresponding indications of local importance;

create a list of the local top multimedia streams with the corresponding indications of local importance;

receive via the network interface unit, lists of remote top multimedia streams from one or more other servers among the plurality of servers, wherein each list of remote top multimedia streams is based on indications of local importance corresponding to one of the other servers;

generate a list of global top multimedia streams, selected from the list of local top multimedia streams and at least one list of remote top multimedia streams received through the network interface unit, based on the indications of local importance corresponding to the local top multimedia streams and at least one indication of remote importance corresponding to at least one remote top multimedia stream;

cause the network interface unit to forward the list of global top multimedia streams to the one or more other servers;

determine which of the global top multimedia streams are selected from the local multimedia streams and which of the global top multimedia streams are selected from the remote top multimedia streams to be included in the list of global top multimedia streams;

receive, from one or more other servers of the plurality of servers, any remote top multimedia streams that are determined to be in the list of global top multimedia streams, wherein each of the remote top multimedia streams comprises a remote video stream and a remote audio stream;

mix the local audio streams corresponding to the local multimedia streams that are determined to be on the list of global top multimedia streams with the remote audio streams corresponding to the received remote top multimedia streams that are determined to be on the list of global top multimedia streams to create a mixed multimedia stream comprising a mixed audio stream and a plurality of global video streams, and

cause the network interface unit to transmit the mixed multimedia stream to the at least one endpoint device.

8. The apparatus of claim 7, wherein the processor is further configured to transmit to one or more of the other servers, the list of local top multimedia streams with the corresponding indications of local importance.

9. The apparatus of claim 7, wherein the processor is further configured to mix the local multimedia streams that are determined to be on the list of global top multimedia streams with the received remote top multimedia streams that are determined to be on the list of global top multimedia streams to create a fully mixed multimedia stream, and to transmit the fully mixed multimedia stream to the at least one endpoint device associated with the apparatus.

13. The apparatus of claim 7, wherein the processor is further configured to cause the network interface unit to forward to the one or more other servers, the local multimedia streams that are determined to be on the list of global top multimedia streams and the received remote top multimedia streams that are determined to be on the list of global top multimedia streams without mixing.

14. The apparatus of claim 7, wherein each of the indications of local importance comprises a loudness of the local audio stream associated with the corresponding local multimedia stream, and wherein each of the indications of remote importance comprises a loudness of the remote audio stream associated with the corresponding remote multimedia stream.

10. One or more non-transitory computer readable storage media encoded with software comprising computer executable instructions and when the software is executed operable to cause a processor to:

select a plurality of local top multimedia streams among a plurality of local multimedia streams received at a first server among a plurality of servers, wherein each local multimedia stream has a corresponding indication of local importance, and wherein the selection of the plurality of local top multimedia streams is based on the corresponding indications of local importance, and wherein each local multimedia stream comprises a local video stream and a local audio stream;

create a list of the local top multimedia streams with the corresponding indications of local importance;

receive lists of remote top multimedia streams from one or more other servers among the plurality of servers, wherein each list of remote top multimedia streams is based on indications of local importance corresponding to one of the other servers;

generate a list of global top multimedia streams, selected from the list of local top multimedia streams and at least one list of remote top multimedia streams received through a network interface unit, based on the indications of local importance corresponding to the local top multimedia streams and at least one indication of remote importance corresponding to at least one remote top multimedia stream;

forward the list of global top multimedia streams to the one or more other servers;

determine which of the global top multimedia streams are selected from the local multimedia streams and which of the global top multimedia streams are selected from the remote top multimedia streams to be included in the list of global top multimedia streams;

receive, from one or more other servers, any remote top multimedia streams that are determined to be in the list of global top multimedia streams, wherein each of the received remote top multimedia streams comprises a remote video stream and a remote audio stream;

mix the local audio streams corresponding to the local multimedia streams that are determined to be on the list of global top multimedia streams with the remote audio streams corresponding to the received remote top multimedia streams that are determined to be on the list of global top multimedia streams to create a mixed multimedia stream comprising a mixed audio stream and a plurality of global video streams, and

cause the network interface unit to transmit the mixed multimedia stream to the at least one endpoint device.

11. The one or more non-transitory computer readable storage media of claim 10, wherein the computer executable instructions cause the processor to transmit from the first server to one or more of the other servers, the list of local top multimedia streams with the corresponding indications of local importance.

12. The one or more non-transitory computer readable storage media of claim 10, wherein the computer executable instructions cause the processor to mix the local multimedia streams that are determined to be on the list of global top multimedia streams with the received remote top multimedia streams that are determined to be on the list of global top multimedia streams to create a fully mixed multimedia stream, and transmit the fully mixed multimedia stream to the at least one endpoint device associated with the first server.

15. The one or more non-transitory computer readable storage media of claim 10, wherein the computer executable instructions cause the processor to forward to the one or more other servers, the local multimedia streams that are determined to be on the list of global top multimedia streams and the received remote top multimedia streams that are determined to be on the list of global top multimedia streams without mixing.

16. The one or more non-transitory computer readable storage media of claim 10, wherein each of the indications of local importance comprises a loudness of the local audio stream associated with the corresponding local multimedia stream, and wherein each of the indications of remote importance comprises a loudness of the remote audio stream associated with the corresponding remote multimedia stream.