Audio mixing method, apparatus and system

• US 9,456,273 B2
• Filed: 03/26/2014
• Issued: 09/27/2016
• Est. Priority Date: 10/13/2011
• Status: Active Grant

• Current Assignee: Huawei Device Company Limited (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Huawei Device Company Limited (Huawei Investment & Holding Co., Ltd.)
• Inventors: Wang, Dongqi, Zhan, Wuzhou
• Primary Examiner(s): Kuntz, Curtis
• Assistant Examiner(s): Truong, Kenny
• Application Number:
  US14/225,536
• Publication Number:
  US 20140205115A1
• Time in Patent Office: 916 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: H04L 12/18 for broadcast or conference , e.g. multicast
• CPC Class Codes: H04L 12/1822 Conducting the conference, e.g. admission, detection, selection or grouping of participants, correlating users to one or more conference sessions, prioritising transmission
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/4038 with floor control
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/764 at the destination reformatting of additional data in video clients H04N21/4355
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04M 3/568 audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04R 3/00 Circuits for transducers , loudspeakers or microphones
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407

Abstract

An audio mixing method, apparatus and system, which can ensure sound quality after audio mixing and reduce consumption of computing resources. The method includes: receiving an audio stream of each site, and analyzing the audio stream of each site to obtain a sound characteristic value of a sound source object; selecting, according to a descending sequence of sound characteristic values of sound source objects, a predetermined number of sound source objects from the sound source objects to serve as main sound source objects; determining, according to a relationship between a target site and the sites where the main sound source objects are located, audio streams that require audio mixing for the target site; and performing audio mixing on the audio streams that require audio mixing for the target site and sending the audio streams after the audio mixing to the target site.

Claims

1. An audio mixing method, comprising:

receiving, from a first site, a first audio stream that comprises a first plurality of sound source objects;

receiving, from a second site, a second audio stream that comprises a second plurality of sound source objects;

receiving, from a third site, a third audio stream that comprises a third plurality of sound source objects;

analyzing the first audio stream to obtain a sound characteristic value of each of the plurality of sound source objects;

analyzing the second audio stream to obtain a sound characteristic value of each of the second plurality of sound source objects;

analyzing the third audio stream to obtain a sound characteristic value of each of the third plurality of sound source objects;

selecting, according to a descending sequence of sound characteristic values of the first plurality of sound source objects, the second plurality of sound source objects, and the third plurality of sound source objects, a predetermined number of sound source objects from the first plurality of sound source objects, the second plurality of sound source objects, and the third plurality of sound source objects to serve as main sound source objects,

wherein a portion, but not all, of each of the first plurality of sound source objects, the second plurality of sound source objects, and the third plurality of sound source objects are selected as main sound source objects for audio mixing, and

wherein a sound source object that is not selected as a main sound source object in an audio stream that contains a main sound source object is muted without muting the main sound source object or the audio stream;

determining each site selected from among the first site, the second site, and the third site from which each of the main sound source objects were received in an audio stream;

determining a target site selected from among the first site, the second site, and the third site;

determining audio streams that require audio mixing as audio streams received from each site selected from among the first site, the second site, and the third site, but not the target site, from which each of the main sound source objects were received in the audio stream; and

either performing audio mixing on the determined audio streams that require audio mixing for the target site, and sending the audio streams after the audio mixing to the target site;

or

sending to the target site the determined audio streams that require audio mixing for the target site to perform audio mixing in the target site.

2. The audio mixing method according to claim 1, wherein analyzing an audio stream to obtain the sound characteristic value of a sound source object comprises:

decoding the audio stream; and

calculating the sound characteristic value of the sound source object.

3. The audio mixing method according to claim 1, wherein analyzing an audio stream to obtain the sound characteristic value of a sound source object comprises extracting the sound characteristic value of the sound source object from the audio stream.

4. The audio mixing method according to claim 1, wherein performing the audio mixing on the determined audio streams that require audio mixing for the target site comprises:

separating the main sound source objects from the determined audio streams; and

performing audio mixing on the main sound source objects according to a relationship between the target site, the first site, the second site, and the third site.

5. The audio mixing method according to claim 4, wherein performing audio mixing on the main sound source objects according to the relationship between the target site, the first site, the second site, and the third site comprises performing audio mixing on the main sound source objects that were not received from the target site.

6. The audio mixing method according to claim 4, wherein performing audio mixing on the main sound source objects according to the relationship between the target site, the first site, the second site, and the third site comprises performing audio mixing on all the main sound source objects when the target site is not one of the sites from which the main sound source objects were received.

7. The audio mixing method according to claim 1, further comprising performing, on a terminal device in the target site, audio mixing on the determined audio streams that require audio mixing for the target site.

8. An audio mixing apparatus comprising:

a computer processor configured to;

receive an audio stream from a first site, a second site, and a third site, wherein each audio stream comprises a plurality of sound source objects;

analyze the audio stream of the first site, the second site, and the third site to obtain a sound characteristic value of each sound source object;

select, according to a descending sequence of sound characteristic values of the sound source objects, a predetermined number of the sound source objects to serve as main sound source objects, wherein a portion, but not all, of the plurality of sound source objects are selected as main sound source objects for audio mixing, and wherein a sound source object that is not selected as a main sound source object in an audio stream that contains a main sound source object is muted without muting the main sound source object or the audio stream;

determine each site selected from among the first site, the second site, and the third site from which each of the main sound source objects were received in an audio stream;

determine a target site selected from among the first site, the second site, and the third site; and

determine audio streams that require audio mixing as audio streams received from each site selected from among the first site, the second site, and the third site, but not the target site, from which each of the main sound source objects were received in the audio stream; and

a transmitter coupled to the computer processor and configured to either;

perform audio mixing on the determined audio streams that require audio mixing for the target site, and send the audio stream after the audio mixing to the target site;

or

send, to the target site, the determined audio streams that require audio mixing for the target site to perform audio mixing in the target site.

9. The audio mixing apparatus according to claim 8, wherein the computer processor is further configured to:

decode each received audio stream; and

calculate the sound characteristic value of each sound source object of each received audio stream.

10. The audio mixing apparatus according to claim 8, wherein the computer processor is further configured to extract the sound characteristic value of each sound source object from each received audio stream.

11. The audio mixing apparatus according to claim 8, wherein the computer processor is further configured to:

separate the main sound source objects from the determined audio streams; and

perform audio mixing on the main sound source objects according to a relationship between the target site, the first site, the second site, and the third site.

12. The audio mixing apparatus according to claim 11, wherein the computer processor is further configured to determine whether a main sound source object was received from the target site, and wherein the transmitter is further configured to perform audio mixing on the main sound source objects except main sound source objects received from the target site when the computer processor determines that a main sound source object was received from the target site.

13. The audio mixing apparatus according to claim 11, wherein the computer processor is further configured to determine whether a main sound source object was received from the target site, and wherein the transmitter is further configured to perform audio mixing on all the main sound source objects when the computer processor determines that a main sound source object was not received from the target site.

14. An audio mixing system, comprising:

a site terminal configured to;

collect an audio signal from at least one sound source object;

perform spatial audio object coding on the collected audio signal to form a down-mixed audio stream;

send the down-mixed audio stream and a spatial side information to an audio mixing apparatus, wherein the spatial side information comprises;

a maximum energy value among energy values of each sound source object in the down-mixed audio stream; and

a ratio of an energy value of each sound source object in the down-mixed audio stream to the maximum energy value among energy values of each sound source object in the down-mixed audio stream,

wherein the audio mixing apparatus comprises a computer processor configured to;

receive a down-mixed audio stream and spatial side information from a plurality of site terminals, wherein at least some of the plurality of site terminals comprise a plurality of sound source objects; and

analyze the down-mixed audio stream and spatial side information of each site terminal to obtain a sound characteristic value of each sound source object corresponding to each site terminal;

select, according to a descending sequence of sound characteristic values of the first plurality of sound source objects, the second plurality of sound source objects, and the third plurality of sound source objects, a predetermined number of sound source objects from the first plurality of sound source objects, the second plurality of sound source objects, and the third plurality of sound source objects to serve as main sound source objects, wherein a portion, but not all, of each of the first plurality of sound source objects, the second plurality of sound source objects, and the third plurality of sound source objects are selected as main sound source objects for audio mixing;

select, according to a descending sequence of sound characteristic values of sound source objects, a predetermined number of sound source objects from the sound source objects to serve as main sound source objects;

determine the site terminals where the main sound source objects are located; and

determine, according to a relationship between a target site terminal and the site terminals where the main sound source objects are located, the down-mixed audio streams that require audio mixing for the target site terminal, and wherein the computer processor is further configured to either;

perform audio mixing on the determined down-mixed audio streams that require audio mixing for the target site terminal to form a mixed audio stream, and send the mixed audio stream after the audio mixing to the target site terminal;

or

send, to the target site terminal via a transmitter, the determined down-mixed audio streams that require audio mixing for the target site terminal to perform audio mixing.