Spectral shaping for audio mixing

• US 9,338,553 B2
• Filed: 08/03/2014
• Issued: 05/10/2016
• Est. Priority Date: 04/18/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Lovitt, Andrew W.
• Primary Examiner(s): King, Simon
• Application Number:
  US14/450,257
• Publication Number:
  US 20140334641A1
• Time in Patent Office: 646 Days
• Field of Search: 381/98, 381/119
• US Class Current: 1/1
• CPC Class Codes: G10L 19/008 Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing
• CPC Class Codes: G10L 2021/02087 the noise being separate speech, e.g. cocktail party
• CPC Class Codes: G10L 21/02 Speech enhancement, e.g. noise reduction or echo cancellation reducing echo effects in line transmission systems H04B3/20; echo suppression in hands-free telephones H04M9/08
• CPC Class Codes: G10L 21/0232 Processing in the frequency domain
• CPC Class Codes: G11B 27/038 Cross-faders therefor
• CPC Class Codes: H04M 2203/5072 Multiple active speakers conducting a computer conference, e.g. Admission detection H04L12/1822
• CPC Class Codes: H04M 3/568 audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30
• CPC Class Codes: H04R 2430/00 Signal processing covered by H04R, not provided for in its groups
• CPC Class Codes: H04R 3/04 for correcting frequency response

Abstract

Techniques are described herein that are capable of spectrally shaping audio signal(s) for audio mixing. Spectrally shaping an audio signal means modifying a frequency spectrum of the audio signal. A frequency spectrum of an audio signal is a representation of the audio signal in the frequency domain. For instance, a frequency spectrum may be represented using multiple frequency bands. The frequency spectrum may be modified by modifying characteristic(s) (e.g., magnitude, phase, etc.) one or more of the frequency bands.

Claims

1. A system comprising:

an analyzer configured to analyze a first frequency spectrum of a first audio signal to determine a first portion of the first frequency spectrum that has a magnitude that is lower than magnitudes of other portions of the first frequency spectrum;

a modification module configured to modify the first frequency spectrum to decrease the magnitude of the first portion of the first frequency spectrum with respect to the other portions of the first frequency spectrum in response to the magnitude of the first portion of the first frequency spectrum being lower than the magnitudes of the other portions of the first frequency spectrum,

the modification module further configured to modify a second frequency spectrum of a second audio signal to increase a magnitude of a first portion of the second frequency spectrum with respect to other portions of the second frequency spectrum in response to the magnitude of the first portion of the first frequency spectrum being lower than the magnitudes of the other portions of the first frequency spectrum, the first portion of the first frequency spectrum and the first portion of the second frequency spectrum corresponding to a common set of frequencies; and

a mixer configured to mix the first audio signal and the second audio signal to provide a mixed audio signal.

2. The system of claim 1, wherein the analyzer is further configured to compare selection criteria that are based on the first frequency spectrum to a plurality of frequency spectrums of a plurality of respective designated audio signals to determine which frequency spectrum of the plurality of frequency spectrums satisfies the selection criteria to a greater extent than other frequency spectrums in the plurality of frequency spectrums;

and

wherein the system further comprises;

a selection module configured to select the second audio signal from the plurality of designated audio signals to be mixed with the first audio signal based on the second frequency spectrum satisfying the selection criteria to a greater extent than other frequency spectrums in the plurality of frequency spectrums.

3. The system of claim 1, wherein the first audio signal has a specified perceptual characteristic;

and

wherein the system further comprises;

a gain module configured to apply a first gain to the first audio signal and a second gain to the second audio signal, such that the mixed audio signal has the specified perceptual characteristic.

4. The system of claim 1, wherein a first time period, a second time period, a third time period, a fourth time period, and a fifth time period occur in consecutive temporal order;

and

wherein the mixer comprises;

a provision module configured to provide the first audio signal having a first gain equal to a first upper threshold for a duration of the first time period and for a duration of the fifth time period,

the provision module further configured to provide the second audio signal having a second gain equal to a second lower threshold for the duration of the first time period and for the duration of the fifth time period,

the provision module further configured to provide the first audio signal having the first gain equal to a first lower threshold for a duration of the third time period,

the provision module further configured to provide the second audio signal having the second gain equal to a second upper threshold for the duration of the third time period; and

a gain module configured to smoothly decrease the first gain from the first upper threshold to the first lower threshold during the second time period,

the gain module further configured to smoothly increase the second gain from the second lower threshold to the second upper threshold during the second time period,

the gain module further configured to smoothly increase the first gain from the first lower threshold to the first upper threshold during the fourth time period,

the gain module further configured to smoothly decrease the second gain from the second upper threshold to the second lower threshold during the fourth time period.

5. The system of claim 1, wherein the common set of frequencies represents disjoint frequency ranges.

6. The system of claim 1, wherein the modification module is configured to decrease the magnitude of the first portion of the first frequency spectrum to approximately zero.

7. A method comprising:

analyzing a frequency spectrum of a first audio signal to determine frequency spectrum criteria for selection of a second audio signal to be mixed with the first audio signal;

comparing the frequency spectrum criteria to a plurality of frequency spectrums of a plurality of respective designated audio signals, using at least one processor, to determine a first frequency spectrum of the plurality of frequency spectrums that satisfies the frequency spectrum criteria to a greater extent than other frequency spectrums in the plurality of frequency spectrums; and

selecting a second audio signal from the plurality of designated audio signals to be mixed with the first audio signal based on the first frequency spectrum corresponding to the second audio signal.

8. The method of claim 7, wherein analyzing the frequency spectrum comprises:

analyzing the frequency spectrum of the first audio signal to determine at least one frequency band in the frequency spectrum of the first audio signal that has a lesser magnitude than other frequency bands in the frequency spectrum of the first audio signal; and

determining the frequency spectrum criteria to indicate a greater magnitude in the at least one frequency band than in the other frequency bands.

9. The method of claim 8, wherein the at least one frequency band represents disjoint frequency ranges.

10. The method of claim 7, further comprising:

modifying the first frequency spectrum based on the frequency spectrum of the first audio signal to provide a modified first frequency spectrum; and

mixing the second audio signal having the modified first frequency spectrum and the first audio signal to provide a mixed audio signal.

11. The method of claim 7, further comprising:

modifying the frequency spectrum of the first audio signal based on the first frequency spectrum to provide a modified frequency spectrum; and

mixing the first audio signal having the modified frequency spectrum and the second audio signal to provide a mixed audio signal.

12. The method of claim 7, wherein analyzing the frequency spectrum of the first audio signal comprises:

analyzing the frequency spectrum of the first audio signal having a specified perceptual characteristic; and

wherein the method further comprises;

applying a first gain to the first audio signal and a second gain to the second audio signal; and

mixing the first audio signal and the second audio signal to provide a mixed audio signal having the specified perceptual characteristic in response to applying the first gain to the first audio signal and the second gain to the second audio signal.

13. The method of claim 7, further comprising:

mixing the first audio signal and the second audio signal for a duration that includes a first time period, a second time period, a third time period, a fourth time period, and a fifth time period that occur in consecutive temporal order;

wherein mixing the first audio signal and the second audio signal comprises;

providing the first audio signal having a first gain equal to a first upper threshold for a duration of the first time period and for a duration of the fifth time period;

providing the second audio signal having a second gain equal to a second lower threshold for the duration of the first time period and for the duration of the fifth time period;

smoothly decreasing the first gain from the first upper threshold to a first lower threshold during the second time period;

smoothly increasing the second gain from the second lower threshold to a second upper threshold during the second time period;

providing the first audio signal having the first gain equal to the first lower threshold for a duration of the third time period;

providing the second audio signal having the second gain equal to the second upper threshold for the duration of the third time period;

smoothly increasing the first gain from the first lower threshold to the first upper threshold during the fourth time period; and

smoothly decreasing the second gain from the second upper threshold to the second lower threshold during the fourth time period.

14. A system comprising:

an analyzer configured to analyze a frequency spectrum of a first audio signal to determine frequency spectrum criteria for selection of a second audio signal to be mixed with the first audio signal;

a comparison module configured to compare the frequency spectrum criteria to a plurality of frequency spectrums of a plurality of respective designated audio signals to determine a first frequency spectrum of the plurality of frequency spectrums that satisfies the frequency spectrum criteria to a greater extent than other frequency spectrums in the plurality of frequency spectrums; and

a selection module configured to select a second audio signal from the plurality of designated audio signals to be mixed with the first audio signal based on the first frequency spectrum corresponding to the second audio signal.

15. The system of claim 14, wherein the analyzer is configured to analyze the frequency spectrum of the first audio signal to determine at least one frequency band in the frequency spectrum of the first audio signal that has a lesser magnitude than other frequency bands in the frequency spectrum of the first audio signal;

and

wherein the analyzer is configured to determine the frequency spectrum criteria to indicate a greater magnitude in the at least one frequency band than in the other frequency bands.

16. The system of claim 15, wherein the at least one frequency band represents disjoint frequency ranges.

17. The system of claim 14, further comprising:

a modification module configured to modify the first frequency spectrum based on the frequency spectrum of the first audio signal to provide a modified first frequency spectrum; and

a mixer configured to mix the second audio signal having the modified first frequency spectrum and the first audio signal to provide a mixed audio signal.

18. The system of claim 14, further comprising:

a modification module configured to modify the frequency spectrum of the first audio signal based on the first frequency spectrum to provide a modified frequency spectrum; and

a mixer configured to mix the first audio signal having the modified frequency spectrum and the second audio signal to provide a mixed audio signal.

19. The system of claim 14, wherein the frequency spectrum of the first audio signal has a specified perceptual characteristic;

and

wherein the system further comprises;

a gain module configured to apply a first gain to the first audio signal and a second gain to the second audio signal; and

a mixer configured to mix the first audio signal and the second audio signal to provide a mixed audio signal having the specified perceptual characteristic in response to application of the first gain to the first audio signal and the second gain to the second audio signal.

20. The system of claim 14, further comprising:

a mixer configured to mix the first audio signal and the second audio signal for a duration that includes a first time period, a second time period, a third time period, a fourth time period, and a fifth time period that occur in consecutive temporal order;

wherein the mixer comprises a provision module and a gain module,

the provision module configured to provide the first audio signal having a first gain equal to a first upper threshold for a duration of the first time period and for a duration of the fifth time period,

the provision module further configured to provide the second audio signal having a second gain equal to a second lower threshold for the duration of the first time period and for the duration of the fifth time period,

the gain module configured to smoothly decrease the first gain from the first upper threshold to a first lower threshold during the second time period,

the gain module further configured to smoothly increase the second gain from the second lower threshold to a second upper threshold during the second time period,

the provision module further configured to provide the first audio signal having the first gain equal to the first lower threshold for a duration of the third time period,

the provision module further configured to provide the second audio signal having the second gain equal to the second upper threshold for the duration of the third time period,

the gain module further configured to smoothly increase the first gain from the first lower threshold to the first upper threshold during the fourth time period,

the gain module further configured to smoothly decrease the second gain from the second upper threshold to the second lower threshold during the fourth time period.