Dereverberation of multi-channel audio streams

• US 20060210089A1
• Filed: 06/24/2005
• Published: 09/21/2006
• Est. Priority Date: 03/16/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Tashev, Ivan I., Allred, Daniel
• Granted Patent: US 7,844,059 B2
• Time in Patent Office: Days
• US Class Current: 381/66
• CPC Class Codes: G10L 19/008 Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing
• CPC Class Codes: G10L 2021/02082 the noise being echo, reverberation of the speech
• CPC Class Codes: H04S 2420/07 Synergistic effects of band splitting and sub-band processing
• CPC Class Codes: H04S 7/305 Electronic adaptation of stereophonic audio signals to reverberation of the listening space H04S7/301 takes precedence

Abstract

A system and process for dereverberation of multi-channel audio streams is presented which uses reverberation suppression techniques. In general, the present system and process builds a frequency dependent model of the reverberation decay and uses spectral subtraction-based reverberation reduction to achieve the aforementioned suppression. This dereverberation system and process can be used to improve automatic speech recognition (ASR) results with minimal CPU overhead.

Claims

1. A computer-implemented process for dereverberation of a multi-channel audio stream, comprising using a computer to perform the following process actions:

estimating reverberation decay parameters for each of a prescribed number of frequency sub-bands for each audio channel of the multi-channel audio stream assuming a frequency dependent model of the reverberation decay; and

suppressing the reverberation component of each frame of each channel of the audio stream that it is desired to dereverberate via a spectral subtraction-based reverberation reduction using the estimated reverberation decay parameters.

2. The process of claim 1, wherein said reverberation decay parameters comprise a decay time constant and a reverberation-to-signal ratio (RSR).

3. The process of claim 2, wherein the process action of estimating the decay time constant parameter for each of the prescribed number of frequency sub-bands for each audio channel of the multi-channel audio stream, comprises the actions of:

estimating a reverberation time of a space where the audio associated with the audio stream is captured, said reverberation time being defined as the time required for sound levels to decrease by 60 dB;

for each audio channel,

identifying the next portion of the audio stream associated with the channel under consideration that exhibits reverberation but no speech components for a period greater than the estimated reverberation time,

designating the identified portion of the audio stream associated with the channel under consideration as a reverberation period,

for each of the prescribed number of frequency sub-bands,

measuring the energy exhibited in a prescribed number of the frames of the audio stream in the reverberation period for the frequency sub-band under consideration,

establishing an energy equation for each frame of the audio stream in the reverberation period for the frequency sub-band under consideration, whose energy has been measured and which was captured after a second prescribed number of the frames in the reverberation period, to produce a system of energy equations,

solving the system of energy equations to establish values for a reverberation energy factor, a noise floor energy and the decay time constant parameter for the frequency sub-band and channel under consideration.

4. The process of claim 3, wherein the process action of establishing an energy equation, comprises a process action of establishing the equation S(k)=A.exp(−

kT/{tilde over (τ

)})+B where S(k) is the energy of the frequency sub-band under consideration measured for frame k where k ranges between the first frame in the reverberation period following the initial number of frames in which it is not desired to suppress the reverberation and the total number of frames in the period which is equal to said reverberation time divided by a frame duration T, and where A is the unknown reverberation energy factor, B is the unknown noise floor energy, and {tilde over (τ

)} is the unknown decay time constant parameter.

5. The process of claim 3, wherein the process action of estimating the RSR parameter for each of a prescribed number of frequency sub-bands for each audio channel of the multi-channel audio stream, comprises an action of, for each frequency sub-band and audio channel, computing the RSR as the reverberation energy factor divided by the energy measured for a frame of the audio stream in the reverberation period for the frequency sub-band and audio channel under consideration that was captured a third prescribed number of frames prior to the frame under consideration.

6. The process of claim 2, wherein the process action of suppressing the reverberation component of each frame of each channel of the audio stream that it is desired to dereverberate, comprises the actions of:

computing a reverberation reduction factor which controls the amount of reverberation suppression imposed;

computing a reverberation energy for each of a group of frequencies of interest; and

suppressing the reverberation component for each frequency of interest using the reverberation reduction factor, and reverberation energy established for the frequency of interest under consideration.

7. The process of claim 6, wherein the process action of computing the reverberation reduction factor, comprises the actions of:

setting the reverberation factor to 1 whenever λ

{overscore (α

)}_n−

χ

is greater than 1, wherein {overscore (α

)}_n is the average momentary reverberation-to-signal ratio of the frame n under consideration, λ

is used to control the {overscore (α

)}_n and is set so that the dereverberation starts when the signal-to-reverberation ratio (SRR) is less than a prescribed dB level wherein SRR is equal to the inverse of the RSR, and χ

is used to set the value of {overscore (α

)}_n at which the reverberation reduction starts and is defined as the average momentary reverberation-to-signal ratio across said frequency sub-bands measured on a clean speech signal;

setting the reverberation factor to 0 whenever λ

{overscore (α

)}_n−

χ

is less than 0; and

setting the reverberation factor to λ

{overscore (α

)}_n−

χ

whenever λ

{overscore (α

)}_n−

χ

falls in a range from 0 to 1.

8. The process of claim 7, wherein the average momentary reverberation-to-signal ratio is computed as

9. The process of claim 7, wherein the process action of computing the reverberation reduction factor further comprises an action of smoothing the reverberation reduction factor prior to suppressing the reverberation components.

10. The process of claim 9, wherein the process action of smoothing the reverberation reduction factor comprises computing the smoothed reverberation reduction factor as

11. The process of claim 10, wherein the process action of smoothing the reverberation reduction factor further comprises initially computing the adaptation time constant, said computation comprising the actions of:

setting the adaptation time constant equal to the prescribed maximum value whenever μ

σ

_R²T is greater than said maximum adaptation time constant value, wherein μ

is an adjustment parameter designed to constrain the decay time constant to a desired deviation of the relative RSR σ

_R²;

setting the adaptation time constant equal to a prescribed minimum value whenever μ

σ

_R²T is less than said minimum adaptation time constant value; and

setting the adaptation time constant equal to μ

σ

_R²T whenever μ

σ

_R²T falls in a range from the minimum adaptation time constant value to the maximum adaptation time constant value.

12. The process of claim 11, wherein the desired deviation of the relative RSR for the frame under consideration σ

_Rn² is defined as

13. The process of claim 9, wherein the process action of suppressing the reverberation component for each frequency of interest, comprises the actions of:

setting the reverberation suppressed signal for the frame under consideration at the frequency of interest under consideration to be the product of the signal associated with the frame under consideration at the frequency of interest under consideration and

$\frac{S_{Y_n}<br><br>\left(f\right)-\mathrm{\beta <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>S_{{\mathrm{ℛ<br><br>}}_s}<br><br>\left(f\right)}{S_{Y_n}<br><br>\left(f\right)},$

whenever S_Yn(f)>

S(f), where S<sub>γ

n</sub> (f) is the energy of the signal for the frame n under consideration and the frequency of interests under consideration, β

is the smoothed reverberation reduction factor of the frame under consideration, S(f) is the reverberation energy of the frame n under consideration and the frequency of interests under consideration; and

setting the reverberation suppressed signal for the frame under consideration at the frequency of interest under consideration to be the product of the signal associated with the frame under consideration at the frequency of interest under consideration and (1−

β

) whenever S<sub>γ

n</sub>(f) is not greater then (f).

14. The process of claim 6, wherein the process action of computing the reverberation energy for each of a group of frequencies of interest, comprises, for each frame at each frequency of interest, the actions of:

for each of the frequency sub-bands,

estimating a momentary decay time constant, and

estimating a momentary RSR parameter;

computing a decay time constant associated with the frame under consideration by linearly interpolating between the previously-computed values of the momentary decay time constant for the frequency sub-bands closest to the frequency of interest under consideration;

computing a RSR parameter associated with the frame under consideration by linearly interpolating between the previously-computed values of the momentary RSR parameter for the frequency sub-bands closest to the frequency of interest under consideration; and

computing the reverberation energy for the frame under consideration as

$S_{\text{\hspace{1em}<br><br>}<br><br>{\mathrm{ℛ<br><br>}}_{\text{\hspace{1em}<br><br>}<br><br>s}}<br><br>\left(f\right)=\mathrm{\alpha <br><br>}<br><br>\left(f\right)<br><br>S_{\text{\hspace{1em}<br><br>}<br><br>Y_{\text{\hspace{1em}<br><br>}<br><br>n<br><br>\text{\hspace{1em}<br><br>}-\text{\hspace{1em}<br><br>}<br><br>N}}<br><br>\left(f\right)<br><br>{\mathrm{e<br><br>}}^{-\text{\hspace{1em}<br><br>}<br><br>\frac{\mathrm{NT}}{\text{\hspace{1em}<br><br>}<br><br>\mathrm{\tau <br><br>}<br><br>\left(f\right)}},$

wherein S(f) is the reverberation energy of the frame n under consideration and the frequency of interests under consideration, α

(f) is the estimated momentary RSR parameter of the frame under consideration at the frequency of interest under consideration, τ

(f) is the estimated momentary decay time constant of the frame under consideration at the frequency of interest under consideration, T is the frame duration, N is the number of frames in a prescribed reverberation period for which it is not desired to suppress the reverberation, and S_Yn-N (f) is the energy measured for a previous frame captured N frames back from the frame under consideration at the frequency of interest under consideration.

15. The process of claim 14, wherein the process action of estimating the momentary decay time constant for each frame at each frequency sub-band, comprises the actions of:

computing an adaptation time constant which controls how fast the reverberation decay parameters are allowed to change in response to reverberation changes; and

estimating the momentary decay time constant for the frame under consideration at the frequency sub-band under consideration as

${\mathrm{\tau <br><br>}}_{\text{\hspace{1em}<br><br>}<br><br>n}<br><br>\left(l\right)={\mathrm{\tau <br><br>}}_{\text{\hspace{1em}<br><br>}<br><br>n<br><br>\text{\hspace{1em}<br><br>}-\text{\hspace{1em}<br><br>}<br><br>1}<br><br>\left(l\right)+\frac{T}{\text{\hspace{1em}<br><br>}<br><br>{\mathrm{\tau <br><br>}}_{\text{\hspace{1em}<br><br>}<br><br>A}}<br><br>\left[{\text{\hspace{1em}<br><br>}<br><br>\widetilde{\mathrm{\tau <br><br>}}}_n<br><br>\left(l\right)-{\mathrm{\tau <br><br>}}_{\text{\hspace{1em}<br><br>}<br><br>n<br><br>\text{\hspace{1em}<br><br>}-\text{\hspace{1em}<br><br>}<br><br>1}<br><br>\left(l\right)\right],$

wherein τ

_n(l) is the momentary decay time constant for the frame under consideration n at frequency sub-band under consideration l, τ

_n-1(l) is the momentary decay time constant for the frame immediately preceding the frame under consideration at frequency sub-band under consideration, τ

_A is the adaptation time constant, and {tilde over (τ

)}_n(l) is said decay time constant for the frame under consideration at frequency sub-band under consideration.

16. The process of claim 15, wherein the process action of estimating the momentary RSR parameter for each frame at each frequency sub-band, comprises an action of estimating the momentary decay time constant for the frame under consideration at the frequency sub-band under consideration as

17. The process of claim 16, wherein the process action of computing the adaptation time constant, comprises the actions of:

setting the adaptation time constant equal to a prescribed maximum value whenever μ

σ

_R²T is greater than said maximum adaptation time constant value, wherein μ

is an adjustment parameter designed to constrain the decay time constant to a desired deviation of the relative RSR σ

_R²;

setting the adaptation time constant equal to a prescribed minimum value whenever μ

σ

_R²T is less than said minimum adaptation time constant value; and

setting the adaptation time constant equal to μ

σ

_R²T whenever μ

σ

_R²T falls in a range from the minimum adaptation time constant value to the maximum adaptation time constant value.

18. The process of claim 17, wherein the desired deviation of the relative RSR for the frame under consideration σ

_Rn² is defined as

19. A computer-readable medium having computer-executable instructions for performing the process actions recited in claim 1.

20. A system for suppressing reverberation in a multi-channel audio stream, comprising:

a general purpose computing device;

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

estimate reverberation decay parameters for each of a prescribed number of frequency sub-bands for each audio channel of the multi-channel audio stream assuming a frequency dependent model of the reverberation decay, and

suppress the reverberation component of each frame of each channel of the audio stream that it is desired to dereverberate via a spectral subtraction-based reverberation reduction using the estimated reverberation decay parameters.