Method and apparatus for multi-sensory speech enhancement

• US 7,574,008 B2
• Filed: 09/17/2004
• Issued: 08/11/2009
• Est. Priority Date: 09/17/2004
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Zhengyou, Droppo, James G., Acero, Alejandro, Liu, Zicheng, Huang, Xuedong David
• Primary Examiner(s): Chin; Vivian
• Assistant Examiner(s): Olaniran; Fatimat O
• Application Number:
  US10/944,235
• Publication Number:
  US 20060072767A1
• Time in Patent Office: 1,789 Days
• Field of Search: 381 711- 7114, 381 941- 949, 381/122, 381/92, 704/233, 704/208, 704/226, 700/94
• US Class Current: 381/94.7
• CPC Class Codes: G10L 2021/02161 Number of inputs available containing the signal or the noise to be suppressed
• CPC Class Codes: G10L 21/0208 Noise filtering
• CPC Class Codes: H04R 2460/13 Hearing devices using bone conduction transducers
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407

Abstract

A method and apparatus determine a channel response for an alternative sensor using an alternative sensor signal and an air conduction microphone signal. The channel response is then used to estimate a clean speech value using at least a portion of the alternative sensor signal.

Claims

1. A method comprising:

for each time frame of a set of time frames, generating an alternative sensor value representing an alternative sensor signal using an alternative sensor other than an air conduction microphone;

for each time frame of the set of time frames, generating an air conduction microphone value;

identifying which frames in the set of frames do not contain speech from a speaker based on the energy level of the alternative sensor signal;

within the frames identified as not containing speech from the speaker, performing speech detection on the air conduction microphone values to determine which frames contain background speech and which frames do not contain background speech;

using alternative sensor values for the frames identified as not containing speech from the speaker and not containing background speech to determine a variance for noise of the alternative sensor;

using alternative sensor values and air conduction microphone values for the frames identified as not containing speech from the speaker but containing background speech to determine a channel response of the alternative sensor to background speech;

using the alternative sensor values and the air conduction microphone values for the set of time frames to estimate a value for a channel response of the alternative sensor to speech from the speaker; and

using the channel response of the alternative sensor to speech from the speaker, the channel response of the alternative sensor to background speech, and the variance for noise of the alternative sensor to estimate a noise-reduced value for each time frame in the set of time frames.

2. The method of claim 1 wherein estimating a value for a channel response comprises finding an extreme of an objective function.

3. The method of claim 1 further comprising using the estimate of the noise-reduced value to estimate a value for a background speech signal produced by a background speaker.

4. The method of claim 1 wherein estimating a value for the channel response of the alternative sensor to speech from the speaker comprises estimating a single channel response value for all of the time frames in the set of time frames.

5. The method of claim 4 wherein estimating a noise-reduced value comprises estimating a separate noise-reduced value for each time frame in the set of time frames.

6. The method of claim 1 wherein estimating a value for a channel response of the alternative sensor to speech from the speaker comprises estimating the value for a current frame by weighting values for the alternative sensor signal and the air conduction microphone signal in the current frame more heavily than values for the alternative sensor signal and the air conduction microphone signal in a previous frame.

7. A computer-readable storage medium having stored thereon computer-executable instructions that when executed by a processor cause the processor to perform steps comprising:

receiving values for an alternative sensor signal and an air conduction microphone signal for each of a set of time frames, the air conduction microphone signal comprising speech from a speaker and noise;

determining a channel response for a channel from the speaker to an alternative sensor using the values for the entire set of time frames for the alternative sensor signal and the values for the entire set of time frames for the air conduction microphone signal using;

$H=\frac{\begin{array}{c}\underset{t=1}{\overset{T}{\sum <br><br>}}<br><br>\left({\mathrm{\sigma <br><br>}}_z^2<br><br>{<br><br>B_t<br><br>}^2-{\mathrm{\sigma <br><br>}}_w^2<br><br>{<br><br>Y_t<br><br>}^2\right)\pm <br><br>\\ \sqrt{{\left(\underset{t=1}{\overset{T}{\sum <br><br>}}<br><br>\left({\mathrm{\sigma <br><br>}}_z^2<br><br>{<br><br>B_t<br><br>}^2-{\mathrm{\sigma <br><br>}}_w^2<br><br>{<br><br>Y_t<br><br>}^2\right)\right)}^2+}\\ 4<br><br>{\mathrm{\sigma <br><br>}}_z^2<br><br>{\mathrm{\sigma <br><br>}}_w^2<br><br>{<br><br>\underset{t=1}{\overset{T}{\sum <br><br>}}<br><br>B_t^{*}<br><br>Y_t<br><br>}^2\end{array}}{2<br><br>{\mathrm{\sigma <br><br>}}_z^2<br><br>\underset{t=1}{\overset{T}{\sum <br><br>}}<br><br>B_t^{*}<br><br>Y_t}$

where H is the channel response for a channel from the speaker to the alternative sensor, B_t is value of the alternative sensor signal for time frame t, B*_t is the complex conjugate of B_t, |B_t| is the magnitude of B_t, Y_t is the value of the air conduction microphone signal for time frame t, |Y_t| is the magnitude of Y_t, σ

_z² is a variance for noise in the air conduction microphone signal, σ

_w² is a variance for noise in the alternative sensor signal and T is the number of frames in the set of time frames; and

using the channel response and a value for the alternative sensor signal for one time frame in the set of time frames to estimate a clean speech value for the time frame.

8. The computer-readable storage medium of claim 7 wherein the channel response comprises a channel response to a clean speech signal.

9. A method of identifying a clean speech signal, the method comprising:

using an alternative sensor signal from an alternative sensor other than an air conduction microphone to determine periods when a speaker is producing speech and periods when the speaker is not producing speech;

performing speech detection on portions of an air conduction microphone signal associated with the periods when the speaker is not producing speech to identify which portions of the periods are no-speech portions and which portions of the periods are background speech portions;

estimating a noise variance that describes noise in the alternative sensor signal during no-speech portions of the periods;

using the background speech portions of the alternative sensor signal to estimate a background speech channel response for a channel from a background speaker to the alternative sensor;

receiving values for the alternative sensor signal and the air conduction microphone signal for each of a set of time frames;

using the noise variance, the values for the alternative sensor signal for the set of time frames and the values for the air conduction microphone for the set of time frames to estimate a channel response for a channel representing a path from the speaker to an alternative sensor for at least one time frame in the set of time frames; and

using the channel response and the background speech channel response to estimate a value for the clean speech signal for each time frame in the set of time frames that the channel response was estimated from.

10. The method of claim 9 further comprising using the no-speech portions to estimate noise parameters that describe noise in the air conduction microphone signal.

11. The method of claim 9 further comprising determining an estimate of a background speech value.

12. The method of claim 11 wherein determining an estimate of a background speech value comprises using the estimate of the clean speech value to estimate the background speech value.

13. The method of claim 9 further comprising using a prior model of the channel response to estimate the clean speech value.