Method and apparatus for multi-sensory speech enhancement

• US 20050114124A1
• Filed: 11/26/2003
• Published: 05/26/2005
• Est. Priority Date: 11/26/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Sinclair, Michael J., Deng, Li, Zhang, Zhengyou, Huang, Xuedong D., Droppo, James G., Acero, Alejandro, Liu, Zicheng, Zheng, Yanli
• Granted Patent: US 7,447,630 B2
• Time in Patent Office: Days
• US Class Current: 704/228
• CPC Class Codes: G10L 2021/02165 Two microphones, one receiving mainly the noise signal and the other one mainly the speech signal
• CPC Class Codes: G10L 21/0208 Noise filtering

Abstract

A method and system use an alternative sensor signal received from a sensor other than an air conduction microphone to estimate a clean speech value. The estimation uses either the alternative sensor signal alone, or in conjunction with the air conduction microphone signal. The clean speech value is estimated without using a model trained from noisy training data collected from an air conduction microphone. Under one embodiment, correction vectors are added to a vector formed from the alternative sensor signal in order to form a filter, which is applied to the air conductive microphone signal to produce the clean speech estimate. In other embodiments, the pitch of a speech signal is determined from the alternative sensor signal and is used to decompose an air conduction microphone signal. The decomposed signal is then used to determine a clean signal estimate.

Claims

1. A method of determining an estimate for a noise-reduced value representing a portion of a noise-reduced speech signal, the method comprising:

generating an alternative sensor signal using an alternative sensor other than an air conduction microphone;

converting the alternative sensor signal into at least one alternative sensor vector; and

adding a correction vector to the alternative sensor vector to form the estimate for the noise-reduced value.

2. The method of claim 1 wherein generating an alternative sensor signal comprises using a bone conduction microphone to generate the alternative sensor signal.

3. The method of claim 1 wherein adding a correction vector comprises adding a weighted sum of a plurality of correction vectors.

4. The method of claim 3 wherein each correction vector corresponds to a mixture component and each weight applied to a correction vector is based on the probability of the correction vector'"'"'s mixture component given the alternative sensor vector.

5. The method of claim 1 further comprising training a correction vector through steps comprising:

generating an alternative sensor training signal;

converting the alternative sensor training signal into an alternative sensor training vector;

generating a clean air conduction microphone training signal;

converting the clean air conduction microphone training signal into an air conduction training vector; and

using the difference between the alternative sensor training vector and the air conduction training vector to form the correction vector.

6. The method of claim 5 wherein training a correction vector further comprises training a separate correction vector for each of a plurality of mixture components.

7. The method of claim 1 further comprising generating a refined estimate of a noise-reduced value through steps comprising:

generating an air conduction microphone signal;

converting the air conduction microphone signal into an air conduction vector;

estimating a noise value;

subtracting the noise value from the air conduction vector to form an air conduction estimate;

combining the air conduction estimate and the estimate for the noise-reduced value to form the refined estimate for the noise-reduced value.

8. The method of claim 7 wherein combining the air conduction estimate and the estimate for the noise-reduced value comprises combining the air conduction estimate and the estimate for the noise-reduced value in the power spectrum domain.

9. The method of claim 8 further comprising using the refined estimate for the noise-reduced value to form a filter.

10. The method of claim 1 wherein forming the estimate for the noise-reduced value comprises forming the estimate without estimating noise.

11. The method of claim 1 further comprising:

generating a second alternative sensor signal using a second alternative sensor other than an air conduction microphone;

converting the second alternative sensor signal into at least one second alternative sensor vector;

adding a correction vector to the second alternative sensor vector to form a second estimate for the noise-reduced value; and

combining the estimate for the noise-reduced value with the second estimate for the noise-reduced value to form a refined estimate for the noise-reduced value.

12. A method of determining an estimate of a clean speech value, the method comprising:

receiving an alternative sensor signal from a sensor other than an air conduction microphone;

receiving an air conduction microphone signal from an air conduction microphone;

identifying a pitch for a speech signal based on the alternative sensor signal;

using the pitch to decompose the air conduction microphone signal into a harmonic component and a residual component; and

using the harmonic component and the residual component to estimate the clean speech value.

13. The method of claim 12 wherein receiving an alternative sensor signal comprises receiving an alternative sensor signal from a bone conduction microphone.

14. A computer-readable medium having computer-executable instructions for performing steps comprising:

receiving an alternative sensor signal from an alternative sensor that is not an air conduction microphone; and

using the alternative sensor signal to estimate a clean speech value without using a model trained from noisy training data collected from an air conduction microphone.

15. The computer-readable medium of claim 14 wherein receiving an alternative sensor signal comprises receiving a sensor signal from a bone conduction microphone.

16. The computer-readable medium of claim 14 wherein using the alternative sensor signal to estimate a clean speech value comprises:

converting the alternative sensor signal into at least one alternative sensor vector; and

adding a correction vector to an alternative sensor vector.

17. The computer-readable medium of claim 16 wherein adding a correction vector comprises adding a weighted sum of a plurality of correction vectors, each correction vector being associated with a separate mixture component.

18. The computer-readable medium of claim 17 wherein adding a weighted sum of a plurality of correction vectors comprises using a weight that is based on the probability of a mixture component given the alternative sensor vector.

19. The computer-readable medium of claim 14 further comprising receiving a noisy test signal from an air conductive microphone and using the noisy test signal with the alternative sensor signal to estimate the clean speech value.

20. The computer-readable medium of claim 19 wherein using the noisy test signal comprises generating a noise model from the noisy test signal.

21. The computer-readable medium of claim 20 wherein using the noisy test signal further comprises:

converting the noisy test signal into at least one noisy test vector;

subtracting a mean of the noise model from the noisy test vector to form a difference; and

using the difference to estimate the clean speech value.

22. The computer-readable medium of claim 21 further comprising:

forming an alternative sensor vector from the alternative sensor signal;

adding a correction vector to the alternative sensor vector to form an alternative sensor estimate of the clean speech value; and

determining a weighted sum of the difference and the alternative sensor estimate to form the estimate of the clean speech value.

23. The computer-readable medium of claim 22 wherein the estimate of the clean speech value is in the power spectrum domain.

24. The computer-readable medium of claim 23 further comprising using the estimate of the clean speech value to form a filter.

25. The computer-readable medium of claim 14 wherein using the alternative sensor signal to estimate a clean speech value further comprises:

determining a pitch for a speech signal based on the alternative sensor signal; and

using the pitch to estimate the clean speech value.

26. The computer-readable medium of claim 25 wherein using the pitch to estimate the clean speech value comprises:

receiving a noisy test signal from an air conduction microphone; and

decomposing the noisy test signal into a harmonic component and a residual component based on the pitch.

27. The computer-readable medium of claim 26 further comprising using the harmonic component and the residual component to estimate the clean speech value.

28. The computer-readable medium of claim 14 wherein estimating a clean speech value further comprises not estimating noise.

29. The computer-readable medium of claim 14 further comprising:

receiving a second alternative sensor signal from a second alternative sensor that is not an air conduction microphone; and

using the second alternative sensor signal with the alternative sensor signal to estimate the clean speech value.