System and method for performing speech enhancement using a deep neural network-based signal

• US 10,074,380 B2
• Filed: 08/03/2016
• Issued: 09/11/2018
• Est. Priority Date: 08/03/2016
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Wung, Jason, Pishehvar, Ramin, Giacobello, Daniele, Atkins, Joshua D.
• Primary Examiner(s): Niu, Feng
• Assistant Examiner(s): Brinich, Stephen
• Application Number:
  US15/227,885
• Publication Number:
  US 20180040333A1
• Time in Patent Office: 769 Days
• Field of Search: 704 1- 2, 704 10- 11, 704 18- 20
• CPC Class Codes: G10L 2021/02082 the noise being echo, reverberation of the speech
• CPC Class Codes: G10L 21/0232 Processing in the frequency domain
• CPC Class Codes: G10L 25/30 using neural networks
• CPC Class Codes: G10L 25/87 Detection of discrete points within a voice signal

Abstract

Method for performing speech enhancement using a Deep Neural Network (DNN)-based signal starts with training DNN offline by exciting a microphone using target training signal that includes signal approximation of clean speech. Loudspeaker is driven with a reference signal and outputs loudspeaker signal. Microphone then generates microphone signal based on at least one of: near-end speaker signal, ambient noise signal, or loudspeaker signal. Acoustic-echo-canceller (AEC) generates AEC echo-cancelled signal based on reference signal and microphone signal. Loudspeaker signal estimator generates estimated loudspeaker signal based on microphone signal and AEC echo-cancelled signal. DNN receives microphone signal, reference signal, AEC echo-cancelled signal, and estimated loudspeaker signal and generates a speech reference signal that includes signal statistics for residual echo or for noise. Noise suppressor generates a clean speech signal by suppressing noise or residual echo in the microphone signal based on speech reference signal. Other embodiments are described.

Claims

1. A system for performing speech enhancement using a Deep Neural Network (DNN)-based signal comprising:

a loudspeaker to output a loudspeaker signal, wherein the loudspeaker is being driven by a reference signal;

at least one microphone to receive at least one of;

a near-end speaker signal, an ambient noise signal, or the loudspeaker signal and to generate a microphone signal;

an acoustic-echo-canceller (AEC) to receive the reference signal and the microphone signal, and to generate an AEC echo-cancelled signal;

a loudspeaker signal estimator to receive the microphone signal and the AEC echo-cancelled signal and to generate an estimated loudspeaker signal; and

a deep neural network (DNN) to receive the microphone signal, the reference signal, the AEC echo-cancelled signal, and the estimated loudspeaker signal, and to generate a clean speech signal,

wherein the DNN is trained offline by exciting the at least one microphone using a target training signal that includes a signal approximation of clean speech.

2. The system of claim 1, wherein the DNN generating the clean speech signal includes:

the DNN generating at least one of;

an estimate of non-linear echo in the microphone signal that is not cancelled by the AEC, an estimate of residual echo in the microphone signal, or an estimate of ambient noise power level in the microphone signal, and

the DNN generating the clean speech signal based on the estimate of non-linear echo in the microphone signal that is not cancelled by the AEC, the estimate of residual echo in the microphone signal, or the estimate of ambient noise power level.

3. The system of claim 1, wherein the DNN is one of a deep feed-forward neural network, a deep recursive neural network, or a deep convolutional neural network.

4. The system of claim 1, further comprising:

a time-frequency transformer to transform the microphone signal, the reference signal, the AEC echo-cancelled signal and the estimated loudspeaker signal from a time domain to a frequency domain, wherein the DNN receives and processes the microphone signal, the reference signal, the AEC echo-cancelled signal and the estimated loudspeaker signal in the frequency domain, and the DNN to generate the clean speech signal in the frequency domain; and

a frequency-time transformer to transform the clean speech signal in the frequency domain to a clean speech signal in the time domain.

5. The system of claim 4, further comprising:

a plurality of feature processors, each feature processor to respectively extract and transmit features of the microphone signal, the reference signal, the AEC echo-cancelled signal and the estimated loudspeaker signal to the DNN.

6. The system of claim 5, wherein each of the feature processors include:

a smoothed power spectral density (PSD) unit to calculate a smoothed PSD, and

a feature extractor to extract one of the features of the microphone signal, the reference signal, the AEC echo-cancelled signal and the estimated loudspeaker signal,

a first normalization unit to normalize the smoothed PSD using a global mean and variance from training data, and

a second normalization unit to normalize the extracted one of the features using a global mean and variance from the training data, and

wherein the system further includes;

a plurality of feature buffers to receive the normalized smoothed PSD and the normalized extracted feature from each of the feature processors, respectively, and to respectively buffer the extracted features with a number of past or future frames.

7. The system of claim 5, wherein

the microphone signal, the reference signal, the AEC echo-cancelled signal and the estimated loudspeaker signal in the frequency domain are complex signals including a magnitude component and a phase component.

8. The system of claim 7, wherein each of the feature processors include:

a smoothed power spectral density (PSD) unit to calculate a smoothed PSD, and

a feature extractor to extract one of the features of the microphone signal, the reference signal, the AEC echo-cancelled signal and the estimated loudspeaker signal,

a first normalization unit to normalize the smoothed PSD using a global mean and variance from the training data, and

a second normalization unit to normalize the extracted one of the features using a global mean and variance from training data, and

wherein the system further includes;

a plurality of feature buffers to receive the normalized smoothed PSD and the normalized extracted feature from each of the feature processors, respectively, and to respectively buffer the extracted features with a number of past or future frames.

9. A system for performing speech enhancement using a Deep Neural Network (DNN)-based signal comprising:

a loudspeaker to output a loudspeaker signal, wherein the loudspeaker is being driven by a reference signal;

at least one microphone to receive at least one of;

a near-end speaker signal, an ambient noise signal, or the loudspeaker signal and to generate a microphone signal;

an acoustic-echo-canceller (AEC) to receive the reference signal and the microphone signal, and to generate an AEC echo-cancelled signal;

a loudspeaker signal estimator to receive the microphone signal and the AEC echo-cancelled signal and to generate an estimated loudspeaker signal; and

a deep neural network (DNN) to receive the microphone signal, the reference signal, the AEC echo-cancelled signal, and the estimated loudspeaker signal, and to generate a speech reference signal that includes signal statistics for residual echo or signal statistics for noise,

wherein the DNN is trained offline by exciting the at least one microphone using a target training signal that includes a signal approximation of clean speech.

10. The system of claim 9, wherein the speech reference signal that includes signal statistics for residual echo or signal statistics for noise includes at least one of:

an estimate of non-linear echo in the microphone signal that is not cancelled by the AEC, an estimate of residual echo in the microphone signal, or an estimate of ambient noise power level in the microphone signal.

11. The system of claim 9, wherein the DNN is one of a deep feed-forward neural network, a deep recursive neural network, or a deep convolutional neural network.

12. The system of claim 9, further comprising:

a time-frequency transformer to transform the microphone signal, the reference signal, the AEC echo-cancelled signal and the estimated loudspeaker signal from a time domain to a frequency domain, wherein the DNN receives and processes the microphone signal, the reference signal, the AEC echo-cancelled signal and the estimated loudspeaker signal in the frequency domain, and the DNN to generate the speech reference in the frequency domain.

13. The system of claim 12, further comprising:

a noise suppressor to receive the AEC echo-cancelled signal and the speech reference in the frequency domain, to suppress noise or residual echo in the microphone signal based on the speech reference and to output a clean speech signal in the frequency domain; and

a frequency-time transformer to transform the clean speech signal in the frequency domain to a clean speech signal in the time domain.

14. The system of claim 13, further comprising

a plurality of feature processors, each feature processor to respectively extract and transmit features of the microphone signal, the reference signal, the AEC echo-cancelled signal and the estimated loudspeaker signal to the DNN.

15. The system of claim 14, wherein each of the feature processors include:

a smoothed power spectral density (PSD) unit to calculate a smoothed PSD, and

a feature extractor to extract one of the features of the microphone signal, the reference signal, the AEC echo-cancelled signal and the estimated loudspeaker signal,

a first normalization unit to normalize the smoothed PSD using a global mean and variance from training data, and

a second normalization unit to normalize the extracted one of the features using a global mean and variance from the training data, and

wherein the system further includes;

a plurality of feature buffers to receive the normalized smoothed PSD and the normalized extracted feature from each of the feature processors, respectively, and to respectively buffer the extracted features with a number of past or future frames.

16. A method for performing speech enhancement using a Deep Neural Network (DNN)-based signal comprising:

training a deep neural network (DNN) offline by exciting at least one microphone using a target training signal that includes a signal approximation of clean speech;

driving a loudspeaker with a reference signal, wherein the loudspeaker outputs a loudspeaker signal;

generating by the at least one microphone a microphone signal based on at least one of;

a near-end speaker signal, an ambient noise signal, or the loudspeaker signal;

generating by an acoustic-echo-canceller (AEC) an AEC echo-cancelled signal based on the reference signal and the microphone signal;

generating by a loudspeaker signal estimator an estimated loudspeaker signal based on the microphone signal and the AEC echo-cancelled signal;

receiving by the DNN the microphone signal, the reference signal, the AEC echo-cancelled signal, and the estimated loudspeaker signal; and

generating by the DNN a speech reference signal that includes signal statistics for residual echo or signal statistics for noise based on the microphone signal, the reference signal, the AEC echo-cancelled signal, and the estimated loudspeaker signal.

17. The method of claim 16, wherein the speech reference signal that includes signal statistics for residual echo includes at least one of:

an estimate of non-linear echo in the microphone signal that is not cancelled by the AEC, an estimate of residual echo in the microphone signal, or an estimate of ambient noise power level in the microphone signal.

18. The method of claim 17, further comprising:

generating by a noise suppressor a clean speech signal by suppressing noise or residual echo in the microphone signal based on speech reference signal.