Hybrid reference signal for acoustic echo cancellation

• US 9,653,060 B1
• Filed: 02/09/2016
• Issued: 05/16/2017
• Est. Priority Date: 02/09/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Hilmes, Philip Ryan, Ayrapetian, Robert
• Primary Examiner(s): Bernardi, Brenda
• Application Number:
  US15/019,108
• Time in Patent Office: 462 Days
• CPC Class Codes: G10K 2210/3012 Algorithms
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2021/02082 the noise being echo, reverberation of the speech
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming
• CPC Class Codes: H04M 9/082 using echo cancellers echo cancellers per se H04B3/23

Abstract

An echo cancellation system that uses a combined reference signal using a playback reference signal and an adaptive reference signal. The playback reference signal is generated from a playback signal sent to a loudspeaker and the adaptive reference signal is generated using beamforming on microphone inputs corresponding to audio received from the loudspeaker. The system applies a low pass filter to the playback reference signal and applies a high pass filter to the adaptive reference signal to generate the combined reference signal. The system may remove the combined reference signal from target signals associated with the microphone inputs to isolate speech included in the target signals.

Claims

1. A computer-implemented method for cancelling an echo from an audio signal to isolate received speech, the method comprising:

sending first playback audio data to a first wireless speaker;

receiving first input audio data from a first microphone of a microphone array, the first input audio data including a first representation of audible sound output by the first wireless speaker and speech input;

receiving second input audio data from a second microphone of the microphone array, the second input audio data including a second representation of the audible sound output by the first wireless speaker and the speech input;

determining a first portion of combined input audio data, the combined input audio data comprising at least the first input audio data and the second input audio data, the first portion of the combined input audio data comprising a first portion of the first input audio data corresponding to a first direction and a first portion of the second input audio data corresponding to the first direction;

determining a second portion of the combined input audio data, the second portion of the combined input audio data comprising a second portion of the first input audio data corresponding to a second direction and a second portion of the second input audio data corresponding to the second direction;

selecting at least the first portion of the combined input audio data as a target signal on which to perform echo cancellation;

blocking high frequencies of the first playback audio data using a low pass filter to generate a first reference signal, the first reference signal including the first playback audio data below a first cutoff frequency;

blocking low frequencies of the second portion of the combined input audio data using a high pass filter to generate a second reference signal, the second reference signal including the second portion of the combined input audio data above the first cutoff frequency;

generating an input reference signal by combining the first reference signal and the second reference signal;

removing the input reference signal from the target signal to generate a first output audio signal that includes the speech input;

performing speech recognition processing on the first output audio signal to determine a command; and

executing the command.

2. The computer-implemented method of claim 1, further comprising:

determining a propagation delay of the first input audio data relative to the first playback audio data;

generating second playback audio data by delaying the first playback audio data by the propagation delay;

determining a frequency offset between the second playback audio data and the first input audio data;

generating third playback audio data using the second playback audio data by one of;

removing at least one sample of the second playback audio data per cycle to compensate for the frequency offset, or

adding a duplicate copy of at least one sample of the second playback audio data to the second playback audio data to compensate for the frequency offset;

blocking high frequencies of the third playback audio data using the low pass filter to generate the first reference signal, the first reference signal including the third playback audio data below the first cutoff frequency.

3. The computer-implemented method of claim 1, further comprising:

determining that the second portion of the combined input audio data corresponds to a highest amplitude representation of the audible sound output of a plurality of portions of the combined input audio data;

determining that an amplitude of the second portion of the combined input audio data is above a threshold;

associating the second portion of the combined input audio data with the first wireless speaker by selecting the second portion as the second reference signal; and

selecting remaining portions of the plurality of portions as the target signal.

4. The computer-implemented method of claim 1, further comprising:

determining, using a fixed beamforming technique, the first and the second portions of the combined input audio data;

determining that a first amplitude associated with the first portion of the combined input audio data is below a threshold;

determining that a second amplitude associated with the second portion of the combined input audio data is above the threshold;

determining, using an adaptive beamforming technique, a third portion of the combined input audio data, the third portion of the combined input audio data comprising a third portion of the first input audio data corresponding to the second direction and a third portion of the second input audio data corresponding to the second direction; and

blocking low frequencies of the third portion of the combined input audio data using the high pass filter to generate the second reference signal, the second reference signal including the third portion of the combined input audio data above the first cutoff frequency.

5. A computer-implemented method, comprising:

sending first playback audio data to a first wireless speaker;

receiving first input audio data from a first microphone of a microphone array, the first input audio data including a first representation of sound output by the first wireless speaker and speech input;

receiving second input audio data from a second microphone of the microphone array, the second input audio data including a second representation of the audible sound output by the first wireless speaker and the speech input;

determining a first portion of combined input audio data, the combined input audio data comprising at least the first input audio data and the second input audio data, the first portion of the combined input audio data comprising a first portion of the first input audio data corresponding to a first direction and a first portion of the second input audio data corresponding to the first direction;

determining a second portion of the combined input audio data, the second portion of the combined input audio data comprising a second portion of the first input audio data corresponding to a second direction and a second portion of the second input audio data corresponding to the second direction;

generating first reference data based on the first playback audio data, the first reference data having frequencies below a first cutoff frequency;

generating second reference data based on the second portion of the combined input audio data, the second reference data having frequencies above the first cutoff frequency; and

generating a reference signal by combining the first reference data and the second reference data.

6. The computer-implemented method of claim 5, further comprising:

selecting at least the first portion of the combined input audio data as a first target signal;

removing the reference signal from the first target signal to generate first output audio data that includes the speech input;

selecting at least a third portion of the combined input audio data as a second target signal;

removing the reference signal from the second target signal to generate second output audio data that includes the speech input;

determining a first signal to noise ratio associated with the first output audio data;

determining a second signal to noise ratio associated with the second output audio data;

determining that the first signal to noise ratio is larger than the second signal to noise ratio;

performing speech recognition processing on the first output audio data to determine a command; and

executing the command.

7. The computer-implemented method of claim 5, further comprising:

determining a propagation delay of the first input audio data relative to the first playback audio data;

generating second playback audio data by delaying the first playback audio data by the propagation delay;

determining a frequency offset between the second playback audio data and the first input audio data;

generating third playback audio data by one of;

removing at least one sample of the second playback audio data per cycle based on the frequency offset, and

adding a duplicate copy of at least one sample of the second playback audio data to the second playback audio data based on the frequency offset;

blocking high frequencies of the third playback audio data using a low pass filter to generate the first reference data, the first reference data including the third playback audio data below the first cutoff frequency.

8. The computer-implemented method of claim 5, further comprising:

determining that the second portion of the combined input audio data corresponds to a highest amplitude of a plurality of portions of the combined input audio data;

determining that an amplitude of the second portion of the combined input audio data is above a threshold; and

associating the second portion of the combined input audio data with the first wireless speaker.

9. The computer-implemented method of claim 5, further comprising:

determining that an amplitude associated with the second portion of the combined input audio data is above a threshold;

determining that a highest amplitude associated with remaining portions of a plurality of portions of the combined input audio data is below the threshold;

generating the second reference data based on at least the second portion of the combined input audio data;

selecting at least the first portion of the combined input audio data as a target signal; and

removing the reference signal from the target signal to generate first output audio data that includes the speech input.

10. The computer-implemented method of claim 5, further comprising:

determining that the speech input is associated with the first direction;

selecting at least the first portion of the combined input audio data as a target signal;

determining that the second direction is opposite the first direction;

generating the second reference data based on at least the second portion of the combined input audio data; and

removing the reference signal from the target signal to generate first output audio data that includes the speech input.

11. The computer-implemented method of claim 5, further comprising:

determining that the second portion of the combined input audio data corresponds to a highest amplitude of a plurality of portions;

determining that an amplitude of the second portion of the combined input audio data is below a threshold;

selecting the first portion of the combined input audio data as a first target signal;

determining that the second direction is opposite the first direction;

generating the second reference data based on the second portion of the combined input audio data;

removing the reference signal from the first target signal to generate first output audio data that includes the speech input;

selecting the second portion of the combined input audio data as a second target signal;

generating third reference data based on the first portion of the combined input audio data, the third reference data having frequencies above the first cutoff frequency;

generating a second reference signal by combining the first reference data and the third reference data; and

removing the second reference signal from the second target signal to generate second output audio data that includes the speech input.

12. The computer-implemented method of claim 5, further comprising:

determining, using a fixed beamforming technique, the first and the second portions of the combined input audio data;

determining that a first amplitude associated with the first portion of the combined input audio data is below a threshold;

determining that a second amplitude associated with the second portion of the combined input audio data is above the threshold;

determining, using an adaptive beamforming technique, a third portion of the combined input audio data, the third portion of the combined input audio data comprising a third portion of the first input audio data corresponding to the second direction and a third portion of the second input audio data corresponding to the second direction;

blocking low frequencies of the third portion of the combined input audio data using a high pass filter to generate the second reference data, the second reference data including the third portion of the combined input audio data above the first cutoff frequency.

13. A device, comprising:

at least one processor;

a memory device including instructions operable to be executed by the at least one processor to configure the device to;

send first playback audio data to a first wireless speaker;

receive first input audio data from a first microphone of a microphone array, the first input audio data including a first representation of sound output by the first wireless speaker and speech input;

receive second input audio data from a second microphone of the microphone array, the second input audio data including a second representation of the audible sound output by the first wireless speaker and the speech input;

determine a first portion of combined input audio data, the combined input audio data comprising at least the first input audio data and the second input audio data, the first portion of the combined input audio data comprising a first portion of the first input audio data corresponding to a first direction and a first portion of the second input audio data corresponding to the first direction;

determine a second portion of the combined input audio data, the second portion of the combined input audio data comprising a second portion of the first input audio data corresponding to a second direction and a second portion of the second input audio data corresponding to the second direction;

generate first reference data based on the first playback audio data, the first reference data having frequencies below a first cutoff frequency;

generate second reference data based on the second portion of the combined input audio data, the second reference data having frequencies above the first cutoff frequency; and

generate a reference signal by combining the first reference data and the second reference data.

14. The device of claim 13, wherein the instructions further configure the device to:

select at least the first portion of the combined input audio data as a first target signal;

remove the reference signal from the first target signal to generate first output audio data that includes the speech input;

select at least a third portion of the combined input audio data as a second target signal;

remove the reference signal from the second target signal to generate second output audio data that includes the speech input;

determine a first signal to noise ratio associated with the first output audio data;

determine a second signal to noise ratio associated with the second output audio data;

determine that the first signal to noise ratio is larger than the second signal to noise ratio;

perform speech recognition processing on the first output audio data to determine a command; and

execute the command.

15. The device of claim 13, wherein the instructions further configure the device to:

determine a propagation delay of the first input audio data relative to the first playback audio data;

generate second playback audio data by delaying the first playback audio data by the propagation delay;

determine a frequency offset between the second playback audio data and the first input audio data;

generate third playback audio data by one of;

removing at least one sample of the second playback audio data per cycle based on the frequency offset, and

adding a duplicate copy of at least one sample of the second playback audio data to the second playback audio data based on the frequency offset;

blocking high frequencies of the third playback audio data using a low pass filter to generate the first reference data, the first reference data including the third playback audio data below the first cutoff frequency.

16. The device of claim 13, wherein the instructions further configure the device to:

determine that the second portion of the combined input audio data corresponds to a highest amplitude of a plurality of portions of the combined input audio data;

determine that an amplitude of the second portion of the combined input audio data is above a threshold; and

associate the second portion of the combined input audio data with the first wireless speaker.

17. The device of claim 13, wherein the instructions further configure the device to:

determine that an amplitude associated with the second portion of the combined input audio data is above a threshold;

determine that a highest amplitude associated with remaining portions of a plurality of portions of the combined input audio data is below the threshold;

generate the second reference data based on at least the second portion of the combined input audio data;

select at least the first portion of the combined input audio data as a target signal; and

remove the reference signal from the target signal to generate first output audio data that includes the speech input.

18. The device of claim 13, wherein the instructions further configure the device to:

determine that the speech input is associated with the first direction;

select at least the first portion of the combined input audio data as a target signal;

determine that the second direction is opposite the first direction;

generate the second reference data based on at least the second portion of the combined input audio data; and

remove the reference signal from the target signal to generate first output audio data that includes the speech input.

19. The device of claim 13, wherein the instructions further configure the device to:

determine that the second portion of the combined input audio data corresponds to a highest amplitude of a plurality of portions;

determine that an amplitude of the second portion of the combined input audio data is below a threshold;

select the first portion of the combined input audio data as a first target signal;

determine that the second direction is opposite the first direction;

generate the second reference data based on the second portion of the combined input audio data;

remove the reference signal from the first target signal to generate first output audio data that includes the speech input;

select the second portion of the combined input audio data as a second target signal;

generate third reference data based on the first portion of the combined input audio data, the third reference data having frequencies above the first cutoff frequency;

generate a second reference signal by combining the first reference data and the third reference data; and

remove the second reference signal from the second target signal to generate second output audio data that includes the speech input.

20. The device of claim 13, wherein the instructions further configure the device to:

determine, using a fixed beamforming technique, the first and the second portions of the combined input audio data;

determine that a first amplitude associated with the first portion of the combined input audio data is below a threshold;

determine that a second amplitude associated with the second portion of the combined input audio data is above the threshold;

determine, using an adaptive beamforming technique, a third portion of the combined input audio data, the third portion of the combined input audio data comprising a third portion of the first input audio data corresponding to the second direction and a third portion of the second input audio data corresponding to the second direction;

blocking low frequencies of the third portion of the combined input audio data using a high pass filter to generate the second reference data, the second reference data including the third portion of the combined input audio data above the first cutoff frequency.