Adaptive directional audio enhancement and selection

• US 9,813,808 B1
• Filed: 03/14/2013
• Issued: 11/07/2017
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Yang, Jun
• Primary Examiner(s): Chin, Vivian
• Assistant Examiner(s): Kurr, Jason R
• Application Number:
  US13/826,740
• Time in Patent Office: 1,699 Days
• Field of Search: 381 66, 381 711, 381 7111, 381 941, 381 95, 381 96, 381 92, 381122, 37940601, 37940602, 37940603, 37940608, 37940609, 37942001, 37942002, 37942003, 4555691, 455570
• CPC Class Codes: H04M 9/08 Two-way loud-speaking telephone systems with means for conditioning the signal, e.g. for suppressing echoes for one or both directions of traffic
• CPC Class Codes: H04R 2201/401 2D or 3D arrays of transducers
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04R 3/007 Protection circuits for transducers
• CPC Class Codes: H04R 3/04 for correcting frequency response
• CPC Class Codes: H04R 5/04 Circuit arrangements, e.g. for selective connection of amplifier inputs/outputs to loudspeakers, for loudspeaker detection, or for adaptation of settings to personal preferences or hearing impairments combinations of amplifiers H03F3/68; stereophonic systems H04S

Abstract

A hands-free audio device has a receive path and a transmit path, which may operate at different audio sampling rates. The transmit path has an interference suppressor that receives a reference signal from the receive path and that suppresses interference in microphone signals received from a microphone array. The interference suppressor is followed in the transmit path by a multi-channel adaptive beamformer that produce a plurality of directional audio signals. A beam selector is configured to select one of the directional audio signals based on voice activity, echo detection, and signal energy.

Claims

1. A hands-free audio system, comprising:

a speaker configured to generate audio based at least in part on a received output audio signal;

a microphone array comprising a plurality of spatially separated microphones, wherein an individual microphone of the plurality of spatially separated microphones produces a respective individual audio signal, wherein the respective individual audio signal potentially contains a near-end voice component and interference with respect to the near-end voice component, and wherein the interference comprises one or more of (a) an ambient noise component or (b) an acoustic echo component resulting from the audio;

one or more interference suppressors configured to produce processed audio signals by suppressing the interference in the individual audio signals based, at least in part on, a reference audio signal;

one or more audio beamformers, an individual beamformer of the one or more audio beamformers being configured to produce a directional audio signal based at least in part on the processed audio signals, wherein the directional audio signal emphasizes sound from a particular direction relative to the microphone array;

one or more voice activity detectors, wherein a voice activity detector of the one or more voice activity detectors is configured to detect at least one of a presence of the near-end voice component in the directional audio signal or a presence of a far-end voice component in the directional audio signal;

one or more signal energy measurers, wherein a signal energy measurer of the one or more signal energy measurers is configured to measure a signal energy of the directional audio signal; and

a signal selector configured to select the directional audio signal based at least in part on (a) the presence of the near-end voice component detected by the voice activity detector in the directional audio signal, (b) the presence of the far-end voice component detected by the voice activity detector in the directional audio signal, and (c) the signal energy measured by the signal energy measurer.

2. The system of claim 1, wherein:

the one or more voice activity detectors are configured to identify a subset of the directional audio signals that contain the near-end voice component; and

the signal selector is further configured to select the directional audio signal from the subset based at least in part on measured signal energies of directional audio signals of the subset.

3. The system of claim 1, wherein the signal selector is further configured to select the directional audio signal based at least in part on measured signal energies of the directional audio signals and on that the individual audio signals contain near-end audio or far-end audio.

4. The system of claim 1, wherein:

the one or more voice activity detectors are configured to identify a subset of directional audio signals that contain the near-end voice component; and

to select the directional audio signal, the signal selector is configured to (a) select a low-energy directional audio signal of the subset in response to a presence of far-end audio and absence of near-end audio in the directional audio signals of the subset, wherein the low-energy directional audio signal has a lowest signal energy among the directional audio signals of the subset; and

(b) select a high-energy directional audio signal of the subset in response to the presence of near-end audio in the directional audio signals of the subset, wherein the high-energy directional audio signal has a highest signal energy among the directional audio signals of the subset.

5. The system of claim 1, wherein a first one of the audio beamformers is responsive to a first processed audio signal of the processed audio signals and a second processed audio signal of the processed audio signals, the first one of the audio beamformers comprising:

a first adaptive filter stage configured to estimate a voice signal based at least in part on the first processed audio signal and to subtract the voice signal from the second processed audio signal to produce an intermediate audio signal; and

a second adaptive filter stage configured to estimate an interference signal based at least in part on the intermediate audio signal and to subtract the interference signal from the first processed audio signal.

6. The system of claim 1, further comprising one or more processing components configured to produce, based at least in part on the received output audio signal, the reference audio signal.

7. The hands-free audio system of claim 1, further comprising a speech recognition component configured to recognize speech represented by the directional audio signal selected by the signal selector.

8. The hands-free audio system of claim 1, wherein to select the directional audio signal, the signal selector is configured to select the directional audio signal that includes a lowest signal energy based, at least in part, on an absence of near-end audio in the directional audio signal.

9. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform acts comprising:

suppressing, based at least in part on a reference audio signal, interference in a first audio signal received from a microphone array to produce a first processed audio signal, wherein the interference includes one or more of an echo component and an ambient noise component;

suppressing, based at least in part on the reference audio signal, the interference in a second audio signal received from the microphone array to produce a second processed audio signal;

producing directional audio signals based at least in part on the first processed audio signal and the second processed audio signal, wherein at least a first directional audio signal of the directional audio signals emphasizes sound from a particular direction relative to the microphone array;

detecting a presence of far-end audio in the directional audio signals;

determining a first signal energy of the first directional audio signal;

determining a second signal energy of a second directional audio signal of the directional audio signals, wherein the second signal energy is greater than the first signal energy; and

selecting the first directional audio signal based, at least in part, on detecting the presence of the far-end audio and the second signal energy being greater than the first signal energy.

10. The one or more non-transitory computer-readable media of claim 9, the acts further comprising:

receiving the first audio signal at a first sample rate;

receiving an output audio signal at a second sample rate;

processing the output audio signal to produce a processed output audio signal, wherein the processed output audio signal is for output to a speaker; and

converting the processed output audio signal from the second sample rate to the first sample rate to generate the reference audio signal.

11. The one or more non-transitory computer-readable media of claim 10, wherein suppressing the interference in the first audio signal comprises adaptively filtering the first audio signal using the reference audio signal to estimate the echo component.

12. The one or more non-transitory computer-readable media of claim 9, the acts further comprising measuring relative energies of the directional audio signals, wherein selecting the directional audio signal is further based at least in part on measuring the relative energies of the directional audio signals.

13. The one or more non-transitory computer-readable media of claim 9, wherein selecting the first directional audio signal comprises selecting a low-energy directional audio signal based at least in part on detecting the far-end audio in the directional audio signals.

14. The one or more non-transitory computer-readable media of claim 9, wherein producing the first directional audio signal of the one or more directional audio signals comprises adaptively filtering the first processed audio signal and the second processed audio signal.

15. The one or more non-transitory computer-readable media of claim 9, wherein producing the first directional audio signal of the one or more directional audio signals comprises:

adaptively estimating a voice signal based at least in part on the first processed audio signal;

subtracting the estimated voice signal from the second processed audio signal to produce an intermediate audio signal;

adaptively estimating an interference signal based at least in part on the intermediate audio signal; and

subtracting the estimated interference signal from the first processed audio signal.

16. The one or more non-transitory computer-readable media of claim 9, the acts further comprising:

detecting an absence of near-end audio in the directional audio signals, and

wherein selecting the first directional audio signal is further based on detecting the absence of the near-end audio.

17. A method comprising:

suppressing, based at least in part on a reference audio signal, interference in a first audio signal received from a microphone array to produce a first processed audio signal, wherein the interference includes one or more of an echo component and a noise component;

suppressing, based at least in part on the reference audio signal, the interference in a second audio signal received from the microphone array to produce a second processed audio signal;

producing directional audio signals based at least in part on the first processed audio signal and the second processed audio signal, wherein at least a first directional audio signal of the directional audio signals emphasizes sound from a particular direction relative to the microphone array;

detecting a presence of far-end audio in the directional audio signals;

determining a first signal energy of the first directional audio signal;

determining a second signal energy of a second directional audio signal of the directional audio signals, wherein the second signal energy is greater than the first signal energy; and

selecting the first directional audio signal from the directional audio signals based, at least in part, on detecting the presence of the far-end audio and the second signal energy being greater than the first signal energy.

18. The method of claim 17, further comprising:

receiving the first audio signal at a first sample rate;

receiving an output audio signal at a second sample rate;

processing the output audio signal in order to produce a processed output audio signal, wherein the processed output audio signal is for output to a speaker; and

converting the processed output audio signal from the second sample rate to the first sample rate to generate the reference audio signal.

19. The method of claim 17, wherein selecting the first directional audio signal comprises selecting a low-energy directional audio signal of the directional audio signals based at least in part on detecting the far-end audio in the directional audio signals.

20. The method of claim 17, further comprising:

detecting an absence of near-end audio in the directional audio signals,

wherein selecting the first directional audio signal is further based on detecting the absence of the near-end audio in the directional audio signals.