Adaptive directional audio enhancement and selection

• US 10,250,975 B1
• Filed: 10/06/2017
• Issued: 04/02/2019
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Yang, Jun
• Primary Examiner(s): Kurr, Jason R
• Application Number:
  US15/726,831
• Time in Patent Office: 543 Days
• Field of Search: 381 66, 381 711, 381 7111, 381 941, 381 95, 381 96, 381 92, 381122, 37940601, 37940602, 37940603, 37940608, 37940609, 37942001, 37942002, 37942003, 4555691, 455570
• CPC Class Codes: H04M 9/08 Two-way loud-speaking telephone systems with means for conditioning the signal, e.g. for suppressing echoes for one or both directions of traffic
• CPC Class Codes: H04R 2201/401 2D or 3D arrays of transducers
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04R 3/007 Protection circuits for transducers
• CPC Class Codes: H04R 3/04 for correcting frequency response
• CPC Class Codes: H04R 5/04 Circuit arrangements, e.g. for selective connection of amplifier inputs/outputs to loudspeakers, for loudspeaker detection, or for adaptation of settings to personal preferences or hearing impairments combinations of amplifiers H03F3/68; stereophonic systems H04S

Abstract

A hands-free audio device has a receive path and a transmit path, which may operate at different audio sampling rates. The transmit path has an interference suppressor that receives a reference signal from the receive path and that suppresses interference in microphone signals received from a microphone array. The interference suppressor is followed in the transmit path by a multi-channel adaptive beamformer that produce a plurality of directional audio signals. A beam selector is configured to select one of the directional audio signals based on voice activity, echo detection, and signal energy.

Claims

1. An electronic device comprising:

a first microphone;

a second microphone;

one or more processors; and

one or more computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processor to perform operations comprising;

generating a first audio signal using the first microphone;

generating a second audio signal using the second microphone;

producing a first directional audio signal based at least in part on the first audio signal and the second audio signal;

producing a second directional audio signal based at least in part on the first audio signal and the second audio signal;

detecting presence of near-end audio in the first directional audio signal;

detecting presence of far-end audio in the second directional audio signal;

determining a first signal energy associated with the first directional audio signal;

determining a second signal energy associated with the second directional audio signal;

determining that the first signal energy is greater than the second signal energy;

selecting the first directional audio signal based at least in part on the presence of the near-end audio in the first directional audio signal and based at least in part on the first signal energy being greater than the second signal energy; and

sending the first directional audio signal.

2. The electronic device as recited in claim 1, the operations further comprising:

suppressing first interference in the first audio signal to produce a first processed audio signal; and

suppressing second interference in the second audio signal to produce a second processed audio signal,

wherein;

the producing the first directional audio signal comprises producing the first directional audio signal based at least in part on the first processed audio signal and the second processed audio signal; and

the producing the second directional audio signal comprises producing the second directional audio signal based at least in part on the first processed audio signal and the second processed audio signal.

3. The electronic device as recited in claim 1, wherein producing the first directional audio signal comprises:

estimating a voice signal based at least in part on the first audio signal;

subtracting at least a portion of the voice signal from the second audio signal to produce an intermediate audio signal;

estimating an interference signal based at least in part on the intermediate audio signal; and

subtracting at least a portion of the interference signal from the first audio signal to produce the first directional audio signal.

4. The electronic device as recited in claim 1, wherein:

the detecting the presence of the near-end audio in the first directional audio signal is based at least in part on the first signal energy; and

the detecting the presence of the far-end audio in the second directional audio signal is based at least in part on the second signal energy.

5. The electronic device as recited in claim 1, the operations further comprising:

identifying the first directional audio signal as including a threshold presence of voice activity; and

identifying the second directional audio signal as including the threshold presence of voice activity.

6. The electronic device as recited in claim 1, wherein the first directional audio signal represents user speech, and wherein the operations further comprise:

analyzing the first directional audio signal;

identifying the user speech represented by the first directional audio signal; and

causing output of content based at least in part on identifying the user speech.

7. The electronic device as recited in claim 1, further comprising a speaker, and wherein the operations further comprise:

causing the speaker to output content represented by a third audio signal,

wherein at least a portion of the far-end audio corresponds to the content output by the speaker.

8. A method comprising:

receiving a first audio signal from a first microphone;

receiving a second audio signal from a second microphone;

producing directional audio signals based at least in part on the first audio signal and the second audio signal;

detecting presence of near-end audio in at least one of the directional audio signals;

detecting presence of far-end audio in at least one of the directional audio signals;

determining a signal energy associated with a first directional audio signal of the directional audio signals;

selecting the first directional audio signal based at least in part on the presence of the near-end audio and the signal energy; and

sending the first directional audio signal.

9. The method as recited in claim 8, wherein the signal energy comprises a first signal energy, and wherein the method further comprises:

determining a second signal energy associated with a second directional audio signal of the directional audio signals; and

determining that the first signal energy is greater than the second signal energy,

wherein selecting the first directional audio signal is further based at least in part on the determining that the first signal energy is greater than the second signal energy.

10. The method as recited in claim 8, further comprising:

determining that the signal energy corresponds to a highest signal energy among the directional audio signals,

wherein selecting the first directional audio signal is further based at least in part on the determining that the signal energy corresponds to the highest signal energy.

11. The method as recited in claim 8, further comprising:

suppressing first interference in the first audio signal to produce a first processed audio signal; and

suppressing second interference in the second audio signal to produce a second processed audio signal,

wherein the producing the directional audio signals comprises producing, based at least in part on the first processed audio signal and the second processed audio signal, the directional audio signals.

12. The method as recited in claim 8, wherein the producing the directional audio signals comprises at least:

estimating a voice signal based at least in part on the first audio signal;

subtracting at least a portion of the voice signal from the second audio signal to produce an intermediate audio signal;

estimating an interference signal based at least in part on the intermediate audio signal; and

subtracting at least a portion of the interference signal from the first audio signal to produce the first directional audio signal.

13. The method as recited in claim 8, wherein the signal energy comprises a first signal energy, and wherein the method further comprises:

identifying a subset of the directional audio signals that include a threshold presence of voice activity, the subset including at least the first directional audio signal and a second directional audio signal of the directional audio signals; and

determining a second signal energy associated with the second directional audio signal,

wherein the selecting the first directional audio signal is further based at least in part on the second signal energy.

14. The method as recited in claim 8, wherein the signal energy comprises a first signal energy, and wherein the method further comprises:

determining a second signal energy associated with a second directional audio signal of the directional audio signals,

wherein;

the detecting the presence of the near-end audio in the at least one of the directional audio signals comprises detecting the presence of the near-end audio in the first directional audio signal based at least in part on the first signal energy; and

the detecting the presence of the far-end audio in the at least one of the directional audio signals comprises detecting the presence of the far-end audio in the second directional audio signal based at least in part on the second signal energy.

15. The method as recited in claim 8, wherein the first directional audio signal represents user speech, and wherein the method further comprises:

analyzing the first directional audio signal;

identifying the user speech represented by the first directional audio signal; and

outputting content based at least in part on the user speech.

16. The method as recited in claim 8, further comprising:

causing a speaker to output content represented by a third audio signal,

wherein at least a portion of the far-end audio corresponds to the content output by the speaker.

17. A system comprising:

one or more processors; and

one or more computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

receiving a first audio signal;

receiving a second audio signal;

producing directional audio signals based at least in part on the first audio signal and the second audio signal;

detecting presence of far-end audio in at least one of the directional audio signals;

determining a signal energy associated with a first directional audio signal of the directional audio signals;

determining a second signal energy associated with a second directional audio signal of the directional audio signals;

determining that the second signal energy is greater than the first signal energy;

selecting the first directional audio signal based at least in part on the presence of the far-end audio and the second signal energy being greater than the first signal energy; and

sending the first directional audio signal.

18. The system as recited in claim 17, wherein

the detecting the presence of the far-end audio in the at least one of the directional audio signals comprises detecting the presence of the far-end audio in the second directional audio signal based at least in part on the second signal energy.

19. The system as recited in claim 17, the operations further comprising:

suppressing first interference in the first audio signal to produce a first processed audio signal; and

suppressing second interference in the second audio signal to produce a second processed audio signal,

wherein the producing the directional audio signals comprises producing, based at least in part on the first processed audio signal and the second processed audio signal, the directional audio signals.

20. The system as recited in claim 17, wherein the producing the directional audio signals comprises at least:

estimating a voice signal based at least in part on the first audio signal;

subtracting at least a portion of the voice signal from the second audio signal to produce an intermediate audio signal;

estimating an interference signal based at least in part on the intermediate audio signal; and

subtracting at least a portion of the interference signal from the first audio signal to produce the first directional audio signal.