Frequency based beamforming

• US 9,521,486 B1
• Filed: 02/04/2013
• Issued: 12/13/2016
• Est. Priority Date: 02/04/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Barton, William Folwell
• Primary Examiner(s): Gay, Sonia
• Assistant Examiner(s): Le, Phan
• Application Number:
  US13/758,868
• Time in Patent Office: 1,408 Days
• Field of Search: 381/94.7, 381/92, 381/94.1, 381/94.2
• US Class Current: 1/1
• CPC Class Codes: H04R 2420/07 Applications of wireless loudspeakers or wireless microphones
• CPC Class Codes: H04R 2430/03 Synergistic effects of band splitting and sub-band processing
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407

Abstract

An acoustic device captures sound using two or more microphones and filters the sound into a plurality of sub-bands. For each of the plurality of sub-bands, the device identifies sound captured by the microphones from at least two directions and attenuates the sound captured by the microphones, within each of the sub-bands, in substantially one of the directions.

Claims

1. A method comprising:

capturing sound, at two or more microphones of a device, from an environment;

generating output data based on the sound;

filtering the output data into a plurality of portions, each portion corresponding to one of a plurality of frequency sub-bands; and

processing a first portion of the plurality of portions, the first portion corresponding to a first frequency sub-band, the processing comprising;

identifying first audio data in the first portion associated with a user;

determining a first direction related to the first audio data by triangulating a first position of the user relative to the device based on an a first analysis of the output data, the first analysis including determining a first order in which the two or more microphones captured sound associated with the user;

identifying that second audio data is present within the first portion, the second audio data corresponding to a source different from the user;

classifying the second audio data as background noise;

determining that the second audio data is causing a reduction to a speech-to-noise ratio;

determining, using the first portion and a first beamformer, the first beamformer configured to operate on data corresponding to a first frequency sub-band of the plurality of frequency sub-bands, a second direction related to the source by triangulating a second position of the source relative to the device based on a second analysis of the output data, the second analysis including determining a second order in which the two or more microphones captured sound associated with the second audio data, the second direction different than the first direction; and

adding a weighted signal to further output of the two or more microphones to determine;

attenuated third audio data corresponding to the first frequency sub-band and the second direction, and

amplified fourth audio data corresponding to the first frequency sub-band.

2. The method of claim 1, wherein the weighted signal is based on sound corresponding to the second direction captured by the two or more microphones.

3. The method of claim 1, wherein the first direction is a direction that has previously been associated with the user.

4. The method of claim 1, wherein the first audio data is a highest amplitude audio data within the output data.

5. The method of claim 1, wherein the weighted signal is added to the further output of the two or more microphones based at least in part on a delay.

6. The method of claim 1, further comprising adding the weighted signal to outputs of select microphones of the two or more microphones.

7. The method of claim 1, wherein:

the two or more microphones include at least a first microphone and a second microphone; and

the first direction and the second direction are determined based on a delay of sound captured by the first microphone relative to sound captured by the second microphone.

8. A system comprising:

two or more microphones to capture sound and output audio data;

a filter configured to filter the audio data into a first portion corresponding to a first frequency range and a second portion corresponding to a second frequency range different from the first frequency range;

a first beamformer configured to;

operate on the first portion of the audio data but not on the second portion of the audio data,

determine first audio data from a first microphone of the two or more microphones corresponds to a sound and was received by the first microphone at a first time,

determine second audio data from a second microphone of the two or more microphones corresponds to the sound and was received by the second microphone at a second time,

determine a delay between the first time and the second time,

determine a first direction based on the delay,

attenuate third audio data output by the two or more microphones within the first frequency range in substantially the first direction,

determine a second direction corresponds to an estimated position of a user, and

amplify fourth audio data within the first frequency range, the audio data corresponding to substantially the second direction; and

a second beamformer configured to;

operate on the second portion of the audio data but not on the first portion of the audio data, and

attenuate fifth audio data corresponding to sound captured by the two or more microphones within the second frequency range and corresponding to substantially a third direction.

9. The system of claim 8, further comprising an orchestrator to analyze output audio data of the two or more microphones to identify the first direction and the third direction based on the sound captured by the two or more microphones over an entirety of a third frequency range including the first frequency range and the second frequency range.

10. A device comprising:

two or more microphones to capture audio and output audio data;

a filter configured to filter the audio data into a first portion corresponding to a first frequency range and a second portion corresponding to a second frequency range different from the first frequency range;

a first beamformer configured to operate on the first portion of the audio data but not on the second portion of the audio data;

a second beamformer configured to operate on the second portion of the audio data but not on the first portion audio data;

one or more processors for processing output audio data of the two or more microphones; and

one or more computer-readable media storing computer-executable instructions which, when executed cause the one or more processors to process the output audio data of the two or more microphones to;

filter output of the two or more microphones into the first portion and the second portion;

identify a first sound signal associated with a user, the first sound signal within the first portion of the frequency range and corresponding to a sound that was received by a first microphone at a first time;

identify a second sound signal associated with the user, the second sound signal within the first portion of the frequency range and corresponding to the sound that was received by a second microphone at a second time;

determine a delay between the first time and the second time,

determine a first direction of the first sound signal relative to the device based on the delay, the first direction corresponding to a user;

amplify the first sound signal corresponding to the first direction; and

use the second beamformer to attenuate a third sound signal captured by the two or more microphones, the third sound signal within the second frequency range in and corresponding to a direction other than the first direction.

11. The device of claim 10, wherein the first sound signal associated with the user is identified based on a stored voice pattern of the user.

12. The device of claim 10, wherein the first sound signal associated with the user is identified based on a predetermined direction.

13. The device of claim 10, wherein a strongest signal, within the first frequency range, is identified as the first sound signal associated with the user.

14. The device of claim 10, wherein the computer-readable media stores instructions which, when executed by the one or more processors, cause the one or more processors to combine audio from the first frequency range and the second frequency range into an output signal.

15. The device of claim 10, wherein the direction other than the first direction is associated with sources of background noise.

16. The device of claim 10, wherein audio data output by the two or more microphones is attenuated by adding a weighted signal to the output of the two or more microphones.

17. One or more non-transitory computer-readable media storing computer-executable instructions which, when executed by one or more processors, cause the one or more processors to perform acts comprising:

receiving microphone output signals from two or more microphones;

filtering the microphone output signals into a first portion corresponding to a first frequency range and a second portion corresponding to a second frequency range different from the first frequency range;

identifying, within the first portion and using a stored voice pattern of the user, a first sound signal associated with a user;

identifying, within the first portion, a second sound signal not associated with the user; and

processing the first portion using a first beamformer configured to operate on data corresponding to the first frequency range but not on data corresponding to the second frequency range, to;

identify a first time that a sound associated with the first sound signal was received by a first microphone,

identify a second time that the sound was received by a second microphone,

determine a delay between the first time and the second time,

determine a first direction associated with the user based on the delay,

amplify the first sound signal corresponding to the first direction, and

attenuate the second sound signal.

18. The one or more non-transitory computer-readable media of claim 17, wherein identifying the first sound signal is based on a relative energy of sound captured by the two or more microphones represented in the microphone output signals.

19. The one or more non-transitory computer-readable media of claim 17, wherein identifying the first sound signal is based on a known direction.

20. One or more non-transitory computer-readable media storing computer-executable instructions which, when executed by one or more processors, cause the one or more processors to perform acts comprising:

receiving microphone output signals from two or more microphones;

filtering the microphone output signals into a first portion corresponding to a first frequency range and a second portion corresponding to a second frequency range different from the first frequency range;

identifying, within the first portion of the frequency range and using a stored voice pattern of the user, a first sound signal associated with a user from a first direction;

identifying, within the second portion, a second sound signal not associated with the user;

processing the first portion audio using a first beamformer configured to operate on data corresponding to the first frequency range but not on data corresponding to the second frequency range, to;

identify a first time that a sound associated with the first sound signal was received by a first microphone,

identify a second time that the sound was received by a second microphone,

determine a delay between the first time and the second time,

determine the first direction associated with the user based on the delay, and

amplify the first sound signal from the first direction; and

processing the second portion using a second beamformer configured to operate on data corresponding to the second frequency range but not on data corresponding to the first frequency range, to attenuate the second sound signal.