Signal-enhancing beamforming in an augmented reality environment

• US 9,973,848 B2
• Filed: 06/21/2011
• Issued: 05/15/2018
• Est. Priority Date: 06/21/2011
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Chhetri, Amit S., Velusamy, Kavitha, Crump, Edward Dietz
• Primary Examiner(s): Chin, Vivian
• Assistant Examiner(s): Tran, Con P
• Application Number:
  US13/165,620
• Publication Number:
  US 20120327115A1
• Time in Patent Office: 2,520 Days
• Field of Search: 381 92, 381 2, 381 77, 381 81, 381 82, 381 85, 345633, 345632, 345629, 345619, 700 94, 349 5, 352104, 352138, 352 25, 352133, 352 11
• CPC Class Codes: H04R 1/406 microphones
• CPC Class Codes: H04R 2201/401 2D or 3D arrays of transducers
• CPC Class Codes: H04R 2201/403 Linear arrays of transducers
• CPC Class Codes: H04R 2430/20 Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic H04R2203/12 takes precedence
• CPC Class Codes: H04R 2430/21 Direction finding using differential microphone array [DMA]
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407

Abstract

An augmented reality environment allows interaction between virtual and real objects. Beamforming techniques are applied to signals acquired by an array of microphones to allow for simultaneous spatial tracking and signal acquisition from multiple users. Localization information such as from other sensors in the environment may be used to select a particular set of beamformer coefficients and resulting beampattern focused on a signal source. Alternately, a series of beampatterns may be used iteratively to localize the signal source in a computationally efficient fashion. The beamformer coefficients may be pre-computed.

Claims

1. An augmented reality system comprising:

a processor;

a microphone array comprising a plurality of microphones coupled to the processor, a first microphone of the plurality of microphones configured to generate first signal data from a first audio signal source and a second microphone of the plurality of microphones configured to generate second signal data from a second audio signal source;

a projector coupled to the processor and configured to generate structured light;

a camera coupled to the processor and configured to receive the structured light;

a beamformer coefficient datastore configured to store a set of beamformer coefficients, individual beamformer coefficients of the set of beamformer coefficients being associated with a different beampattern of one or more beampatterns; and

one or more computer-executable instructions that are executable by the processor to;

determine a first location of the first audio signal source and a second location of the second audio signal source;

select a first set of beamformer coefficients from the beamformer coefficient datastore based at least in part upon the first location of the first audio signal source and first directional data associated with the first audio signal source, the first set of beamformer coefficients corresponding to a first beampattern of the one or more beampatterns; and

select a second set of beamformer coefficients from the beamformer coefficient datastore based at least in part upon the second location of the second audio signal source and second directional data associated with the second audio signal source, the second set of beamformer coefficients corresponding to a second beampattern of the one or more beampatterns, the first beampattern causing an attenuation of noise output by the second audio signal source based at least in part on a distance between the first location and the second location.

2. The system of claim 1, wherein each of the one or more beampatterns includes a main lobe, and wherein the one or more computer-executable instructions are further executable by the processor to select the first beampattern by placing the first location of the first audio signal source within a main lobe of the first beampattern.

3. The system of claim 1, wherein each of the one or more beampatterns includes a null region, and wherein the one or more computer-executable instructions are further executable by the processor to select the first beampattern by placing the first location of the first audio signal source in a null region of the first beampattern.

4. The system of claim 1, wherein the one or more computer-executable instructions are further executable by the processor to select the first beampattern by determining that a main lobe beamwidth is proportionate to an accuracy of the first location of the first audio signal source.

5. The system of claim 1, wherein the plurality of microphones are configured to be placed in a planar arrangement when operational.

6. The system of claim 1, wherein the plurality of microphones are configured to be placed in a three-dimensional arrangement when operational.

7. The system of claim 1, wherein the one or more computer-executable instructions are further executable by the processor to apply the first set of beamformer coefficients associated with first beampattern to the first signal data to generate processed data.

8. The system of claim 7, wherein the one or more computer-executable instructions are further executable by the processor to filter the processed data.

9. The system of claim 7, wherein the one or more computer-executable instructions are further executable by the processor to determine an audible gesture based at least in part on the processed data.

10. The system of claim 1, further comprising a signal datastore configured to store signal data for processing.

11. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed, cause one or more processors to perform acts comprising:

acquiring, at a first microphone of a microphone array, first signal data from a first signal source;

acquiring, at a second microphone of the microphone array, second signal data from a second signal source;

determining a first location of the first signal source and a second location of the second signal source;

selecting a first set of beamformer coefficients based at least in part upon the first location of the first signal source and first directional data associated with the first signal source, the first set of beamformer coefficients corresponding to a first beampattern; and

selecting a second set of beamformer coefficients based at least in part upon the second location of the second signal source and second directional data associated with the second signal source, the second set of beamformer coefficients corresponding to a second beampattern, the first beampattern causing an attenuation of at least one of noise or echo output by the second signal source based at least in part on a distance between the first location and the second location.

12. The one or more non-transitory computer-readable storage media of claim 11, wherein at least one of the first set of beamformer coefficients or the second set of beamformer coefficients are calculated prior to acquiring at least one of the first signal data or the second signal data.

13. The one or more non-transitory computer-readable storage media of claim 11, the acts further comprising:

determining an imprecise direction of at least one of the first signal source or the second signal source relative to the microphone array; and

generating processed data based at least in part on application of at least one of the first set of beamformer coefficients or the second set of beamformer coefficients.

14. The one or more non-transitory computer-readable storage media of claim 13, the acts further comprising analyzing the processed data.

15. The one or more non-transitory computer-readable storage media of claim 14, the analyzing comprising recognizing speech in the processed data.

16. The one or more non-transitory computer-readable storage media of claim 14, the analyzing comprising recognizing an audible gesture in the processed data.

17. The one or more non-transitory computer-readable storage media of claim 11, the acts further comprising selectively adjusting gain of at least one of the first microphone or the second microphone.

18. The one or more non-transitory computer-readable storage media of claim 17, wherein selectively adjusting the gain comprises altering analog gain of at least one of the first microphone or the second microphone.

19. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed, cause one or more processors to perform acts comprising:

acquiring first signal data of a first signal source from a first microphone of a microphone array;

acquiring second signal data of a second signal source from a second microphone of the microphone array;

determining a first location of the first signal source and a second location of the second audio signal source;

selecting a first set of beamformer coefficients based at least in part upon the first location of the first signal source and first directional data associated with the first signal source, the first set of beamformer coefficients corresponding to a first beampattern;

selecting a second set of beamformer coefficients based at least in part upon the second location of the second signal source and second directional data associated with the second signal source, the second set of beamformer coefficients corresponding to a second beampattern;

applying, to the first signal data, the first set of beamformer coefficients; and

applying, to the second signal data, the second set of beamformer coefficients, at least one of the first beampattern or the second beampattern causing at least one of an attenuation or an elimination of noise associated with the second signal data.

20. The one or more non-transitory computer-readable storage media of claim 19, the acts further comprising determining one or more characteristics of the at least one of the first beampattern or the second beampattern, the one or more characteristics being associated with at least one of beampattern direction, topology, size, relative gain, or frequency response.

21. The one or more non-transitory computer-readable storage media of claim 19, the acts further comprising applying the first set of beamformer coefficients to the first signal data and applying the second set of beamformer coefficients to the second signal data in parallel.

22. The one or more non-transitory computer-readable storage media of claim 19, wherein the first beampattern encompasses a first volume.

23. The one or more non-transitory computer-readable storage media of claim 22, wherein the second beampattern encompasses a second volume that is disposed predominantly within the first volume.

24. The one or more non-transitory computer-readable storage media of claim 19, the acts further comprising:

determining that the first beampattern contains a first maximum signal strength from the first signal data as compared to first other beampatterns of a first set of beampatterns associated with the first signal data; and

determining that the second beampattern contains a second maximum signal strength from the second signal data as compared to second other beampatterns of a second set of beampatterns associated with the second signal data.