Sound source localization using multiple microphone arrays

• US 9,489,948 B1
• Filed: 03/13/2015
• Issued: 11/08/2016
• Est. Priority Date: 11/28/2011
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Chu, Wai C., Crump, Edward Dietz
• Primary Examiner(s): Chin, Vivian
• Assistant Examiner(s): Hamid, Ammar
• Application Number:
  US14/658,001
• Time in Patent Office: 606 Days
• Field of Search: 381/28, 381/92, 381/91, 381/122, 704/200, 704/233
• US Class Current: 1/1
• CPC Class Codes: G10L 15/28 Constructional details of speech recognition systems
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: H04R 1/08 Mouthpieces; Microphones; Attachments therefor
• CPC Class Codes: H04R 2227/003 Digital PA systems using, e.g. LAN or internet
• CPC Class Codes: H04R 2420/03 Connection circuits to selectively connect loudspeakers or headphones to amplifiers
• CPC Class Codes: H04R 2430/21 Direction finding using differential microphone array [DMA]
• CPC Class Codes: H04R 27/00 Public address systems circuits for preventing acoustic reaction H04R3/02; circuits for distributing signals to loudspeakers H04R3/12; monitoring or testing arrangements for public address systems H04R29/007; amplifiers H03F
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407

Abstract

An augmented reality environment allows interaction between virtual and real objects. Multiple microphone arrays of different physical sizes are used to acquire signals for spatial tracking of one or more sound sources within the environment. A first array with a larger size may be used to track an object beyond a threshold distance, while a second array having a size smaller than the first may be used to track the object up to the threshold distance. By selecting different sized arrays, accuracy of the spatial location is improved.

Claims

1. A method comprising:

determining, by a device, first signal data associated with a sound source;

determining, based at least partly on the first signal data, that a first distance between the device and the sound source exceeds a threshold distance;

selecting a first set of microphones that includes a first microphone and that is associated with the device to detect a first voice command from the sound source;

determining, by the device, second signal data associated with the sound source;

determining, based at least partly on the second signal data, that a second distance between the device and the sound source is less than or equal to the threshold distance; and

selecting a second set of microphones to detect a second voice command from the sound source, the second set of microphones including the first microphone and one or more other microphones that are associated with the device but that are not included in the first set of microphones.

2. The method as recited in claim 1, further comprising:

determining a candidate location of the sound source using a depth sensor;

determining a location of the sound source; and

setting the threshold distance based at least partly on a correspondence of the candidate location to the location.

3. The method as recited in claim 1, further comprising:

determining that the first microphone is active;

determining that a second microphone of the one or more other microphones is inactive; and

creating the second set of microphones by including the first microphone in the one or more other microphones and excluding the second microphone from the one or more other microphones.

4. The method as recited in claim 3, further comprising determining that the second microphone is inactive by at least one of:

determining that the second microphone is not currently collecting data;

or

determining that the second microphone is currently collecting the data, but that the data is at least one of disregarded or discarded.

5. The method as recited in claim 1, further comprising determining that the first microphone is an omnidirectional microphone.

6. The method as recited in claim 1, further comprising determining a location of the sound source based at least partly on a time difference of arrival between the second microphone and a third microphone of the one or more other microphones.

7. A system comprising:

one or more processors;

memory;

a first set of microphones that is coupled to the one or more processors and that is configured to generate first signal data from a sound source, the first set of microphones including a first microphone;

a second set of microphones that is coupled to the one or more processors and that is configured to generate second signal data from the sound source, the second set of microphones including the first microphone and one or more other microphones that are not included in the first set of microphones; and

one or more modules stored in the memory and executable by the one or more processors to;

determine, based at least partly on the first signal data, that a first distance to the sound source exceeds a threshold distance; and

select the first microphone to detect a first voice command from the sound source.

8. The system as recited in claim 7, wherein:

the one or more modules are further executable by the one or more processors to;

determine, based at least partly on the second signal data, that a second distance to the sound source is less than or equal to the threshold distance; and

select the second set of microphones to detect a second voice command from the sound source.

9. The system as recited in claim 8, wherein the one or more modules are further executable by the one or more processors to:

determine that a second microphone of the one or more other microphones is active;

determine that a third microphone of the one or more other microphones is inactive; and

creating the second set of microphones by including the second microphone in the one or more other microphones and excluding the third microphone from the one or more other microphones.

10. The system as recited in claim 9, wherein the one or more modules are further executable by the one or more processors to at least one of:

determine that the third microphone is not currently collecting data;

or

determine that the third microphone is currently collecting the data, but that the data is at least one of disregarded or discarded.

11. The system as recited in claim 7, further comprising a depth sensor to determine a candidate location of the sound source, and wherein the one or more modules are further executable by the one or more processors to determine the threshold distance based at least partly on the candidate location.

12. The system as recited in claim 7, wherein the first microphone is an omnidirectional microphone.

13. The system as recited in claim 7, wherein the one or more modules are further executable by the one or more processors to determine a location of the sound source based at least partly on a time difference of arrival between the first microphone and a second microphone of the one or more other microphones.

14. A system comprising:

one or more processors;

memory;

a first set of microphones including multiple microphones that are coupled to the one or more processors and that are configured to generate first signal data from a sound source; and

one or more modules stored in the memory and executable by the one or more processors to;

determine, based at least partly on the first signal data, that a first distance to the sound source is less than or equal to a threshold distance; and

select the first set of microphones to detect a first voice command from the sound source, the multiple microphones of the first set of microphones including a common microphone that is also included in a second set of microphones.

15. The system as recited in claim 14, wherein:

the second set of microphones are coupled to the one or more processors and are configured to generate second signal data from the sound source; and

the one or more modules are further executable by the one or more processors to;

determine, based at least partly on the second signal data, that a second distance to the sound source exceeds the threshold distance; and

select the second set of microphones to detect a second voice command from the sound source.

16. The system as recited in claim 14, wherein the one or more modules are further executable by the one or more processors to:

determine that the common microphone is active;

determine that a second microphone of the multiple microphones is inactive; and

creating the first set of microphones by including the common microphone in the multiple microphones and excluding the second microphone from the multiple microphones.

17. The method as recited in claim 16, wherein the one or more modules are further executable by the one or more processors to at least one of:

determine that the second microphone is not currently collecting data;

or

determine that the second microphone is currently collecting the data, but that the data is at least one of disregarded or discarded.

18. The method as recited in claim 14, wherein the common microphone is an omnidirectional microphone.

19. The method as recited in claim 14, wherein the one or more modules are further executable by the one or more processors to determine a location of the sound source based at least partly on a time difference of arrival between the common microphone and a second microphone of the multiple microphones.

20. The method as recited in claim 14, wherein the first set of microphones is determined based at least partly on a current location of each microphone included in the multiple microphones.