System and process for sound source localization using microphone array beamsteering

• US 7,394,907 B2
• Filed: 06/16/2003
• Issued: 07/01/2008
• Est. Priority Date: 06/16/2003
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Tashev, Ivan
• Primary Examiner(s): Chin; Vivian
• Assistant Examiner(s): Tran; Con P
• Application Number:
  US10/462,324
• Publication Number:
  US 20040252845A1
• Time in Patent Office: 1,842 Days
• Field of Search: 381/92, 381/91, 381/56, 381/122, 348/14.01, 348/14.08, 379/202.01, 379/206.01
• US Class Current: 381/92
• CPC Class Codes: G01S 3/8034 wherein the signals are derived simultaneously
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407

Abstract

A system and process for sound source localization (SSL) utilizing beamsteering is presented. The present invention provides for improved beamsteering with less drain on system resources while providing accurate, real time results. To accomplish this, the present SSL system and process rejects as much as possible extraneous audio frames and analyzes only those frames exhibiting a well defined sound source. In addition, the number of beams is reduced as much as possible to save on processing time, but a full scan of the working volume is still made with the beams. And finally, interpolation is used to increase the precision of the technique.

Claims

1. A system for finding the direction to a sound source from a microphone array in a prescribed search area, comprising:

a microphone array having a plurality of audio sensors;

a general purpose computing device;

a computer-readable medium having computer-executable instructions, wherein the computing device is directed by the instructions of the computer-readable medium to,

(a) package the signals produced by the array microphones into frames,

(b) select the frames representing the earliest captured frame of each microphone signal which have not been previously processed,

(c) subject the selected frames to a noise suppression procedure,

(d) classify the frames based on whether they exhibit evidence of a significant sound source,

(e) whenever it is found that the frames do not exhibit evidence of a significant sound source, repeat program modules (b) through (e) as appropriate,

(f) whenever it is found that the frames do exhibit evidence of a significant sound source,

compute a full spectrum energy comprising an energy value for each of a prescribed set of audio frequency sub-bands within an overall frequency band for a beam in each of a prescribed number of directions, wherein a beam represents a region surrounding a direction extending away from the microphone array wherein sounds originating within that region remain unaffected, while sounds originating outside that region are suppressed, and wherein the prescribed number of directions represent a set of search directions extending away from the microphone array which when a beam is associated with each of the search directions covers the entire search area without significant overlap between adjacent beams extending along those directions, and

for each computed energy value, apply a weighting factor associated with the frequency sub-band of the computed energy value, from a set of weighting factors comprising one for each of said set of audio frequency sub-bands, to the computed energy value, wherein each weighting factor reflects an energy level which is expected to be exhibited by the sound source at the frequency sub-band associated with the weighting factor relative to all the other sub-bands within the overall frequency band,

(g) identify the search direction exhibiting the maximum energy across the overall frequency band, and

(h) designate the maximum energy direction as the direction to the sound source.

2. The system of claim 1, wherein the program module for classifying the frames based on whether they exhibit evidence of a significant sound source, comprises sub-modules for:

computing the overall energy exhibited by the currently selected frames;

computing the noise floor energy value associated with the selected frames;

determining whether the overall energy exceeds a prescribed multiple of the noise floor energy value;

whenever the overall energy does not exceed the prescribed multiple of the noise floor energy value, designating that the currently selected frames do not exhibit evidence of a significant sound source; and

whenever the overall energy does exceed the prescribed multiple of the noise floor energy value, designating that the currently selected frames do exhibit evidence of a significant sound source.

3. The system of claim 1, wherein the region associated with each beam that remains unaffected while sounds originating outside that region are suppressed has the same size and shape as the other beams, and wherein the search directions associated with the beams represent evenly distributed angles about a center of the microphone array and wherein the number of beams employed represents the minimum number of beams possible while still covering the entire search area.

4. The system of claim 1, wherein the sound source is a human voice, and wherein a graph of the set of weighting factors, each of which ranges between 0 and 1, plotted against the set of audio frequency sub-bands represents a bell shaped curve with a maximum at about 1000 Hz and limits at about 500 and 1500 Hz.

5. The system of claim 1, wherein each weighting factor S(f) is defined as

$S$

( f ) = { 0.5 + 0.5 ⁢

⁢

cos ⁡

( 2 ⁢

⁢

π

⁡

( f - 1000 ) 1000 ) for ⁢

⁢

f ⁢

⁢

between ⁢

⁢

about 500 ⁢

⁢

and ⁢

⁢

1500 ⁢

⁢

Hz 0 ⁢

otherwise ,

wherein f is the frequency associated with the weighting factor sub-band.

6. The system of claim 1, wherein the program module for identifying the search direction exhibiting the maximum energy, comprises sub-modules for:

identifying the greatest weighted energy value computed for a beam;

identifying the smallest weighted energy value computed for a beam;

determining if there is at least a prescribed energy difference between the greatest and smallest weighted energy values;

repeating program modules (b) through (h) as appropriate, whenever it is found that the difference between the greatest and smallest weighted energy values does not exceed the prescribed energy difference; and

designating the search direction exhibiting the maximum energy as being the search direction associated with the beam identified as having the greatest weighted energy, whenever it is found that the difference between the greatest and smallest weighted energy values does exceed the prescribed energy difference.

7. The system of claim 6, wherein the sub-module for determining if there is at least the prescribed energy difference between the greatest and smallest weighted energy values, comprises sub-modules for:

computing the average weighted energy of all the beams;

computing the difference between the greatest and smallest weighted energy values and dividing said difference by the average weighted energy to produce an energy ratio;

determining if the energy ratio is less than a prescribed constant representing the minimum allowed energy difference;

designating that there is at least the prescribed energy difference between the greatest and smallest weighted energy values if it is determined the energy ratio equals or exceeds the prescribed constant; and

designating that there is not at least the prescribed energy difference between the greatest and smallest weighted energy values if it is determined the energy ratio is less than the prescribed constant.

8. The system of claim 7, wherein the prescribed constant is about 0.3.

9. The system of claim 8, wherein the program module for designating the maximum energy direction as the direction to the sound source comprises a sub-module for interpolating between the search direction exhibiting the maximum energy and two or more adjacent search directions to refine the direction from the microphone array that represents the sound source direction.

10. The system of claim 9, wherein the sub-module for interpolating between the search direction exhibiting the maximum energy and two or more adjacent search directions to refine the direction from the microphone array that represents the sound source direction, comprises sub-modules for:

characterizing the search direction exhibiting the maximum energy and said two or more adjacent search directions as an approximation surface which can be defined as a polynomial having an order equal to the number of neighboring search directions, such that for each search direction considered, the energy associated with that direction is set equal to a set of unknown coefficients and that direction in polynomial form thereby creating a linear system of equations equaling the number of unknown coefficients;

solving the linear system of equations to determine the unknown coefficients;

estimating a maximum of the approximation surface as the direction associated with a zero of the first derivative of said polynomial equation; and

designating the direction associated with the estimated approximation surface maximum as the sound source direction.

11. The system of claim 1, comprising a program module for refining the sound source direction by respectively combining the sound source direction computed over a series of frame sets.

12. The system of claim 11, wherein the program module for refining the sound source direction by respectively combining the sound source direction computed over a series of frame sets, comprises one of (i) averaging said sound source directions, (ii) subjecting the sound source directions to a Kalman filtering procedure, or (iii) subjecting the sound source directions to a particle filtering procedure.

13. A computer-readable medium having computer-executable instructions;

for finding the direction to a sound source from a microphone array in a prescribed search area using a beamsteering approach, said computer-executable instructions comprising using a computer to perform the following process actions;

determining the minimum number of beams it would take to cover the entire search area without significant overlap;

establishing a set of search directions extending away from the microphone array which equal said minimum number of beams and which are spaced equally from each other in such a way as to cover the entire search area while preventing any significant overlap between adjacent beams extending along those directions;

estimating the audio energy within the beams associated with each of the set of search directions, said estimation comprising computing an energy estimate as

$E_{\mathrm{est}}<br><br>\left(d_l\right)={\int <br><br>}_0^{\frac{f_s}{2}}<br><br>E_{\mathrm{est}}<br><br>\left(d_l,f\right)<br><br>S<br><br>\left(f\right)<br><br><br><br>d<br><br>f,$

wherein f_s is a prescribed sampling frequency, d_l is a search direction, f belongs to a prescribed set of audio frequencies and S(f) is a weight function associated with frequency f, and wherein S(f) ranges between 0 and 1 and reflects an energy level at frequency f which is expected to be exhibited by the sound source relative to all the other frequencies within said set of audio frequencies;

identifying the search direction exhibiting the maximum energy;

designating the maximum energy direction as the direction to the sound source.

14. The computer-readable medium of claim 13, further comprising an instruction for segmenting the signals produced by the array microphones into sets of contemporaneously captured frames prior to performing the computing, identifying and designating process actions, and wherein the computing, identifying and designating process actions are performed on a frame set-by-frame set basis.

15. The process of claim 14, further comprising the process actions of:

identifying for each frame set whether it exhibits evidence of a significant sound source related to the type of sound being sought; and

eliminating any frame set that is deemed to lack evidence of said significant sound source from consideration prior to performing the computing, identifying and designating process actions.

16. The process of claim 13, wherein all directions are measured in spherical coordinates comprising pitch and yaw angles from an origin at the microphone array.

17. The process of claim 16, wherein the prescribed search area comprises a hemisphere.

18. The process of claim 16, wherein the pitch angle is fixed and the prescribed search area comprises a 360 degree excursion of the yaw angle.

19. The process of claim 16, wherein the pitch angle is fixed and the prescribed search area comprises less than a 360 degree excursion of the yaw angle.

20. The process of claim 13, wherein each beam has the same size and shape.

21. A computer-readable medium having computer-executable instructions for finding the direction to a sound source from a microphone array in a prescribed search area, said computer-executable instructions comprising:

determining the minimum number of beams it will take to cover the entire search area without significant overlap, wherein a beam represents a region surrounding a direction extending away from the microphone array wherein sound originating within that region remain unaffected, while sounds originating outside that region are suppressed;

establishing a set of search directions extending away from the microphone array which equal the minimum number of beams and which are spaced equally from each other in such a way as to prevent any significant overlap between adjacent beams extending along those directions;

estimating the audio energy within the beams associated with each of the set of search directions, said estimation comprising computing an energy estimate as

$E_{\mathrm{est}}<br><br>\left(d_l\right)={\int <br><br>}_0^{\frac{f_s}{2}}<br><br>E_{\mathrm{est}}<br><br>\left(d_l,f\right)<br><br>S<br><br>\left(f\right)<br><br><br><br>d<br><br>f,$

wherein f_s is a prescribed sampling frequency, d_l is a search direction, f belongs to a prescribed set of audio frequencies and S(f) is a weight function associated with frequency f, and wherein S(f) ranges between 0 and 1 and reflects an energy level at frequency f which is expected to be exhibited by the sound source relative to all the other frequencies within said set of audio frequencies;

identifying the search direction exhibiting the maximum energy; and

designating the maximum energy direction as the direction to the sound source.