System and process for locating a speaker using 360 degree sound source localization

• US 20040037436A1
• Filed: 08/26/2002
• Published: 02/26/2004
• Est. Priority Date: 08/26/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Rui, Yong
• Granted Patent: US 7,039,199 B2
• Time in Patent Office: Days
• US Class Current: 381/92
• CPC Class Codes: H04R 2201/401 2D or 3D arrays of transducers
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407

Abstract

A system and process is described for estimating the location of a speaker using signals output by a microphone array characterized by multiple pairs of audio sensors. The location of a speaker is estimated by first determining whether the signal data contains human speech components and filtering out noise attributable to stationary sources. The location of the person speaking is then estimated using a time-delay-of-arrival based SSL technique on those parts of the data determined to contain human speech components. A consensus location for the speaker is computed from the individual location estimates associated with each pair of microphone array audio sensors taking into consideration the uncertainty of each estimate. A final consensus location is also computed from the individual consensus locations computed over a prescribed number of sampling periods using a temporal filtering technique.

Claims

1. A computer-implemented process for finding the location of a person speaking using signals output by a microphone array having a plurality of audio sensors, comprising using a computer to perform the following process actions:

inputting the signal generated by each audio sensor of the microphone array;

distinguishing the portion of each of the array sensor signals that contains human speech data from non-speech portions; and

reducing noise attributable to stationary sources in each of the array sensor signals;

locating the position of the person speaking using a time-delay-of-arrival (TDOA) based sound source localization (SSL) technique on those portions of the array sensor signals that contain human speech data.

2. The process of claim 1, wherein the process action of distinguishing the portion of each of the array sensor signals that contains human speech data from the non-speech portions, comprises, for each array sensor signal, the actions of:

sampling the signal to produce a sequence of consecutive blocks of the signal data representing the output of the sensor over a prescribed period of time;

converting each block of signal data to the frequency domain;

initializing the distinguishing action using three consecutive blocks of signal data, said initializing comprising the actions of,

computing the total energy of the blocks,

computing the delta energy of the third block in the sequence by computing the difference between the total energy of said third block and that of the second block in the sequence,

computing a noise floor energy for the second and third blocks, and

computing the delta energy of the noise floor for the third block which represents the difference of the noise floor energy value computed for the third and that computed for the second block; and

for each consecutive block of signal data starting with the third block employed in the initialization action,

computing the total energy of the block if not previously computed,

computing the delta energy of the block if not previously computed, wherein the delta energy represents the difference in total energy between the block under consideration and that of the immediately preceding block of signal data,

computing the delta energy of the noise floor of the block if not previously computed, wherein the delta noise floor energy represents the difference between the last-computed noise floor energy value and that associated with the immediately preceding block of signal data,

determining whether the total energy of the block exceeds a prescribed multiple of the energy of the noise floor of the block and whether the delta energy of the block exceeds a prescribed multiple of the delta energy of the noise floor of the block, and

whenever it is determined that the total energy of the block exceeds the prescribed multiple of the energy of the noise floor of the block and the delta energy of the block exceeds the prescribed multiple of the delta energy of the noise floor of the block, designating the block as one containing human speech components.

3. The process of claim 2, wherein the prescribed multiple of the energy of the noise floor of the block ranges between about 3.0 and about 5.0.

4. The process of claim 2, wherein the prescribed multiple of the delta energy of the noise floor of the block ranges between about 3.0 and about 5.0.

5. The process of claim 2, further comprising, for each block of signal data, the process action of:

whenever it is determined that the total energy of the block does not exceed the prescribed multiple of the energy of the noise floor of the block and the delta energy of the block exceeds the prescribed multiple of the delta energy of the noise floor of the block, determining whether the total energy of the block is less than a second prescribed multiple of the energy of the noise floor of the block and whether the delta energy of the block is less than a second prescribed multiple of the delta energy of the noise floor of the block;

whenever it is determined that the total energy of the block is less than the second prescribed multiple of the energy of the noise floor of the block and the delta energy of the block is less than the second prescribed multiple of the delta energy of the noise floor of the block, designating the block as a noise block and updating the noise floor energy and delta noise floor energy values associated with the array signal from which the block under consideration was captured.

6. The process of claim 5, wherein the prescribed multiple of the energy of the noise floor of the block ranges between about 1.5 and about 2.0.

7. The process of claim 5, wherein the prescribed multiple of the delta energy of the noise floor of the block ranges between about 1.5 and about 2.0.

8. The process of claim 5, wherein the process action of updating the noise floor energy and delta noise floor energy values comprises the actions of:

determining whether the noise level is increasing or decreasing, wherein the noise level is deemed to be increasing whenever the block under consideration has a total energy value within said speech band that exceeds the total energy value within the speech band computed for the immediately preceding block of signal data, and the noise level is deemed to be decreasing whenever the block under consideration has a total energy value within said speech band that is less than the total energy value within the speech band computed for the immediately preceding block of signal data;

whenever the noise level is deemed to be increasing,

setting the noise floor energy equal to the last computed noise floor energy multiplied by a first prescribed factor and adding the product to the product of the last computed noise floor energy value and a value equal to one minus the first prescribed factor, and

setting the delta noise floor energy equal to the last computed delta noise floor energy multiplied by the first prescribed factor and adding the product to the product of the last computed delta noise floor energy value and a value equal to one minus the first prescribed factor; and

whenever the noise level is deemed to be decreasing,

setting the noise floor energy equal to the last computed noise floor energy multiplied by a second prescribed factor and adding the product to the product of the last computed noise floor energy value and a value equal to one minus the second prescribed factor, and

setting the delta noise floor energy equal to the last computed delta noise floor energy multiplied by the second prescribed factor and adding the product to the product of the last computed delta noise floor energy value and a value equal to one minus the second prescribed factor.

9. The process of claim 8, wherein the first prescribed factor is about 0.95, and the second prescribed factor is about 0.05.

10. The process of claim 2, wherein the process action of reducing noise attributable to stationary sources, comprises, for each block of signal data designated as one containing human speech components, the actions of:

performing a bandpass filtering operation which eliminates those frequencies not within the human speech range,

multiplying the block by a ratio representing the total energy of the block within said speech band less the computed noise floor energy associated with the block which is then divided by said total energy of the block.

11. The process of claim 10, wherein the microphone array has at least two synchronized pairs of audio sensors, and wherein the process action of sampling each array signal comprises sampling the signals output by each sensor in each synchronized pair of audio sensors so as to produced a sequence of consecutive, contemporaneous signal data block pairs from each pair of audio sensors.

12. The process of claim 11, wherein the process action of locating the position of the person speaking using those portions of the array sensor signals that contain human speech data, comprises the actions of:

for each contemporaneous signal data block pair sampled from the output of a pair of synchronized audio sensors which has blocks that have been designated as containing human speech components,

estimating the TDOA for the block pair under consideration using a generalized cross-correlation GCC technique,

computing a direction angle representing the angle between a line extending perpendicular to a baseline connecting the locations of the sensors of the audio sensor pair associated with the block pair under consideration from a point on the baseline between the sensors, and a line extending from said point to the apparent location of the speaker, wherein computing the direction angle comprises computing the arcsine of the TDOA estimate multiplied by the speed of sound in air and divided by the length of the baseline between the audio sensors associated with the block pair under consideration, and identifying a mirror angle for the computed direction angle defined as the angle formed between the line extending perpendicular to a baseline connecting the locations of the sensors of the audio sensor pair associated with the block pair under consideration from said point on the baseline between the sensors and a reflection of the line extending from said point to the apparent location of the speaker on the opposite side of the baseline between the sensors;

determining which of the direction angles associated with all the synchronized pairs of audio sensors and their identified mirror angles correspond to approximately the same direction;

deriving a final direction angle based on a weighted combination of the direction and mirror angles determined to correspond to approximately the same direction; and

designating the final direction angle as the location of the speaker.

13. The process of claim 12, wherein the process action of estimating the TDOA for the block pair under consideration using a generalized cross-correlation (GCC) technique, comprises the action of employing a weighting factor to compensate for background noise and reverberations when performing the GCC technique, wherein said weighting function is a combination of a maximum likelihood (ML) weighting function that compensates for background noise and a phase transformation (PHAT) weighting function that compensates for reverberations.

14. The process of claim 13, wherein the ML weighting function is combined with the PHAT weighting function by multiplying the PHAT function by a proportion factor ranging between 0 and 1.0 and multiplying the ML function by one minus the proportion fact, and adding the results, and wherein the proportion factor is selected to reflect the proportion of background noise to reverberations in the environment that the person speaking is present.

15. The process of claim 14, wherein the proportion factor is a fixed value and preset to approximately 0.3.

16. The process of claim 14, wherein the proportion factor is a dynamically selected by setting it equal to the proportion of noise in a block as represented by the previously computed noise floor of that block.

17. The process of claim 12, wherein the process action of deriving the final direction angle based on a weighted combination of the direction and mirror angles determined to correspond to approximately the same direction, comprises an action of assigning a weight to each angle based on how close the line extending from said point on the baseline connecting the locations of the sensors of the audio sensor pair associated with the angle to the estimated location of the speaker is to the line extending perpendicular to that baseline from said point, wherein the weight is greater the closer the lines are to each other.

18. The process of claim 12, wherein the process action of deriving the final direction angle based on a weighted combination of the direction and mirror angles determined to correspond to approximately the same direction, comprises the actions of:

converting the angles to a common coordinate system;

computing Gaussian probabilities to model each direction and mirror angle determined to correspond to approximately the same direction wherein for each of said angles θ

, μ

is the angle and σ

=1/(cos θ

) is an uncertainty factor;

combining the Gaussian probabilities and identifying which of the combined Gaussians represents the highest probability;

designating the μ

value of the identified Gaussian as the final direction angle.

19. The process of claim 12, wherein the process action of deriving the final direction angle based on a weighted combination of the direction and mirror angles determined to correspond to approximately the same direction, comprises the action of employing a maximum likelihood estimation procedure.

20. The process of claim 12, further comprising a process action of refining the location of the speaker, said refining action comprising:

deriving a final direction angle whenever the sensor signal data captured in a sampling period contains human speech data, for a prescribed number of consecutive sampling periods;

combining the individual computed final direction angles to produce a refined final direction angle using a temporal filtering technique; and

designating the refined final direction angle as the refined location of the speaker.

21. A system for estimating the location of a person speaking, comprising:

a microphone array having two or more audio sensor pairs;

a general purpose computing device;

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

input signals generated by each audio sensor of the microphone array;

simultaneously sample the inputted signals to produce a sequence of consecutive blocks of the signal data from each signal, wherein each block of signal data is captured over a prescribed period of time and is at least substantially contemporaneous with blocks of the other signals sampled at the same time;

for each block of noise filtered signal data, determine whether the block contains human speech data;

filter out noise attributable to stationary sources in each of the blocks of the signal data determined to contain human speech data;

estimate the location of the person speaking using a time-delay-of-arrival (TDOA) based sound source localization (SSL) technique on the contemporaneous blocks of filtered signal data determined to contain human speech data for each pair of audio sensors; and

compute a consensus estimated location for the person speaking from the individual location estimates determined from the contemporaneous blocks of filtered signal data found to contain human speech data of each pair of audio sensors.

22. The system of claim 21, further comprising a program module for refining the identified location of the person speaking, said refining module comprising sub-modules for:

computing said consensus location whenever the sensor signal data captured in a prescribed sampling period contains human speech data, for a prescribed number of consecutive sampling periods; and

combining the individual computed consensus locations to produce a refined estimate using a temporal filtering technique.

23. The system of claim 22, wherein the temporal filtering technique is one of (i) a median filtering technique, (ii) a kalman filtering technique, and (iii) a particle filtering technique.

24. The system of claim 21, wherein the computing device comprises a separate stereo-pair sound card for each of said pairs of audio sensors, and wherein for each sound card, the output of each sensor in the associated pair of sensor is input to the sound card and the outputs of the sensor pair are synchronized by the sound card.

25. The system of claim 24, wherein at least two of said two or more pairs of audio sensors are located such that each sensor of each of the two sensor pairs is separated from the other by a prescribed distance, which need not be the same distance for both pairs, and wherein said two pairs of sensors have baselines defined as the line connecting the two sensor of the audio sensor pair which intersect at an intersection point.

26. The system of claim 24, wherein the intersection point corresponds to a location in a space in which the person speaking is present that allows the location of the speaker to be estimated as being anywhere in a 360 degree sweep about the intersection point.

27. The system of claim 26, wherein the program module for estimating the location of the person speaking using a time-delay-of-arrival (TDOA) based sound source localization (SSL) technique on those contemporaneous blocks of signal data determined to contain human speech data for said two pairs of audio sensors comprises sub-modules for:

for each contemporaneous signal data block pair sampled from the output of said two pairs of synchronized audio sensors which has blocks that have been designated as containing human speech components,

estimating the TDOA for the block pair under consideration using a generalized cross-correlation GCC technique, and

computing a direction angle representing the angle between a line extending perpendicular to the baseline of the sensors of the audio sensor pair associated with the block pair under consideration from said intersection point, and a line extending from said intersection point to the apparent location of the speaker, wherein computing the direction angle comprises computing the arcsine of the TDOA estimate multiplied by the speed of sound in air and divided by the length of the baseline between the audio sensors associated with the block pair under consideration.

28. The system of claim 27, wherein the program module for computing the consensus estimated location for the person speaking, comprises sub-modules for:

identifying a mirror angle for the computed direction angle associated with each of said two pairs of synchronized audio sensors, wherein the mirror angle is defined as the angle formed between the line extending perpendicular to the baseline of the audio sensor pair under consideration from said intersection point and a reflection of the line extending from said intersection point to the apparent location of the speaker on the opposite side of the baseline;

determining which of the direction angles associated with said two synchronized pairs of audio sensors and their identified mirror angles correspond to approximately the same direction; and

deriving the consensus direction angle based on a weighted combination of the direction and mirror angles determined to correspond to approximately the same direction.

29. The system of claim 28, wherein the sub-module for deriving the consensus direction angle based on a weighted combination of the direction and mirror angles determined to correspond to approximately the same direction, comprises an action of assigning a weight to each angle based on how close the line extending from said intersection point on the baseline of the audio sensor pair associated with the angle to the estimated location of the speaker is to the line extending perpendicular to that baseline from the intersection point, wherein the weight is greater the closer the lines are to each other.

30. The system of claim 28, wherein the baselines of said two pairs of sensors are substantially perpendicular to each other.

31. A computer-readable medium having computer-executable instructions for estimating the location of a person speaking using signals output by a microphone array having a plurality of synchronized audio sensor pairs, said computer-executable instructions comprising:

inputting the signal generated by each audio sensor of the microphone array;

simultaneously sampling the inputted signals to produce a sequence of consecutive blocks of the signal data from each signal, wherein each block of signal data is captured over a prescribed period of time and is at least substantially contemporaneous with blocks of the other signals sampled at the same time;

for each group of contemporaneous blocks of signal data,

determining whether a block contains human speech data for each block of signal data,

filtering out noise attributable to stationary sources in each of the blocks determined to contain human speech data,

estimating the location of the person speaking using a time-delay-of-arrival (TDOA) based sound source localization (SSL) technique on those contemporaneous blocks of signal data determined to contain human speech data for each pair of synchronized audio sensors, and

computing a consensus estimated location for the person speaking from the individual location estimates determined from the contemporaneous blocks of filtered signal data found to contain human speech data of each pair of synchronized audio sensors;

computing a final consensus location of the person speaking using a temporal filtering technique to combine the individual consensus locations computed over a prescribed number of sampling periods; and

designating the final consensus location as the location of the person speaking.