Robust localization and tracking of simultaneously moving sound sources using beamforming and particle filtering

• US 20060245601A1
• Filed: 04/27/2005
• Published: 11/02/2006
• Est. Priority Date: 04/27/2005
• Status: Abandoned Application

• Current Assignee: Societe De Commercialisation Des Produits De La Recherche Appliquee - Socpra Sciences Et Genie, S.E.C.
• Original Assignee: Societe De Commercialisation Des Produits De La Recherche Appliquee - Socpra Sciences Et Genie, S.E.C.
• Inventors: Valin, Jean-Marc, Rouat, Jean, Michaud, Francois
• Application Number:
  US11/116,117
• Publication Number:
  US 20060245601A1
• Time in Patent Office: Days
• US Class Current: 381/92
• CPC Class Codes: G01S 5/22 Position of source determined by co-ordinating a plurality of position lines defined by path-difference measurements G01S5/28 takes precedence
• CPC Class Codes: H04R 2201/403 Linear arrays of transducers
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407

Abstract

The present invention relates to a system for localizing at least one sound source, comprising a set of spatially spaced apart sound sensors to detect sound from the at least one sound source and produce corresponding sound signals, and a frequency-domain beamformer responsive to the sound signals from the sound sensors and steered in a range of directions to localize, in a single step, the at least one sound source. The present invention is also concerned with a system for tracking a plurality of sound sources, comprising a set of spatially spaced apart sound sensors to detect sound from the sound sources and produce corresponding sound signals, and a sound source particle filtering tracker responsive to the sound signals from the sound sensors for simultaneously tracking the plurality of sound sources. The invention still further relates to a system for localizing and tracking a plurality of sound sources, comprising a set of spatially spaced apart sound sensors to detect sound from the sound sources and produce corresponding sound signals; a sound source detector responsive to the sound signals from the sound sensors and steered in a range of directions to localize the sound sources, and a particle filtering tracker connected to the sound source detector for simultaneously tracking the plurality of sound sources.

Claims

1. A system for localizing and tracking a plurality of sound sources, comprising:

a set of spatially spaced apart sound sensors to detect sound from the sound sources and produce corresponding sound signals;

a sound source detector responsive to the sound signals from the sound sensors and steered in a range of directions to localize the sound sources; and

a particle filtering tracker connected to the sound source detector for simultaneously tracking the plurality of sound sources.

2. A sound source localizing and tracking system as defined in claim 1, wherein the set of sound sensors comprises a predetermined number of omnidirectional microphones arranged in a predetermined array.

3. A sound source localizing and tracking system as defined in claim 1, wherein the sound source detector is a frequency-domain steered beamformer.

4. A sound source localizing and tracking system as defined in claim 3, wherein the steered beamformer comprises:

a calculator of sound power spectra and cross-power spectra of sound signal samples in overlapping windows;

a calculator of cross-correlations by averaging the cross-power spectra over a given period of time;

a calculator of an output energy of the steered beamformer from the calculated cross-correlations; and

a finder of a loudest sound source localized in a given direction, the given direction of the loudest sound source being found by maximizing the output energy of the steered beamformer.

5. A sound source localizing and tracking system as defined in claim 4, wherein the calculator of cross-correlations comprises:

a calculator for computing, in the frequency domain, whitened cross-correlations; and

a weighting function applied to the calculated whitened cross-correlations to act as a mask based on a signal-to-noise ratio.

6. A sound source localizing and tracking system as defined in claim 5, wherein the weighting function is modified to include a reverberation term in a noise estimate in order to make the system more robust to reverberation.

7. A sound source localizing and tracking system as defined in claim 3, wherein the steered beamformer produces an output energy and comprises:

a uniform triangular grid for the surface of a sphere to define directions;

a calculator of sound power spectra and cross-power spectra of sound signal samples in overlapping windows;

a calculator of cross-correlations by averaging the cross-power spectra over a given period of time;

a first algorithm for searching a best direction on the grid of the sphere;

a pre-computed table of time delays of arrival for each pair of sound sensors and each direction on the grid of the sphere; and

a finder of a loudest sound source in a direction of the grid of the sphere, the direction of the loudest sound source being found using the first algorithm and the pre-computed table by maximizing the output energy of the steered beamformer.

8. A sound source localizing and tracking system as defined in claim 7, further comprising a second algorithm for finding another sound source after having removed the contribution of the loudest sound source located by the finder.

9. A sound source localizing and tracking system as defined in claim 7, wherein the steered beamformer further comprises:

a refined grid for the surrounding of a point where a sound source was found in order to find a direction of localization of the found sound source with improved accuracy.

10. A sound source localizing and tracking system as defined in claim 1, wherein the particle filtering tracker models each sound source using a number of particles having respective directions and weights.

11. A sound source localizing and tracking system as defined in claim 1, wherein the particle filtering tracker comprises:

a calculator of a probability that a potential source is a real source.

12. A sound source localizing and tracking system as defined in claim 1, wherein the particle filtering tracker comprises:

a calculator of a probability that a real source corresponds to a potential source detected by the sound source detector.

13. A sound source localizing and tracking system as defined in claim 10, wherein the particle filtering tracker comprises:

a calculator of (a) at least one of a probability that a sound source is observed and a probability that a real sound source corresponds to a potential sound source, and (b) a probability density of observing a sound source at a given particle position; and

a calculator of updated particle weights in response to said probability density and said at least one probability.

14. A sound source localizing and tracking system as defined in claim 1, wherein the particle filtering tracker comprises:

an adder of a new source when a probability that the new source is real is higher than a first threshold.

15. A sound source localizing and tracking system as defined in claim 14, wherein the sound source localizing and tracking system assumes that the added new source exists if a probability of existence of said new source reaches a second threshold.

16. A sound source localizing and tracking system as defined in claim 1, wherein the particle filtering tracker comprises:

a subtractor of a source when the latter source has not been observed for a certain period of time.

17. A sound source localizing and tracking system as defined in claim 13, wherein the particle filtering tracker comprises:

an estimator of a position of each source as a weighted average of the positions of its particles, said estimator being responsive to the calculated, updated particle weights.

18. A system for localizing at least one sound source, comprising:

a set of spatially spaced apart sound sensors to detect sound from said at least one sound source and produce corresponding sound signals; and

a frequency-domain beamformer responsive to the sound signals from the sound sensors and steered in a range of directions to localize, in a single step, said at least one sound source.

19. A sound source localizing system as defined in claim 18, wherein the set of sound sensors comprises a predetermined number of omnidirectional microphones arranged in a predetermined array.

20. A sound source localizing system as defined in claim 18, wherein the steered beamformer comprises:

a calculator of sound power spectra and cross-power spectra of sound signal samples in overlapping windows;

a calculator of cross-correlations by averaging the cross-power spectra over a given period of time;

a calculator of an output energy of the steered beamformer from the calculated cross-correlations; and

a finder of a loudest sound source localized in a given direction, the given direction of the loudest sound source being found by maximizing the output energy of the steered beamformer.

21. A sound source localizing system as defined in claim 20, wherein the calculator of cross-correlations comprises:

a calculator for computing, in the frequency domain, whitened cross-correlations; and

a weighting function applied to the calculated whitened cross-correlations to act as a mask based on a signal-to-noise ratio.

22. A sound source localizing system as defined in claim 21, wherein the weighting function is modified to include a reverberation term in a noise estimate in order to make the system more robust to reverberation.

23. A sound source localizing and tracking system as defined in claim 18, wherein the steered beamformer produces an output energy and comprises:

a uniform triangular grid for the surface of a sphere to define directions;

a calculator of sound power spectra and cross-power spectra of sound signal samples in overlapping windows;

a calculator of cross-correlations by averaging the cross-power spectra over a given period of time;

a first algorithm for searching a best direction on the grid of the sphere;

a pre-computed table of time delays of arrival for each pair of sound sensors and each direction on the grid of the sphere; and

a finder of a loudest sound source in a direction of the grid of the sphere, the direction of the loudest sound source being found using the first algorithm and the pre-computed table by maximizing the output energy of the steered beamformer.

24. A sound source localizing system as defined in claim 23, further comprising a second algorithm for finding another sound source after having removed the contribution of the loudest sound source located by the finder.

25. A sound source localizing and tracking system as defined in claim 23, wherein the steered beamformer further comprises:

a refined grid for the surrounding of a point where a sound source was found in order to find a direction of localization of the found sound source with improved accuracy.

26. A system for tracking a plurality of sound sources, comprising:

a set of spatially spaced apart sound sensors to detect sound from the sound sources and produce corresponding sound signals; and

a sound source particle filtering tracker responsive to the sound signals from the sound sensors for simultaneously tracking the plurality of sound sources.

27. A sound source tracking system as defined in claim 26, wherein the particle filtering tracker models each sound source using a number of particles having respective directions and weights.

28. A sound source tracking system as defined in claim 26, wherein the particle filtering tracker comprises:

a calculator of a probability that a potential source is a real source.

29. A sound source tracking system as defined in claim 26, wherein the particle filtering tracker comprises:

a calculator of a probability that a real source corresponds to a potential source.

30. A sound source tracking system as defined in claim 27, wherein the particle filtering tracker comprises:

a calculator of (a) at least one of a probability that a sound source is observed and a probability that a real sound source corresponds to a potential sound source, and (b) a probability density of observing a sound source at a given particle position; and

a calculator of updated particle weights in response to said probability density and said at least one probability.

31. A sound source tracking system as defined in claim 26, wherein the particle filtering tracker comprises:

an adder of a new source when a probability that the new source is real is higher than a first threshold.

32. A sound source tracking system as defined in claim 31, wherein the sound source tracking system assumes that the added new source exists if a probability of existence of said new source reaches a second threshold.

33. A sound source tracking system as defined in claim 26, wherein the particle filtering tracker comprises:

a subtractor of a source when the latter source has not been observed for a certain period of time.

34. A sound source tracking system as defined in claim 30, wherein the particle filtering tracker comprises:

an estimator of a position of each source as a weighted average of the positions of its particles, said estimator being responsive to the calculated, updated particle weights.

35. A method for localizing and tracking a plurality of sound sources, comprising:

detecting sound from the sound sources through a set of spatially spaced apart sound sensors to produce corresponding sound signals;

localizing the sound sources in response to the sound signals, localizing the sound sources including steering in a range of directions a sound source detector having an output; and

simultaneously tracking the plurality of sound sources, using particle filtering, in relation to the output from the sound source detector.

36. A sound source localizing and tracking method as defined in claim 35, wherein steering a sound source detector comprises steering a frequency-domain beamformer.

37. A sound source localizing and tracking method as defined in claim 36, wherein localizing the sound sources comprises:

computing sound power spectra and cross-power spectra of sound signal samples in overlapping windows;

computing cross-correlations by averaging the cross-power spectra over a given period of time;

computing an output energy of the steered beamformer from the calculated cross-correlations; and

finding a loudest sound source localized in a given direction, the given direction of the loudest sound source being found by maximizing the output energy of the steered beamformer.

38. A sound source localizing and tracking method as defined in claim 37, wherein computing the cross-correlations comprises:

computing, in the frequency domain, whitened cross-correlations; and

applying a weighting function to the computed whitened cross-correlations to act as a mask based on a signal-to-noise ratio.

39. A sound source localizing and tracking method as defined in claim 38, comprising modifying the weighting function by including a reverberation term in a noise estimate in order to make the method more robust to reverberation.

40. A sound source localizing and tracking method as defined in claim 36, wherein localizing the sound sources comprises:

defining a uniform triangular grid for the surface of a sphere to define directions;

computing sound power spectra and cross-power spectra of sound signal samples in overlapping windows;

computing cross-correlations by averaging the cross-power spectra over a given period of time;

pre-computing a table of time delays of arrival for each pair of sound sensors and each direction on the grid of the sphere; and

finding a loudest sound source in a direction of the grid of the sphere, finding the loudest sound source comprising searching a best direction on the grid of the sphere using a first algorithm and the pre-computed table by maximizing an output energy of the steered beamformer.

41. A sound source localizing and tracking method as defined in claim 40, comprising finding another sound source, using a second algorithm, after having removed the contribution of the located, loudest sound source.

42. A sound source localizing and tracking method as defined in claim 40, wherein localizing the sound sources further comprises:

defining a refined grid for the surrounding of a point where a sound source was found in order to find a direction of localization of the found sound source with improved accuracy.

43. A sound source localizing and tracking method as defined in claim 35, wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises modeling each sound source using a number of particles having respective directions and weights.

44. A sound source localizing and tracking method as defined in claim 35, wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises:

computing a probability that a potential source is a real source.

45. A sound source localizing and tracking method as defined in claim 35, wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises:

computing a probability that a real source corresponds to a potential source detected by the sound source detector.

46. A sound source localizing and tracking method as defined in claim 43, wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises:

computing (a) at least one of a probability that a sound source is observed and a probability that a real sound source corresponds to a potential sound source, and (b) a probability density of observing a sound source at a given particle position; and

computing updated particle weights in response to said probability density and said at least one probability.

47. A sound source localizing and tracking method as defined in claim 35, wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises:

adding a new source when a probability that the new source is real is higher than a first threshold.

48. A sound source localizing and tracking method as defined in claim 47, wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises assuming that the added new source exists if a probability of existence of said new source reaches a second threshold.

49. A sound source localizing and tracking method as defined in claim 35, wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises:

removing a sound source when the latter source has not been observed for a certain period of time.

50. A sound source localizing and tracking method as defined in claim 43, wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises:

estimating a position of each source as a weighted average of the positions of its particles, said estimator being responsive to the calculated, updated particle weights.

51. A method for localizing at least one sound source, comprising:

detecting sound from said at least one sound source through a set of spatially spaced apart sound sensors to produce corresponding sound signals; and

localizing, in a single step, said at least one sound source in response to the sound signals, localizing said at least one sound source including steering a frequency-domain beamformer in a range of directions.

52. A sound source localizing method as defined in claim 51, wherein localizing, in a single step, said at least one sound source comprises:

computing sound power spectra and cross-power spectra of sound signal samples in overlapping windows;

computing cross-correlations by averaging the cross-power spectra over a given period of time;

computing an output energy of the steered beamformer from the calculated cross-correlations; and

finding a loudest sound source localized in a given direction, the given direction of the loudest sound source being found by maximizing the output energy of the steered beamformer.

53. A sound source localizing method as defined in claim 52, wherein computing the cross-correlations comprises:

computing, in the frequency domain, whitened cross-correlations; and

applying a weighting function to the computed whitened cross-correlations to act as a mask based on a signal-to-noise ratio.

54. A sound source localizing method as defined in claim 53, comprising modifying the weighting function by including a reverberation term in a noise estimate in order to make the method more robust to reverberation.

55. A sound source localizing method as defined in claim 51, wherein localizing, in a single step, said at least one sound source comprises:

defining a uniform triangular grid for the surface of a sphere to define directions;

computing sound power spectra and cross-power spectra of sound signal samples in overlapping windows;

computing cross-correlations by averaging the cross-power spectra over a given period of time;

pre-computing a table of time delays of arrival for each pair of sound sensors and each direction on the grid of the sphere; and

finding a loudest sound source in a direction of the grid of the sphere, finding the loudest sound source comprising searching a best direction on the grid of the sphere using a first algorithm and the pre-computed table by maximizing an output energy of the steered beamformer.

56. A sound source localizing method as defined in claim 55, comprising finding another sound source, using a second algorithm, after having removed the contribution of the located, loudest sound source.

57. A sound source localizing method as defined in claim 55, wherein localizing, in a single step, said at least one sound source further comprises:

defining a refined grid for the surrounding of a point where a sound source was found in order to find a direction of localization of the found sound source with improved accuracy.

58. A method for tracking a plurality of sound sources, comprising:

detecting sound from the sound sources through a set of spatially spaced apart sound sensors to produce corresponding sound signals; and

simultaneously tracking the plurality of sound sources, using particle filtering responsive to the sound signals from the sound sensors.

59. A sound source tracking method as defined in claim 58, wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises modeling each sound source using a number of particles having respective directions and weights.

60. A sound source tracking method as defined in claim 58, wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises:

computing a probability that a potential source is a real source.

61. A sound source tracking method as defined in claim 58, wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises:

computing a probability that a real source corresponds to a potential source detected by the sound source detector.

62. A sound source tracking method as defined in claim 59, wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises:

computing (a) at least one of a probability that a sound source is observed and a probability that a real sound source corresponds to a potential sound source, and (b) a probability density of observing a sound source at a given particle position; and

computing updated particle weights in response to said probability density and said at least one probability.

63. A sound source tracking method as defined in claim 58, wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises:

adding a new source when a probability that the new source is real is higher than a first threshold.

64. A sound source tracking method as defined in claim 63, wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises assuming that the added new source exists if a probability of existence of said new source reaches a second threshold.

65. A sound source tracking method as defined in claim 58, wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises:

removing a sound source when the latter source has not been observed for a certain period of time.

66. A sound source localizing and tracking method as defined in claim 59, wherein simultaneously tracking the plurality of sound sources, using particle filtering, comprises:

estimating a position of each source as a weighted average of the positions of its particles, said estimator being responsive to the calculated, updated particle weights.