Method and Apparatus for Speech Dereverberation Based On Probabilistic Models Of Source And Room Acoustics

• US 20090110207A1
• Filed: 05/01/2006
• Published: 04/30/2009
• Est. Priority Date: 05/01/2006
• Status: Active Grant

• Current Assignee: Nippon Telegraph and Telephone Corporation, Georgia Tech Research Corporation (University System of Georgia)
• Original Assignee: Nippon Telegraph and Telephone Corporation, Georgia Tech Research Corporation (University System of Georgia)
• Inventors: Juang, Biing-Hwang, Nakatani, Tomohiro
• Granted Patent: US 8,290,170 B2
• Time in Patent Office: Days
• US Class Current: 381/66
• CPC Class Codes: G10L 2021/02082 the noise being echo, reverberation of the speech
• CPC Class Codes: G10L 21/0208 Noise filtering
• CPC Class Codes: G10L 21/0232 Processing in the frequency domain

Abstract

Speech dereverberation is achieved by accepting an observed signal for initialization (1000) and performing likelihood maximization (2000) which includes Fourier Transforms (4000).

Claims

1. A speech dereverberation apparatus comprising:

a likelihood maximization unit that determines a source signal estimate that maximizes a likelihood function, the determination being made with reference to an observed signal, an initial source signal estimate, a first variance representing a source signal uncertainty, and a second variance representing an acoustic ambient uncertainty.

2. The speech dereverberation apparatus according to claim 1, wherein the likelihood function is defined based on a probability density function that is evaluated in accordance with an unknown parameter, a first random variable of missing data, and a second random variable of observed data, the unknown parameter being defined with reference to the source signal estimate, the first random variable of missing data representing an inverse filter of a room transfer function, and the second random variable of observed data being defined with reference to the observed signal and the initial source signal estimate.

3. The speech dereverberation apparatus according to claim 2, wherein the likelihood maximization unit determines the source signal estimate using an iterative optimization algorithm.

4. The speech dereverberation apparatus according to claim 3, wherein the iterative optimization algorithm is an expectation-maximization algorithm.

5. The speech dereverberation apparatus according to claim 1, wherein the likelihood maximization unit further comprises:

an inverse filter estimation unit that calculates an inverse filter estimate with reference to the observed signal, the second variance, and one of the initial source signal estimate and an updated source signal estimate;

a filtering unit that applies the inverse filter estimate to the observed signal, and generates a filtered signal;

a source signal estimation and convergence check unit that calculates the source signal estimate with reference to the initial source signal estimate, the first variance, the second variance, and the filtered signal, the source signal estimation and convergence check unit further determining whether or not a convergence of the source signal estimate is obtained, the source signal estimation and convergence check unit further outputting the source signal estimate as a dereverberated signal if the convergence of the source signal estimate is obtained; and

an update unit that updates the source signal estimate into the updated source signal estimate, the update unit further providing the updated source signal estimate to the inverse filter estimation unit if the convergence of the source signal estimate is not obtained, and the update unit further providing the initial source signal estimate to the inverse filter estimation unit in an initial update step.

6. The speech dereverberation apparatus according to claim 5, wherein the likelihood maximization unit further comprises:

a first long time Fourier transform unit that performs a first long time Fourier transformation of a waveform observed signal into a transformed observed signal, the first long time Fourier transform unit further providing the transformed observed signal as the observed signal to the inverse filter estimation unit and the filtering unit;

an LTFS-to-STFS transform unit that performs an LTFS-to-STFS transformation of the filtered signal into a transformed filtered signal, the LTFS-to-STFS transform unit further providing the transformed filtered signal as the filtered signal to the source signal estimation and convergence check unit;

an STFS-to-LTFS transform unit that performs an STFS-to-LTFS transformation of the source signal estimate into a transformed source signal estimate, the STFS-to-LTFS transform unit further providing the transformed source signal estimate as the source signal estimate to the update unit if the convergence of the source signal estimate is not obtained;

a second long time Fourier transform unit that performs a second long time Fourier transformation of a waveform initial source signal estimate into a first transformed initial source signal estimate, the second long time Fourier transform unit further providing the first transformed initial source signal estimate as the initial source signal estimate to the update unit; and

a short time Fourier transform unit that performs a short time Fourier transformation of the waveform initial source signal estimate into a second transformed initial source signal estimate, the short time Fourier transform unit further providing the second transformed initial source signal estimate as the initial source signal estimate to the source signal estimation and convergence check unit.

7. The speech dereverberation apparatus according to claim 1, further comprising:

an inverse short time Fourier transform unit that performs an inverse short time Fourier transformation of the source signal estimate into a waveform source signal estimate.

8. The speech dereverberation apparatus according to claim 1, further comprising:

an initialization unit that produces the initial source signal estimate, the first variance, and the second variance, based on the observed signal.

9. The speech dereverberation apparatus according to claim 8, wherein the initialization unit further comprises:

a fundamental frequency estimation unit that estimates a fundamental frequency and a voicing measure for each short time frame from a transformed signal that is given by a short time Fourier transformation of the observed signal; and

a source signal uncertainty determination unit that determines the first variance, based on the fundamental frequency and the voicing measure.

10. The speech dereverberation apparatus according to claim 1, further comprising:

an initialization unit that produces the initial source signal estimate, the first variance, and the second variance, based on the observed signal; and

a convergence check unit that receives the source signal estimate from the likelihood maximization unit, the convergence check unit determining whether or not a convergence of the source signal estimate is obtained, the convergence check unit further outputting the source signal estimate as a dereverberated signal if the convergence of the source signal estimate is obtained, and the convergence check unit furthermore providing the source signal estimate to the initialization unit to enable the initialization unit to produce the initial source signal estimate, the first variance, and the second variance based on the source signal estimate if the convergence of the source signal estimate is not obtained.

11. The speech dereverberation apparatus according to claim 10, wherein the initialization unit further comprises:

a second short time Fourier transform unit that performs a second short time Fourier transformation of the observed signal into a first transformed observed signal;

a first selecting unit that performs a first selecting operation to generate a first selected output and a second selecting operation to generate a second selected output, the first and second selecting operations being independent from each other, the first selecting operation being to select the first transformed observed signal as the first selected output when the first selecting unit receives an input of the first transformed observed signal but does not receive any input of the source signal estimate and to select one of the first transformed observed signal and the source signal estimate as the first selected output when the first selecting unit receives inputs of the first transformed observed signal and the source signal estimate, the second selecting operation being to select the first transformed observed signal as the second selected output when the first selecting unit receives the input of the first transformed observed signal but does not receive any input of the source signal estimate and to select one of the first transformed observed signal and the source signal estimate as the second selected output when the first selecting unit receives inputs of the first transformed observed signal and the source signal estimate,

a fundamental frequency estimation unit that receives the second selected output and estimates a fundamental frequency and a voicing measure for each short time frame from the second selected output; and

an adaptive harmonic filtering unit that receives the first selected output, the fundamental frequency and the voicing measure, the adaptive harmonic filtering unit enhancing a harmonic structure of the first selected output based on the fundamental frequency and the voicing measure to generate the initial source signal estimate.

12. The speech dereverberation apparatus according to claim 10, wherein the initialization unit further comprises:

a third short time Fourier transform unit that performs a third short time Fourier transformation of the observed signal into a second transformed observed signal;

a second selecting unit that performs a third selecting operation to generate a third selected output, the third selecting operation being to select the second transformed observed signal as the third selected output when the second selecting unit receives an input of the second transformed observed signal but does not receive any input of the source signal estimate and to select one of the second transformed observed signal and the source signal estimate as the third selected output when the second selecting unit receives inputs of the second transformed observed signal and the source signal estimate;

a fundamental frequency estimation unit that receives the third selected output and estimates a fundamental frequency and a voicing measure for each short time frame from the third selected output; and

a source signal uncertainty determination unit that determines the first variance based on the fundamental frequency and the voicing measure.

13. The speech dereverberation apparatus according to claim 10, further comprising:

an inverse short time Fourier transform unit that performs an inverse short time Fourier transformation of the source signal estimate into a waveform source signal estimate if the convergence of the source signal estimate is obtained.

14. A speech dereverberation apparatus comprising:

a likelihood maximization unit that determines an inverse filter estimate that maximizes a likelihood function, the determination being made with reference to an observed signal, an initial source signal estimate, a first variance representing a source signal uncertainty, and a second variance representing an acoustic ambient uncertainty.

15. The speech dereverberation apparatus according to claim 14, wherein the likelihood function is defined based on a probability density function that is evaluated in accordance with a first unknown parameter, a second unknown parameter, and a first random variable of observed data, the first unknown parameter being defined with reference to a source signal estimate, the second unknown parameter being defined with reference to an inverse filter of a room transfer function, the first random variable of observed data being defined with reference to the observed signal and the initial source signal estimate, the inverse filter estimate being an estimate of the inverse filter of the room transfer function.

16. The speech dereverberation apparatus according to claim 15, wherein the likelihood maximization unit determines the inverse filter estimate using an iterative optimization algorithm.

17. The speech dereverberation apparatus according to claim 14, further comprising:

an inverse filter application unit that applies the inverse filter estimate to the observed signal, and generates a source signal estimate.

18. The speech dereverberation apparatus according to claim 17, wherein the inverse filter application unit further comprises:

a first inverse long time Fourier transform unit that performs a first inverse long time Fourier transformation of the inverse filter estimate into a transformed inverse filter estimate; and

a convolution unit that receives the transformed inverse filter estimate and the observed signal, and convolves the observed signal with the transformed inverse filter estimate to generate the source signal estimate.

19. The speech dereverberation apparatus according to claim 17, wherein the inverse filter application unit further comprises:

a first long time Fourier transform unit that performs a first long time Fourier transformation of the observed signal into a transformed observed signal;

a first filtering unit that applies the inverse filter estimate to the transformed observed signal, and generates a filtered source signal estimate; and

a second inverse long time Fourier transform unit that performs a second inverse long time Fourier transformation of the filtered source signal estimate into the source signal estimate.

20. The speech dereverberation apparatus according to claim 14, wherein the likelihood maximization unit further comprises:

an inverse filter estimation unit that calculates an inverse filter estimate with reference to the observed signal, the second variance, and one of the initial source signal estimate and an updated source signal estimate;

a convergence check unit that determines whether or not a convergence of the inverse filter estimate is obtained, the convergence check unit further outputting the inverse filter estimate as a filter that is to dereverberate the observed signal if the convergence of the source signal estimate is obtained;

a filtering unit that receives the inverse filter estimate from the convergence check unit if the convergence of the source signal estimate is not obtained, the filtering unit further applying the inverse filter estimate to the observed signal and generates a filtered signal;

a source signal estimation unit that calculates the source signal estimate with reference to the initial source signal estimate, the first variance, the second variance, and the filtered signal; and

an update unit that updates the source signal estimate into the updated source signal estimate, the update unit further providing the initial source signal estimate to the inverse filter estimation unit in an initial update step, the update unit further providing the updated source signal estimate to the inverse filter estimation unit in update steps other than the initial update step.

21. The speech dereverberation apparatus according to claim 20, wherein the likelihood maximization unit further comprises:

a second long time Fourier transform unit that performs a second long time Fourier transformation of a waveform observed signal into a transformed observed signal, the second long time Fourier transform unit further providing the transformed observed signal as the observed signal to the inverse filter estimation unit and the filtering unit;

an LTFS-to-STFS transform unit that performs an LTFS-to-STFS transformation of the filtered signal into a transformed filtered signal, the LTFS-to-STFS transform unit further providing the transformed filtered signal as the filtered signal to the source signal estimation unit;

an STFS-to-LTFS transform unit that performs ah STFS-to-LTFS transformation of the source signal estimate info a transformed source signal estimate, the STFS-to-LTFS transform unit further providing the transformed source signal estimate as the source signal estimate to the update unit;

a third long time Fourier transform unit that performs a third long time Fourier transformation of a waveform initial source signal estimate into a first transformed initial source signal estimate, the third long time Fourier transform unit further providing the first transformed initial source signal estimate as the initial source signal estimate to the update unit; and

a short time Fourier transform unit that performs a short time Fourier transformation of the waveform initial source signal estimate into a second transformed initial source signal estimate, the short time Fourier transform unit further providing the second transformed initial source signal estimate as the initial source signal estimate to the source signal estimation unit.

22. The speech dereverberation apparatus according to claim 14, further comprising:

an initialization unit that produces the initial source signal estimate, the first variance, and the second variance, based on the observed signal.

23. The speech dereverberation apparatus according to claim 22, wherein the initialization unit further comprises:

a fundamental frequency estimation unit that estimates a fundamental frequency and a voicing measure for each short time frame from a transformed signal that is given by a short time Fourier transformation of the observed signal; and

a source signal uncertainty determination unit that determines the first variance, based on the fundamental frequency and the voicing measure.

24. A speech dereverberation method comprising:

determining a source signal estimate that maximizes a likelihood function, the determination being made with reference to an observed signal, an initial source signal estimate, a first variance representing a source signal uncertainty, and a second variance representing an acoustic ambient uncertainty.

25. The speech dereverberation method according to claim 24, wherein the likelihood function is defined based on a probability density function that is evaluated in accordance with an unknown parameter, a first random variable of missing data, and a second random variable of observed data, the unknown parameter being defined with reference to the source signal estimate, the first random variable of missing data representing an inverse filter of a room transfer function, the second random variable of observed data being defined with reference to the observed signal and the initial source signal estimate.

26. The speech dereverberation method according to claim 25, wherein the source signal estimate is determined using an iterative optimization algorithm.

27. The speech dereverberation method according to claim 26, wherein the iterative optimization algorithm is an expectation-maximization algorithm.

28. The speech dereverberation method according to claim 24, wherein determining the source signal estimate further comprises:

calculating an inverse filter estimate with reference to the observed signal, the second variance, and one of the initial source signal estimate and an updated source signal estimate;

applying the inverse filter estimate to the observed signal to generate a filtered signal;

calculating the source signal estimate with reference to the initial source signal estimate, the first variance, the second variance, and the filtered signal;

determining whether or not a convergence of the source signal estimate is obtained;

outputting the source signal estimate as a dereverberated signal if the convergence of the source signal estimate is obtained; and

updating the source signal estimate info the updated source signal estimate if the convergence of the source signal estimate is not obtained.

29. The speech dereverberation method according to claim 28, wherein determining the source signal estimate further comprises:

performing a first long time Fourier transformation of a waveform observed signal into a transformed observed signal;

performing an LTFS-to-STFS transformation of the filtered signal into a transformed filtered signal;

performing an STFS-to-LTFS transformation of the source signal estimate into a transformed source signal estimate if the convergence of the source signal estimate is not obtained;

performing a second long time Fourier transformation of a waveform initial source signal estimate into a first transformed initial source signal estimate; and

performing a short time Fourier transformation of the waveform initial source signal estimate into a second transformed initial source signal estimate.

30. The speech dereverberation method according to claim 24, further comprising:

performing an inverse short time Fourier transformation of the source signal estimate into a waveform source signal estimate.

31. The speech dereverberation method according to claim 24, further comprising:

producing the initial source signal estimate, the first variance, and the second variance, based on the observed signal.

32. The speech dereverberation method according to claim 31, wherein producing the initial source signal estimate, the first variance, and the second variance further comprises:

estimating a fundamental frequency and a voicing measure for each short time frame from a transformed signal that is given by a short time Fourier transformation of the observed signal; and

determining the first variance, based on the fundamental frequency and the voicing measure.

33. The speech dereverberation method according to claim 24, further comprising:

producing the initial source signal estimate, the first variance, and the second variance, based on the observed signal;

determining whether or not a convergence of the source signal estimate is obtained;

outputting the source signal estimate as a dereverberated signal if the convergence of the source signal estimate is obtained; and

returning to producing the initial source signal estimate, the first variance, and the second variance if the convergence of the source signal estimate is not obtained.

34. The speech dereverberation method according to claim 33, wherein producing the initial source signal estimate, the first variance, and the second variance further comprises:

performing a second short time Fourier transformation of the observed signal into a first transformed observed signal;

performing a first selecting operation to generate a first selected output, the first selecting operation being to select the first transformed observed signal as the first selected output when receiving an input of the first transformed observed signal without receiving any input of the source signal estimate, the first selecting operation being to select one of the first transformed observed signal and the source signal estimate as the first selected output when receiving inputs of the first transformed observed signal and the source signal estimate;

performing a second selecting operation to generate a second selected output, the second selecting operation being to select the first transformed observed signal as the second selected output when receiving the input of the first transformed observed signal without receiving any input of the source signal estimate, the second selecting operation being to select one of the first transformed observed signal and the source signal estimate as the second selected output when receiving inputs of the first transformed observed signal and the source signal estimate;

estimating a fundamental frequency and a voicing measure for each short time frame from the second selected output; and

enhancing a harmonic structure of the first selected output based on the fundamental frequency and the voicing measure to generate the initial source signal estimate.

35. The speech dereverberation method according to claim 33, wherein producing the initial source signal estimate, the first variance, and the second variance further comprises:

performing a third short time Fourier transformation of the observed signal into a second transformed observed signal;

performing a third selecting operation to generate a third selected output, the third selecting operation being to select the second transformed observed signal as the third selected output when receiving an input of the second transformed observed signal without receiving any input of the source signal estimate, the third selecting operation being to select one of the second transformed observed signal and the source signal estimate as the third selected output when receiving inputs of the second transformed observed signal and the source signal estimate;

estimating a fundamental frequency and a voicing measure for each short time frame from the third selected output; and

determining the first variance based on the fundamental frequency and the voicing measure.

36. The speech dereverberation method according to claim 33, further comprising:

perforating an inverse short time Fourier transformation of the source signal estimate into a waveform source signal estimate if the convergence of the source signal estimate is obtained.

37. A speech dereverberation method comprising:

determining an inverse filter estimate that maximizes a likelihood function, the determination being made with reference to an observed signal, an initial source signal estimate, a first variance representing a source signal uncertainty, and a second variance representing an acoustic ambient uncertainty.

38. The speech dereverberation method according to claim 37, wherein the likelihood function is defined based on a probability density function that is evaluated in accordance with a first unknown parameter, a second unknown parameter, and a first random variable of observed data, the first unknown parameter being defined with reference to a source signal estimate, the second unknown parameter being defined with reference to an inverse filter of a room transfer function, and the first random variable of observed data being defined with reference to the observed signal and the initial source signal estimate, the inverse filter estimate being an estimate of the inverse filter of the room transfer function.

39. The speech dereverberation method according to claim 38, wherein the inverse filter estimate is determined using an iterative optimization algorithm.

40. The speech dereverberation method according to claim 37, further comprising:

applying the inverse filter estimate to the observed signal to generate a source signal estimate.

41. The speech dereverberation method according to claim 40, wherein applying the inverse filter estimate to the observed signal further comprises:

performing a first inverse long time Fourier transformation of the inverse filter estimate into a transformed inverse filter estimate; and

convolving the observed signal with the transformed inverse filter estimate to generate the source signal estimate.

42. The speech dereverberation method according to claim 40, wherein applying the inverse filter estimate to the observed signal further comprises:

performing a first long time Fourier transformation of the observed signal into a transformed observed signal;

applying the inverse filter estimate to the transformed observed signal to generate a filtered source signal estimate; and

performing a second inverse long time Fourier transformation of the filtered source signal estimate into the source signal estimate.

43. The speech dereverberation method according to claim 37, wherein determining the inverse filter estimate further comprises:

calculating an inverse filter estimate with reference to the observed signal, the second variance, and one of the initial source signal estimate and an updated source signal estimate;

determining whether or not a convergence of the inverse filter estimate is obtained;

outputting the inverse filter estimate as a filter that is to dereverberate the observed signal if the convergence of the source signal estimate is obtained;

applying the inverse filter estimate to the observed signal to generate a filtered signal if the convergence of the source signal estimate is not obtained;

calculating the source signal estimate with reference to the initial source signal estimate, the first variance, the second variance, and the filtered signal; and

updating the source signal estimate into the updated source signal estimate.

44. The speech dereverberation method according to claim 43, wherein determining the inverse filter estimate further comprises:

performing a second long time Fourier transformation of a waveform observed signal into a transformed observed signal;

performing an LTFS-to-STFS transformation of the filtered signal into a transformed filtered signal;

performing an STFS-to-LTFS transformation of the source signal estimate into a transformed source signal estimate;

performing a third long time Fourier transformation of a waveform initial source signal estimate into a first transformed initial source signal estimate; and

performing a short time Fourier transformation of the waveform initial source signal estimate info a second transformed initial source signal estimate.

45. The speech dereverberation method according to claim 37, further comprising:

producing the initial source signal estimate, the first variance, and the second variance, based on the observed signal.

46. The speech dereverberation method according to claim 45, wherein producing the initial source signal estimate, the first variance, and the second variance further comprises:

estimating a fundamental frequency and a voicing measure for each short time frame from a transformed signal that is given by a short time Fourier transformation of the observed signal; and

determining the first variance, based on the fundamental frequency and the voicing measure.

47. A program to be executed by a computer to perform a speech dereverberation method comprising:

determining a source signal estimate that maximizes a likelihood function, the determination being made with reference to an observed signal, an initial source signal estimate, a first variance representing a source signal uncertainty, and a second variance representing an acoustic ambient uncertainty.

48. A program to be executed by a computer to perform a speech dereverberation method comprising:

determining an inverse filter estimate that maximizes a likelihood function, the determination being made with reference to an observed signal, an initial source signal estimate, a first variance representing a source signal uncertainty, and a second variance representing an acoustic ambient uncertainty.

49. A storage medium that stores a program to be executed by a computer to perform a speech dereverberation method comprising:

determining a source signal estimate that maximizes a likelihood function, the determination being made with reference to an observed signal, an initial source signal estimate, a first variance representing a source signal uncertainty, and a second variance representing an acoustic ambient uncertainty.

50. A storage medium that stores a program to be executed by a computer to perform a speech dereverberation method comprising:

determining an inverse filter estimate that maximizes a likelihood function, the determination being made with reference to an observed signal, an initial source signal estimate, a first variance representing a source signal uncertainty, and a second variance representing an acoustic ambient uncertainty.