Processing of acoustic waveforms

• US RE36,478 E
• Filed: 04/12/1996
• Issued: 12/28/1999
• Est. Priority Date: 03/18/1985
• Status: Expired due to Term

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Quatieri, Thomas F. Jr., McAulay, Robert J.
• Primary Examiner(s): Knepper, David D.
• Application Number:
  US08/631,222
• Time in Patent Office: 1,355 Days
• Field of Search: 395/2.14, 395/2.18, 395/2.29, 395/2.12, 395/2.13, 395/2.33, 395/2.34, 395/2.7, 395/2.74, 395/2.77, 395/2.78, 704/203-209, 704/220, 704/224, 704/225, 704/261, 704/265, 704/268, 704/269
• US Class Current: 704/206
• CPC Class Codes: G10L 19/02 using spectral analysis, e.g. transform vocoders or subband vocoders

Abstract

A sinusoidal model for acoustic waveforms is applied to develop a new analysis/synthesis technique which characterizes a waveform by the amplitudes, frequencies, and phases of component sine waves. These parameters are estimated from a short-time Fourier transform. Rapid changes in the highly-resolved spectral components are tracked using the concept of "birth" and "death" of the underlying sine waves. The component values are interpolated from one frame to the next to yield a representation that is applied to a sine wave generator. The resulting synthetic waveform preserves the general waveform shape and is perceptually indistinguishable from the original. Furthermore, in the presence of noise the perceptual characteristics of the waveform as well as the noise are maintained. The method and devices are particularly useful in speech coding, time-scale modification, frequency scale modification and pitch modification.

Claims

1. A method of processing an acoustic waveform, the method comprising:

sampling the waveform to obtain a series of discrete samples and constructing therefrom a series of frames, each frame spanning a plurality of samples;

analyzing each frame of samples to extract a set of variable frequency components having individual amplitudes;

matching said variable components from one frame to a next frame such that a component in one frame is matched with a component in a successive frame that has a similar value regardless of shifts in frequency and spectral energy; and

interpolating the matched values of the components from the one frame to the next frame to obtain a parametric representation of the waveform whereby a synthetic waveform can be constructed by generating a set of sine waves corresponding to the interpolated values of the parametric representation.

2. The method of claim 1 wherein the step of sampling further includes determining a pitch period for said waveform and varying the length of the frame in accordance with the pitch period, the length being at least twice the pitch period of the waveform.

3. The method of claim 2 wherein the step of sampling further includes sampling the waveform according to a pitch-adaptive Hamming window.

4. The method of claim 1 wherein the step of analyzing further includes analyzing each frame by Fourier analysis.

5. The method of claim 1 wherein the step of analyzing further includes selecting a harmonic series to approximate the frequency components.

6. The method of claim 5 wherein the step of selecting a harmonic series further includes determining a pitch period for the waveform and varying the number of frequency components in the harmonic series in accordance with the pitch period of the waveform.

7. The method of claim 1 wherein the step of matching includes matching a frequency component of nonzero amplitude from the one frame with a frequency component of nonzero amplitude in the next frame having a similar frequency value.

8. The method of claim 7 wherein said matching further provides for the birth of new frequency components and the death of old frequency components.

9. The method of claim 1 wherein the step of interpolating values further includes defining a series of instantaneous frequency values by interpolating matched frequency components from the one frame to the next frame and then integrating the series of instantaneous frequency values to obtain a series of interpolated phase values.

10. The method of claim 1 wherein the step of interpolating further includes deriving phase values from frequency and phase measurements taken at each frame and then interpolating the phase measurements.

11. The method of claim 1 wherein the step of interpolating is achieved by performing an overlap and add function.

12. The method of claim 1 wherein the method further includes coding the frequency components for digital transmission.

13. The method of claim 12 wherein the frequency components are limited to a predetermined number defined by a plurality of harmonic frequency bins.

14. The method of claim 13 wherein the amplitude of only one of said components is coded for gain and the amplitudes of the others are coded relative to the neighboring component at the next lowest frequency.

15. The method of claim 12 wherein the phases are coded by applying pulse code modulation techniques to a predicted phase residual.

16. The method of claim 12 wherein high frequency regeneration is applied.

17. The method of claim 1 wherein the method further comprises constructing a synthetic waveform by generating a series of constituent sine waves corresponding in frequency and amplitude to the extracted components.

18. The method of claim 17 wherein the time-scale of said reconstructed waveform is varied by changing the rate at which said series of constituent sine waves are interpolated.

19. The method of claim 18 wherein the time-scale is continuously variable over a defined range.

20. The method of claim 17 wherein the pitch of the synthetic waveform is varied by adjusting the frequency of each frequency component while maintaining the overall spectral envelope.

21. The method of claim 1 wherein the method further comprises constructing a synthetic waveform by generating a series of constituent sine waves corresponding in frequency, amplitude, and phase to the extracted components.

22. The method of claim 21 wherein the time-scale of said reconstructed waveform is varied by changing the rate at which said series of constituent sine waves are interpolated.

23. The method of claim 22 wherein the time-scale is continuously variable over a defined range.

24. The device of claim 22 wherein the device further comprises means for constructing a synthetic waveform by generating a series of constituent sine waves corresponding in frequency and amplitude to the extracted components.

25. The device of claim 24 wherein the device further includes means for varying the time-scale of said reconstructed waveform by changing the rate at which said series of constituent sine waves are interpolated.

26. The device of claim 25 wherein the means for varying the time-scale is continuously variable over a defined range.

27. The device of claim 24 wherein the constituent sine waves are further defined by system contributions and excitation contributions and wherein the means for varying the time-scale of said reconstructed waveform further includes means for changing the rate at which parameters defining the system contributions of the sine waves are interpolated.

28. The device of claim 27 wherein the device further includes a scaling means for scaling the frequency components.

29. The device of claim 27 wherein the device further includes a scaling-means for scaling the excitation-contributed frequency components.

30. The method of claim 21 wherein the constituent sine waves are further defined by system contributions and excitation contributions and wherein the time-scale of said reconstructed waveform is varied by changing the rate at which parameters defining the system contributions of the sine waves are interpolated.

31. The method of claim 30 wherein the pitch of the synthetic waveform is altered by adjusting the frequencies of the excitation-contributed frequency components while maintaining the overall spectral envelope.

65. The method of claim 1 wherein the step of matching provides for the birth of new frequency components and the death of old frequency components.

66. The method of claim 65 wherein the birth of a new frequency component establishes a component of zero magnitude in the one frame at the same frequency as a component of the successive frame and the death of an old frequency component establishes a component of zero magnitude in the successive frame at the same frequency as a component of the one frame.

67. The method of claim 66 wherein the acoustic waveform is speech and, in voiced speech, the frequency components approximate harmonics to pitch frequency.

68. The method of claim 65 wherein the acoustic waveform is speech and, in voiced speech, the frequency components approximate harmonics to pitch frequency.

69. The method of claim 1 wherein the step of analyzing comprises identifying the frequency components by peak picking.

70. The method of claim 69 wherein the step of matching provides for the birth of new frequency components and the death of old frequency components.

71. The method of claim 70 wherein the birth of a new frequency component establishes a component of zero magnitude in the one frame at the same frequency as a component of the successive frame and the death of an old frequency component establishes a component of zero magnitude in the successive frame at the same frequency as a component of the one frame.

72. The method of claim 71 wherein the acoustic waveform is speech and, in voiced speech, the frequency components approximate harmonics to pitch frequency.

73. The method of claim 1 wherein the acoustic waveform is speech and, in voiced speech, the frequency components approximate harmonics to pitch frequency.

93. The method of claim 1 wherein the step of interpolating comprises interpolating utilizes a phase interpolation function that is smooth in unwrapping the phase to a phase which, modulo 2π

, achieves the phase of the extracted variable frequency components at frame boundaries.

32. A device for processing an acoustic waveform, the device comprising:

sampling means for sampling the waveform to obtain a series of discrete samples and constructing therefrom a series of frames, each frame spanning a plurality of samples;

analyzing means for analyzing each frame of samples to extract a set of variable frequency components having individual amplitudes;

matching means for matching said variable components from one frame to a next frame such that a component in one frame is matched with a component in a successive frame that has a similar value regardless of shifts in frequency and spectral energy; and

interpolating means for interpolating the matched values of the components from the one frame to the next frame to obtain a parametric representation of the waveform whereby a synthetic waveform can be constructed by generating a set of sine waves corresponding to the interpolated values of the parametric representation.

33. The device of claim 32 wherein the sampling means further includes means for constructing a frame having variable length, which varies in accordance with the pitch period, the length being at least twice the pitch period of the waveform.

34. The device of claim 32 wherein the sampling means further includes means for sampling according to a Hamming window.

35. The device of claim 32 wherein analyzing means further includes means for analyzing each frame by Fourier analysis.

36. The device of claim 32 wherein the analyzing means further includes means for selecting a harmonic series to approximate the frequency components.

37. The device of claim 36 wherein the number of frequency components in the harmonic series varies according to the pitch period of the waveform.

38. The device of claim 32 wherein the matching means includes means for matching a frequency component of nonzero amplitude from the one frame with a frequency component of nonzero amplitude in the next frame having a similar frequency value.

39. The device of claim 38 wherein said matching means further provides for the birth of new frequency components and the death of old frequency components.

40. The device of claim 38 wherein the frequency components are limited to a predetermined number defined by a plurality of harmonic frequency bins.

41. The device of claim 40 wherein the amplitude of only one of said components is coded for gain and the amplitudes of the others are coded relative to the neighboring component at the next lowest frequency.

42. The device of claim 32 wherein the interpolating means further includes means defining a series of instantaneous frequency values by interpolating matched frequency components from the one frame to the next frame and means for integrating the series of instantaneous frequency values to obtain a series of interpolated phase values.

43. The device of claim 32 wherein the interpolating means further includes means for deriving phase values from the frequency and phase measurements taken at each frame and then interpolating the phase measurements.

44. The device of claim 32 wherein the interpolating means further includes means for performing an overlap and add function.

45. The device of claim 32 wherein the device further includes coding means for coding the frequency components for digital transmission.

46. The device of claim 45 wherein the coding means further comprises means for applying pulse code modulation techniques to a predicted phase residual.

47. The device of claim 45 wherein the coding means further comprises means for generating high frequency components.

48. The device of claim 32 wherein the device further comprises means for constructing a synthetic waveform by generating a series of constituent sine waves corresponding in frequency, amplitude, and phase to the extracted components.

49. The device of claim 48 wherein the device further includes means for varying the time-scale of said reconstructed waveform by changing the rate at which said series of constituent sine waves are interpolated.

50. The device of claim 49 wherein the means for varying the time-scale is continuously variable over a defined range.

74. The device of claim 32 wherein the matching means provides for the birth of new frequency components and the death of old frequency components.

75. The device of claim 74 wherein the birth of a new frequency component establishes a component of zero magnitude in the one frame at the same frequency as a component of the successive frame and the death of an old frequency component establishes a component of zero magnitude in the successive frame at the same frequency as a component of the one frame.

76. The device of claim 75 wherein the acoustic waveform is speech and, in voiced speech, the frequency components approximate harmonics to pitch frequency.

77. The device of claim 74 wherein the acoustic waveform is speech and, in voiced speech, the frequency components approximate harmonics to pitch frequency.

78. The device of claim 32 wherein the analyzing means identifies the frequency components by peak picking.

79. The device of claim 78 wherein the matching means provides for the birth of new frequency components and the death of old frequency components.

80. The device of claim 79 wherein the birth of a new frequency component establishes a component of zero magnitude in the one frame at the same frequency as a component of the successive frame and the death of an old frequency component establishes a component of zero magnitude in the successive frame at the same frequency as a component of the one frame.

81. The device of claim 80 wherein the acoustic waveform is speech and, in voiced speech, the frequency components approximate harmonics to pitch frequency.

82. The device of claim 32 wherein the acoustic waveform is speech and, in voiced speech, the frequency components approximate harmonics to pitch frequency.

94. The device of claim 32 wherein the interpolating means utilizes a phase interpolation function that is smooth in unwrapping the phase to a phase which, module 2π

, achieves the phase of the extracted variable frequency components at frame boundaries.

51. A coded speech transmission system comprising:

sampling means for sampling a speech waveform to obtain a series of discrete samples and for constructing therefrom a series of frames, each frame spanning a plurality of samples;

analyzing means for analyzing each frame of samples by Fourier analysis to extract a set of variable frequency components having individual amplitude values;

coding means for coding the component values;

decoding means for decoding the coded values after transmission and for reconstituting the variable components;

matching means for matching the reconstituted, variable components from one frame to a next frame such that a component in one frame is matched with a component in a successive frame that has a similar value regardless of shift in frequency and spectral energy; and

interpolation means for interpolating the values of the frequency components from the one frame to the next frame to obtain a representation of the waveform whereby synthetic speech can be constructed by generating a set of sine waves corresponding to the interpolated values of the parametric representation.

52. The device of claim 51 wherein the coding means further includes means for selecting a harmonic series of bins to approximate the frequency components and the number of bins varies according to the pitch of the waveform.

53. The device of claim 51 wherein the amplitude of only one of said components is coded for gain and the amplitudes of the other components are coded relative to the neighboring component at the next lowest frequency.

54. The device of claim 51 wherein the amplitudes of the components are coded by linear prediction techniques.

55. The device of claim 51 wherein the amplitudes of the components are coded by adaptive delta modulation techniques.

56. The device of claim 51 wherein the analyzing means further comprises means for measuring phase values for each frequency component.

57. The device of claim 56 wherein the coding means further includes means for coding the phase values by applying pulse code modulations to a predicted phase residual.

83. The system of claim 51 wherein the matching means provides for the birth of new frequency components and the death of old frequency components.

84. The system of claim 83 wherein the birth of a new frequency component establishes a component of zero magnitude in the one frame at the same frequency as a component of the successive frame and the death of an old frequency component establishes a component of zero magnitude in the successive frame at the same frequency as a component of the one frame.

85. The system of claim 51 wherein the analyzing means identifies the frequency components by peak picking.

86. The system of claim 85 wherein the matching means provides for the birth of new frequency components and the death of old frequency components.

87. The system of claim 86 wherein the birth of a new frequency component establishes a component of zero magnitude in the one frame at the same frequency as a component of the successive frame and the death of an old frequency component establishes a component of zero magnitude in the successive frame at the same frequency as a component of the one frame.

95. The system of claim 51 wherein the interpolating means utilizes a phase interpolation function that is smooth in unwrapping the phase to a phase which, module 2π

, achieves the phase of the extracted variable frequency components at frame boundaries.

58. A device for altering the time-scale of an audible waveform, the device comprising:

sampling means for sampling the waveform to obtain a series of discrete samples and for constructing therefrom a series of frames, each frame spanning a plurality of samples;

analyzing means for analyzing each frame of samples to extract a set of variable frequency components having individual amplitudes;

matching means for matching said variable components from one frame to a next frame such that a component in one frame is matched with a component in a successive frame that has a similar value regardless of shifts in frequency and spectral energy;

interpolating means for interpolating the amplitude and frequency values of the components from the one frame to the next frame to obtain a representation of the waveform whereby a synthetic waveform can be constructed by generating a set of sine waves corresponding to the interpolated representation;

interpolation rate adjusting means for altering the rate of interpolation; and

synthesizing means for constructing a time-scaled synthetic waveform by generating a series of constituent sine waves corresponding in frequency and amplitude to the extracted components, the sine waves being generated at said alterable interpolation rate.

59. The device of claim 58 wherein the interpolation rate adjusting means is continuously variable over a defined range.

60. The device of claim 58 wherein the analyzing means further comprises means for measuring phase values for each frequency component.

61. The device of claim 60 wherein the component phase values are interpolated by cubic interpolation.

62. The device of claim 60 wherein the interpolation rate adjusting means is continuously variable over a defined range and further includes means for adjusting the rate of phase value interpolations.

63. The device of claim 60 wherein the device further comprises means for separating the measured frequency components into system contributions and excitation contributions and wherein the interpolation rate adjusting means varies the time-scale of the synthetic waveform by altering the rate at which values defining the system contributions are interpolated.

64. The device of claim 63 wherein the interpolation rate adjusting means alters the rate at which the system amplitudes and phases and the excitation amplitudes and frequencies are interpolated.

88. The device of claim 58 wherein the matching means provides for the birth of new frequency components and the death of old frequency components.

89. The device of claim 88 wherein the birth of a new frequency component establishes a component of zero magnitude in the one frame at the same frequency as a component of the successive frame and the death of an old frequency component establishes a component of zero magnitude in the successive frame at the same frequency as a component of the one frame.

90. The device of claim 58 wherein the analyzing means identifies the frequency components by peak picking.

91. The device of claim 90 wherein the matching means provides for the birth of new frequency components and the death of old frequency components.

92. The device of claim 91 wherein the birth of a new frequency component establishes a component of zero magnitude in the one frame at the same frequency as a component of the successive frame and the death of an old frequency component establishes a component of zero magnitude in the successive frame at the same frequency as a component of the one frame.

96. The device of claim 58 wherein the interpolating means utilizes a phase interpolation function that is smooth in unwrapping the phase to a phase which, module 2π

, achieves the phase of the extracted variable frequency components at frame boundaries.