System and methodology for prosody modification

• US 6,377,917 B1
• Filed: 11/04/1999
• Issued: 04/23/2002
• Est. Priority Date: 01/27/1997
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Gimenez de los Galanes, Francisco M., Talkin, David Thieme
• Primary Examiner(s): Dorvil, Richemond
• Assistant Examiner(s): MCFADDEN, SUSAN IRIS
• Application Number:
  US09/355,386
• Time in Patent Office: 901 Days
• Field of Search: 704/220, 704/267, 704/268, 704/200, 704/266, 704/258
• US Class Current: 704/220
• CPC Class Codes: G10L 13/04 Details of speech synthesis systems, e.g. synthesiser structure or memory management
• CPC Class Codes: G10L 13/10 Prosody rules derived from text; Stress or intonation
• CPC Class Codes: G10L 2021/0135 Voice conversion or morphing
• CPC Class Codes: G10L 21/003 Changing voice quality, e.g. pitch or formants
• CPC Class Codes: G10L 21/04 Time compression or expansion

Abstract

A prosody modification system and methodology calculates synchronization marks in an original, quasi-periodic signal to a finer precision than the sampling rate of the original signal. Synthetic synchronization marks are generated according to the desired prosody modification also to a finer precision than the sampling rate of the original signal. Waveforms are extracted from the original signal and are fine-shifted to the exact location on the synthetic time axis by a resampling technique. The fine-shifted waveforms are windowed by an asymmetric filtering window, overlapped, and summed together to produce a synthetic signal.

Claims

1. A method of performing a prosody modification on a quasi-periodic signal, sampled at a sampling interval, to produce a modified signal, said method comprising the machine-implemented steps of:

determining a series of original synchronization marks in said quasi-periodic signal;

determining a series of synthetic synchronization marks based on said original synchronization marks and said prosodic modification;

extracting waveforms from said quasi-periodic signal around one of said original synchronization marks;

shifting said waveforms to one of said synthetic synchronization marks corresponding to said one of said original synchronization marks to produce shifted waveforms, wherein a difference of said one of said original synchronization marks and said one of said synthetic synchronization marks is a non-integral multiple of said sampling interval; and

generating said modified signal based on said shifted waveforms.

2. A method as in claim 1, wherein the step of determining a series of original synchronization marks in said quasi-periodic signal includes the step of determining at least one of said original synchronization marks at a resolution finer than the sampling interval.

3. A method as in claim 2, wherein the step of determining at least one of said original synchronization marks at a resolution finer than the sampling interval includes the step of sampling the quasi-periodic signal at a shorter sampling interval with respect to said sampling interval.

4. A method as in claim 2, wherein the step of determining at least one of said original synchronization marks at a resolution finer than the sampling interval includes fitting a mathematical curve to find a peak in said quasi-periodic signal.

5. A method as in claim 3, wherein said shorter sampling interval is at most one-third of said sampling interval.

6. A method as in claim 1, wherein the step of determining a series of original synchronization marks in said quasi-periodic signal includes the step of determining epochs in said quasi-periodic signal.

7. A method as in claim 1, wherein the step of determining a series of synthetic synchronization marks includes the step of determining at least one of said synthetic synchronization marks at a resolution finer than the sampling interval.

8. A method as in claim 7, wherein the step of determining at least one of said synthetic synchronization marks at a resolution finer than the sampling interval includes the step of determining said at least one of said synthetic synchronization marks by a floating point number having a mantissa of at least twenty-four bits.

9. A method as in claim 1, wherein the step of shifting said waveforms to one of said synthetic synchronization marks corresponding to said one of said original synchronization marks includes the step of resampling said waveforms to adjust said waveforms to said one of said synthetic synchronization marks.

10. A method as in claim 9, wherein the step of shifting said waveforms to one of said synthetic synchronization marks corresponding to said one of said original synchronization marks further includes the step of shifting said waveforms to the nearest previous sampling interval of said one of said synthetic synchronization marks, before said step of resampling is performed.

11. A method as in claim 1, wherein the step of generating said modified signal based on said shifted waveforms includes the steps of:

applying an asymmetric filtering window to said shifted waveforms; and

summing the windowed, shifted waveform to generate said modified signal.

12. A method as in claim 11, wherein:

said asymmetric filtering window has a first section and a second section in juxtaposition with each other;

said first section has an amplitude progressively increasing from zero to a non-zero value along a first width;

said second section has an amplitude progressively decreasing from said non-zero value to zero along a second width; and

said first width is different in size from said second width.

13. A method as in claim 12, wherein:

said first width is the lesser of the interval between said one of said original synchronization marks and a preceding original synchronization mark and the interval between said one of said synthetic synchronization marks and a preceding synthetic synchronization mark; and

said second width is the lesser of the interval between said one of said original synchronization marks and a subsequent original synchronization mark and the interval between said one of said synthetic synchronization marks and a subsequent synthetic synchronization mark.

14. A method as in claim 13, wherein:

said first section is the first half of a Hanning window; and

said second section is the second half of a Hanning window.

15. A method of synthesizing a quasi-periodic signal from an original signal, said method comprising the steps of:

determining a series of original synchronization marks in said original signal;

determining a series of synthetic synchronization marks based on said original synchronization marks and on prosody information;

extracting a waveform from around each of said original synchronization marks by applying a filtering window and time-shifting each waveform according to a respective one of said original synchronization marks and a respective one of said synthetic synchronization marks corresponding to said respective one of said original synchronization marks, wherein each filtering window has a first half-width on one side of a respective original synchronization mark and a second half-width on another side of the respective original synchronization mark, and said first half-width is the lesser of the interval between said respective one of said original synchronization marks and a preceding original synchronization mark and the interval between said respective one of said synthetic synchronization marks and a preceding synthetic synchronization mark; and

summing the extracted waveforms to synthesize said quasi-periodic signal.

16. A method as in claim 15, wherein said step of windowing is performed before said step of time-shifting.

17. A method as in claim 15, wherein:

said filtering window has a first section and a second section in juxtaposition with each other;

said first section has an amplitude progressively increasing from zero to a non-zero value along said first half-width; and

said second section has amplitude progressively decreasing from said non-zero value to zero along said second half-width.

18. A method as in claim 17, wherein:

said second half-width is the lesser of the interval between said one of said original synchronization marks and a subsequent original synchronization mark and the interval between said one of said synthetic synchronization marks and a subsequent synthetic synchronization mark.

19. A method as in claim 18, wherein:

said first section is the first half of a Hanning window; and

said second section is the second half of a Hanning window.

20. A method as in claim 15, wherein said step of windowing is performed after said step of time-shifting.

21. A method as in claim 15, wherein a difference of said one of said original synchronization marks and said one of said synthetic synchronization marks is a non-integral multiple of said sampling interval.

22. A method as in claim 21, wherein the step of determining a series of original synchronization marks in said quasi-periodic signal includes the step of determining at least one of said original synchronization marks at a resolution finer than the sampling interval.

23. A method as in claim 22, wherein the step of determining at least one of said original synchronization marks at a resolution finer than the sampling interval includes the step of sampling the quasi-periodic signal at a shorter sampling interval with respect to said sampling interval.

24. A method as in claim 23, wherein said shorter sampling interval is at most one-third of said sampling interval.

25. A method as in claim 21, wherein the step of determining a series of original synchronization marks in said quasi-periodic signal includes the step of determining epochs in said quasi-periodic signal.

26. A method as in claim 21, wherein the step of determining a series of synthetic synchronization marks includes the step of determining at least one of said synthetic synchronization marks at a resolution finer than the sampling interval.

27. A method as in claim 26, wherein the step of determining at least one of said synthetic synchronization marks at a resolution finer than the sampling interval includes the step of determining said at least one of said synthetic synchronization marks by a floating point number having a mantissa of at least twenty-four bits.

28. A method as in claim 21, wherein the step of shifting said waveforms to one of said synthetic synchronization marks corresponding to said one of said original synchronization marks includes the step of resampling said waveforms to adjust said waveforms to said one of said synthetic synchronization marks.

29. A method as in claim 28, wherein step of shifting said waveforms to one of said synthetic synchronization marks corresponding to said one of said original synchronization marks further includes the step of shifting said waveforms to the nearest previous sampling interval of said one of said synthetic synchronization marks, before said step of resampling is performed.

30. A computer-readable medium bearing instructions for performing a prosody modification on a quasi-periodic signal, sampled at a sampling interval, to produce a modified signal, said instructions arranged, when executed, to cause one or more processors to perform the steps of:

determining a series of original synchronization marks in said quasi-periodic signal;

determining a series of synthetic synchronization marks based on said original synchronization marks and said prosodic modification;

extracting waveforms from said quasi-periodic signal around one of said original synchronization marks;

shifting said waveforms to one of said synthetic synchronization marks corresponding to said one of said original synchronization marks,

wherein a difference of said one of said original synchronization marks and said one of said synthetic synchronization marks is a non-integral multiple of said sampling interval; and

generating said modified signal based on said shifted waveforms.

31. A computer-readable medium as in claim 30, wherein the step of determining a series of original synchronization marks in said quasi-periodic signal includes the step of determining at least one of said original synchronization marks at a resolution finer than the sampling interval.

32. A computer-readable medium as in claim 31, wherein the step of determining at least one of said original synchronization marks at a resolution finer than the sampling interval includes the step of sampling the quasi-periodic signal at a shorter sampling interval with respect to said sampling interval.

33. A computer-readable medium as in claim 32, wherein said shorter sampling interval is at most one-third of said sampling interval.

34. A method as in claim 31, wherein the step of determining at least one of said original synchronization marks at a resolution finer than the sampling interval includes fitting a mathematical curve to find a peak in said quasi-periodic signal.

35. A computer-readable medium as in claim 30, wherein the step of determining a series of original synchronization marks in said quasi-periodic signal includes the step of determining epochs in said quasi-periodic signal.

36. A computer-readable medium as in claim 30, wherein the step of determining a series of synthetic synchronization marks includes the step of determining at least one of said synthetic synchronization marks at a resolution finer than the sampling interval.

37. A computer-readable medium as in claim 36, wherein the step of determining at least one of said synthetic synchronization marks at a resolution finer than the sampling interval includes the step of determining said at least one of said synthetic synchronization marks by a floating point number having a mantissa of at least twenty-four bits.

38. A computer-readable medium as in claim 30, wherein the step of shifting said waveforms to one of said synthetic synchronization marks corresponding to said one of said original synchronization marks includes the step of resampling said waveforms to adjust said waveforms to said one of said synthetic synchronization marks.

39. A computer-readable medium as in claim 38, wherein the step of shifting said waveforms to one of said synthetic synchronization marks corresponding to said one of said original synchronization marks further includes the step of shifting said waveforms to the nearest previous sampling interval of said one of said synthetic synchronization marks, before performed said step of resampling.

40. A computer-readable medium as in claim 30, wherein the step of generating said modified signal based on said shifted waveforms includes the steps of:

applying an asymmetric filtering window to said shifted waveforms; and

summing the windowed, shifted waveform to generate said modified signal.

41. A computer-readable medium as in claim 40, wherein:

said asymmetric filtering window has a first section and a second section in juxtaposition with each other;

said first section has an amplitude progressively increasing from zero to a non-zero value along a first width;

said second section has amplitude progressively decreasing from said non-zero value to zero along a second width; and

said first width is different is size from said second width.

42. A computer-readable medium as in claim 41, wherein:

said first width is the lesser of the interval between said one of said original synchronization marks and a preceding original synchronization mark and the interval between said one of said synthetic synchronization marks and a preceding synthetic synchronization mark; and

said second width is the lesser of the interval between said one of said original synchronization marks and a subsequent original synchronization mark and the interval between said one of said synthetic synchronization marks and a subsequent synthetic synchronization mark.

43. A computer-readable medium as in claim 42, wherein:

said first section is the first half of a Hanning window; and

said second section is the second half of a Hanning window.

44. A computer-readable medium bearing instructions for synthesizing a quasi-periodic signal from an original signal, said instructions arranged, when executed, to cause one or more processors to perform the steps of:

determining a series of original synchronization marks in said original signal;

determining a series of synthetic synchronization marks based on said original synchronization marks and on prosody information;

extracting a waveform from around each of said original synchronization marks by applying a filtering window and time-shifting each waveform according to a respective one of said original synchronization marks and a respective one of said synthetic synchronization marks corresponding to said respective one of said original synchronization marks to form a time-shifted signal;

applying asymmetric filtering windows to the time-shifted signal to extract overlapping frames; and

summing the overlapping frames to synthesize said quasi-periodic signal.

45. A computer-readable medium as in claim 44, wherein each said asymmetric filtering window has a first half-width on one side of a respective original synchronization mark and a second half-width on another side of the respective original synchronization mark, said first half-width different in size from said second half-width.

46. A computer-readable medium as in claim 45, wherein:

said asymmetric filtering window has a first section and a second section in juxtaposition with each other;

said first section has an amplitude progressively increasing from zero to a non-zero value along said first half-width; and

said second section has an amplitude progressively decreasing from said non-zero value to zero along said second half-width.

47. A computer-readable medium as in claim 46, wherein:

said first half-width is the lesser of the interval between said one of said original synchronization marks and a preceding original synchronization mark and the interval between said one of said synthetic synchronization marks and a preceding synthetic synchronization mark; and

said second half-width is the lesser of the interval between said one of said original synchronization marks and a subsequent original synchronization mark and the interval between said one of said synthetic synchronization marks and a subsequent synthetic synchronization mark.

48. A computer-readable medium as in claim 47, wherein:

said first section is the first half of a Hanning window; and

said second section is the second half of a Hanning window.

49. A computer-readable medium as in claim 44, wherein the step of windowing is performed after the step of time-shifting.

50. A computer-readable medium as in claim 45, wherein a difference of said one of said original synchronization marks and said one of said synthetic synchronization marks is a non-integral multiple of said sampling interval.

51. A computer-readable medium as in claim 50, wherein the step of determining a series of original synchronization marks in said quasi-periodic signal includes the step of determining at least one of said original synchronization marks at a resolution finer than the sampling interval.

52. A computer-readable medium as in claim 51, wherein the step of determining at least one of said original synchronization marks a t a resolution finer than the sampling interval includes the step of sampling the quasi-periodic signal at a shorter sampling interval with respect to said sampling interval.

53. A computer-readable medium as in claim 52, wherein said shorter sampling interval is at most one-third of said sampling interval.

54. A computer-readable medium as in claim 50, wherein the step of determining a series of original synchronization marks in said quasi-periodic signal includes the step of determining epochs in said quasi-periodic signal.

55. A computer-readable medium as in claim 50, wherein the step of determining a series of synthetic synchronization marks includes the step of determining at least one of said synthetic synchronization marks at a resolution finer than the sampling interval.

56. A computer-readable medium as in claim 55, wherein the step of determining at least one of said synthetic synchronization marks at a resolution finer than the sampling interval includes the step of determining said at least one of said synthetic synchronization marks by a floating point number having a mantissa of at least twenty-four bits.

57. A computer-readable medium as in claim 50, wherein the step of shifting said waveforms to one of said synthetic synchronization marks corresponding to said one of said original synchronization marks includes the step of resampling said waveforms to adjust said waveforms to said one of said synthetic synchronization marks.

58. A computer-readable medium as in claim 57, wherein step of shifting said waveforms to one of said synthetic synchronization marks corresponding to said one of said original synchronization marks further includes the step of shifting said waveforms to the nearest previous sampling interval of said one of said synthetic synchronization marks, before performed said step of resampling.