Prosodic databases holding fundamental frequency templates for use in speech synthesis

• US 5,905,972 A
• Filed: 09/30/1996
• Issued: 05/18/1999
• Est. Priority Date: 09/30/1996
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Goldsmith, John A., Adcock, James L., Huang, Xuedong D.
• Primary Examiner(s): Hudspeth, David R.
• Assistant Examiner(s): ABEBE, DANIEL DEMELASH
• Application Number:
  US08/724,071
• Time in Patent Office: 960 Days
• Field of Search: 704/268, 704/258, 704/278, 704/4, 704/6, 704/10, 704/7
• US Class Current: 704/268
• CPC Class Codes: G10L 13/04 Details of speech synthesis systems, e.g. synthesiser structure or memory management
• CPC Class Codes: G10L 13/08 Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
• CPC Class Codes: G10L 15/144 Training of HMMs
• CPC Class Codes: G10L 2025/903 using a laryngograph

Abstract

Prosodic databases hold fundamental frequency templates for use in a speech synthesis system. Prosodic database templates may hold fundamental frequency values for syllables in a given sentence. These fundamental frequency values may be applied in synthesizing a sentence of speech. The templates are indexed by tonal pattern markings. A predicted tonal marking pattern is generated for each sentence of text that is to be synthesized, and this predicted pattern of tonal markings is used to locate a best-matching template. The templates are derived by calculating fundamental frequencies on a pursuable basis for sentences that are spoken by a human trainer for a given unlabeled corpus.

Claims

1. In a system for synthesizing speech, a method comprising the computer-implemented steps of:

providing text for which speech is to be synthesized;

providing prosodic templates created from a corpus of words spoken in a prosodic manner where each template holds a sequence of fundamental frequency values for units of speech;

selecting one of the templates for use in establishing prosody for the synthesized speech for the text; and

synthesizing speech for the text using at least one of the fundamental frequencies from the selected template in establishing prosody for the speech.

2. The method of claim 1, further comprising the step of segmenting text into units of speech to establish segmentation for the text.

3. The method of claim 1 wherein the units of speech are syllables.

4. The method of claim 1 wherein the text is a phrase.

5. The method of claim 1 wherein the text is a sentence.

6. The method of claim 1 wherein each prosodic template is indexed by a tonal emphasis marker pattern wherein a tonal marker is provided for each unit of speech in the text.

7. The method of claim 6, further comprising the step of generating a predicted tonal marker pattern for the text, wherein the selecting of one of the templates is performed by selecting a template with an index that best matches the predicted tonal marker pattern.

8. In a system for synthesizing speech, a computer-readable storage medium holding instructions for performing a method comprising the computer-implemented steps of:

providing text for which speech is to be synthesized;

providing prosodic templates created from a corpus of words spoken in a prosodic manner where each template holds a sequence of fundamental frequency values for units of speech;

selecting one of the templates for use in establishing prosody for the synthesized speech for the text; and

synthesizing speech for the text using at least one of the fundamental frequencies from the selected template in establishing prosody for the speech.

9. The computer-readable storage medium of claim 8 wherein the storage medium also holds instructions for segmenting the text into units of speech.

10. The computer-readable storage medium of claim 9 wherein the units of speech are syllables.

11. The computer-readable storage medium of claim 8 wherein each prosodic template is indexed by a tonal emphasis marker pattern wherein a tonal marker is provided for each unit of speech in the text.

12. The computer-readable storage medium of claim 9 wherein the method further comprises the step of:

generating a predicted tonal marker pattern for the text; and

wherein the selecting of one of the templates is performed by selecting a template with an index that best matches the predicted tonal marker pattern.

13. In a system for synthesizing speech, a method comprising the computer-implemented steps of:

providing a prosodic database of fundamental frequencies for units of speech, each entry in said prosodic database being indexed by a pattern of tonal markings that correspond with a degree of emphasis for the units of speech for which fundamental frequencies are held;

performing a natural language parse on a given text;

based on results of the natural language parse, predicting a predicted pattern of tonal markings for the units of speech in the text;

identifying a best-matching index in the prosodic database by comparing the predicted pattern of tonal markings for the units of speech in the text with the indices of the entries of the prosodic database; and

using at least one of the fundamental frequency values in the entry in the prosodic database that is indexed by the best-matching index to establish prosody in synthesizing speech for the text.

14. The method of claim 13 wherein the best-matching index exactly matches the predicted pattern of tonal markings.

15. The method of claim 13 wherein all of the fundamental frequency values in the entry that is indexed by the best-matching index are used in establishing prosody.

16. The method of claim 13 wherein the best-matching index does not exactly match the predicted pattern of tonal markings.

17. The method of claim 13 wherein the tonal markings include a high emphasis tonal marker, a low emphasis tonal marker, a no special emphasis tonal marker and a marker that specifies unmarked stress.

18. The method of claim 17 wherein the best-matching index differs from the predicted pattern of tonal markings in number of consecutive unmarked stress for the units of speech.

19. The method of claim 18, further comprising the steps of:

identifying a non-matching portion of the best-matching index that does not match the predicted pattern of tonal markings and the fundamental frequency values, in the entry in the prosodic database that is indexed by the best-matching index that corresponds to the non-matching portion of the best-matching index;

applying linear interpolation between the bounding fundamental frequency values in the prosodic database entry indexed by the best-matching index that bound the identified fundamental frequency values in the entry that corresponds to the non-matching portion of the best-matching index to create a continuous function between the bounding fundamental frequency values;

resampling the continuous function to obtain a number of fundamental frequency values for unmarked stress units of speech that matches the number of consecutive unmarked stress markers in the predicted pattern of tonal markings; and

using the fundamental frequency values obtained by resampling in establishing prosody in synthesizing speech for the text.

20. The method of claim 13 wherein a Viterbi search is employed to identify the best-matching index.

21. In a system for synthesizing speech, a computer-readable storage medium holding instructions for performing a method comprising the computer-implemented steps of:

providing a prosodic database at fundamental frequencies for units of speech, each entry in said prosodic database being indexed by a pattern of tonal markings that correspond with a degree of emphasis for the units of speech for which fundamental frequencies are held;

performing a natural language parse on a given text;

based on results of the natural language parse, predicting a predicted pattern of tonal markings for the units of speech in the text;

identifying a best-matching index in the prosodic database by comparing the predicted pattern of tonal markings for the units of speech in the text with the indices of the entries of the prosodic database; and

using at least one of the fundamental frequency values in the entry in the prosodic database that is indexed by the best-matching index to establish prosody in synthesizing speech for the text.

22. The computer-readable storage medium of claim 21 wherein the best-matching index exactly matches the predicted pattern of tonal markings.

23. The computer-readable storage medium of claim 21 wherein all of the fundamental frequency values in the entry that is indexed by the best-matching index are used in establishing prosody.

24. The computer-readable storage medium of claim 21 wherein the best-matching index does not exactly match the predicted pattern of tonal markings.

25. The computer-readable storage medium of claim 21 wherein the tonal markings include a high emphasis tonal marker, a low emphasis tonal marker, a no special emphasis tonal marker and a marker that specifies unmarked stress.

26. The computer-readable storage medium of claim 25 wherein the best-matching index differs from the predicted pattern of tonal markings in number of consecutive unmarked stress for the units of speech.

27. The computer-readable storage medium of claim 26 wherein the medium also holds instructions for:

identifying a non-matching portion of the best-matching index that does not match the predicted pattern of tonal markings and the fundamental frequency values, in the entry in the prosodic database that is indexed by the best-matching index that corresponds to the non-matching portion of the best-matching index;

applying linear interpolation between the bounding fundamental frequency values in the prosodic database entry indexed by the best-matching index that bound the identified fundamental frequency values in the entry that corresponds to the non-matching portion of the best-matching index to create a continuous function between the bounding fundamental frequency values;

resampling the continuous function to obtain a number of fundamental frequency values for unmarked stress units of speech that matches the number of consecutive unmarked stress markers in the predicted pattern of tonal markings; and

using the fundamental frequency values obtained by resampling in establishing prosody in synthesizing speech for the text.

28. The computer-readable storage medium of claim 21 wherein a Viterbi search is employed to identify the best-matching index.

29. In a computer system, a method of building a prosodic database, comprising the computer-implemented steps of:

obtaining an acoustical signal for each of multiple corresponding portions of spoken text that are spoken by a human trainer, each said acoustical signal being the signal that results when the human trainer speaks the corresponding portion of text;

obtaining a laryngograph signal for each portion of spoken text from a laryngograph worn by the human trainer when the portions of text are spoken;

segmenting the acoustical signal into segments representing syllables in the text where each syllable includes a vowel section;

segmenting the laryngograph signal into segments that match the segments of the acoustical signal;

calculating a weighted sum of instantaneous fundamental frequencies for the vowel section of each syllable in each portion of text wherein the fundamental frequencies are obtained from the laryngograph signal and weights are obtained from the acoustical signal;

for each portion of text, storing the weighted sum of instantaneous fundamental frequencies for each syllable of the portion of text in the prosodic database; and

using the weighted sums of instantaneous fundamental frequencies in the prosodic database to establish prosody of synthesized speech.

30. The method of claim 29 wherein, for each vowel section of a syllable, the laryngograph signal includes peaks and wherein each instantaneous fundamental frequency is calculated to be 1 divided by a quantity of time that separates immediately adjacent peaks.

31. The method of claim 30 wherein each instantaneous fundamental frequency is weighted based on magnitude of the acoustical signal during a portion of the acoustical signal that corresponds with the immediately adjacent peaks in the laryngograph signal.

32. In a computer system, a computer-readable storage medium holding instructions for performing a method of building a prosodic database comprising the steps of:

obtaining an acoustical signal for each of multiple corresponding portions of spoken text that are spoken by a human trainer, each said acoustical signal being the signal that results when the human trainer speaks the corresponding portion of text;

obtaining a laryngograph signal for each portion of spoken text from a laryngograph worn by the human trainer when the portions of text are spoken;

segmenting the acoustical signal into segments representing syllables in the text where each syllable includes a vowel section;

segmenting the laryngograph signal into segments that match the segments of the acoustical signal;

calculating a weighted sum of instantaneous fundamental frequencies for the vowel section of each syllable in each portion of text wherein the fundamental frequencies are obtained from the laryngograph signal and weights are obtained from the acoustical signal;

for each portion of text, storing the weighted sum of instantaneous fundamental frequencies for each syllable of the portion of text in the prosodic database; and

using the weighted sums of instantaneous fundamental frequencies in the prosodic database to establish prosody of synthesized speech.

33. A text to speech system, comprising:

a parser for parsing input text into units of speech;

a prosodic database holding prosodic templates created from a corpus of words spoken in a prosodic manner wherein each prosodic template holds a sequence of fundamental frequency values for units of speech; and

a speech synthesizer for generating speech corresponding to the input text by using a selected one of the templates in the prosodic database to obtain fundamental frequency values for units of speech in the input text.

34. The text to speech system of claim 33 wherein the system further comprises a tonal pattern predictor for predicting the tonal pattern for the input text and wherein the speech synthesizer uses the predicted tonal pattern for the input text to select the selected template in the prosodic database.

35. In a system for generating speech, a method comprising the computer-implemented steps of:

providing a prosodic database holding prosodic templates for different styles of speech, each prosodic template comprising a sequence of fundamental frequencies associated with using the template'"'"'s respective style of speech while speaking a sequence of language units;

determining what prosodic style to apply to a portion of speech to be generated; and

using at least one of the templates in the prosodic database for the determined prosodic style to generate the portion of speech with the determined prosodic style.

36. In a system for generating speech, a computer-readable medium holding computer-executable instructions for performing a method comprising the computer-implemented steps of:

providing a prosodic database holding prosodic templates for different styles of speech, each prosodic template comprising a sequence of fundamental frequencies associated with using the template'"'"'s respective style of speech while speaking a sequence of language units;

determining what prosodic style to apply to a portion of speech to be generated; and

using at least one of the templates in the prosodic database for the determined prosodic style to generate the portion of speech with the determined prosodic style.

37. In a system for generating speech, a method comprising the computer-implemented steps of:

providing a prosodic database holding prosodic templates of different prosodic styles for a single speaker, each prosodic template comprising a series of fundamental frequencies associated with the single speaker using the template'"'"'s associated prosodic style while speaking a sequence of language units;

determining which of the prosodic styles is to be applied to a portion of speech that is to be generated; and

using at least one of the templates in the prosodic database for the determined prosodic style to generate the portion of speech with the determined prosodic style.

38. In a system for generating speech, a computer-readable medium holding computer-executable instructions for performing a method comprising the computer-implemented steps of:

providing a prosodic database holding prosodic templates of different prosodic styles for a single speaker, each prosodic template comprising a series of fundamental frequencies associated with the single speaker using the template'"'"'s associated prosodic style while speaking a sequence of language units;

determining which of the prosodic styles is to be applied to a portion of speech that is to be generated; and

using at least one of the templates in the prosodic database for the determined prosodic style to generate the portion of speech with the determined prosodic style.