METHODS AND APPARATUS FOR PREDICTING PROSODY IN SPEECH SYNTHESIS

• US 20120191457A1
• Filed: 01/24/2011
• Published: 07/26/2012
• Est. Priority Date: 01/24/2011
• Status: Active Grant

• Current Assignee: Cerence Operating Company (Cerence Inc.)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Minnis, Stephen, Breen, Andrew P.
• Granted Patent: US 9,286,886 B2
• Time in Patent Office: Days
• US Class Current: 704/260
• CPC Class Codes: G10L 13/08 Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
• CPC Class Codes: G10L 13/10 Prosody rules derived from text; Stress or intonation

Abstract

Techniques for predicting prosody in speech synthesis may make use of a data set of example text fragments with corresponding aligned spoken audio. To predict prosody for synthesizing an input text, the input text may be compared with the data set of example text fragments to select a best matching sequence of one or more example text fragments, each example text fragment in the sequence being paired with a portion of the input text. The selected example text fragment sequence may be aligned with the input text, e.g., at the word level, such that prosody may be extracted from the audio aligned with the example text fragments, and the extracted prosody may be applied to the synthesis of the input text using the alignment between the input text and the example text fragments.

Claims

1. A method comprising:

comparing an input text to a data set of text fragments to select a corresponding text fragment for at least a portion of the input text, the corresponding text fragment being associated with spoken audio, wherein the corresponding text fragment does not exactly match the at least a portion of the input text because at least one word is present in one of the matching text fragment and the at least a portion of the input text, but not in both;

determining an alignment of the corresponding text fragment with the at least a portion of the input text; and

using a computer, synthesizing speech from the at least a portion of the input text, wherein the synthesizing comprises extracting prosody from the spoken audio and applying the extracted prosody using the alignment of the corresponding text fragment with the at least a portion of the input text.

2. The method of claim 1, wherein selecting the corresponding text fragment comprises:

identifying a first marker included in the at least a portion of the input text;

identifying a class of the first marker; and

selecting the corresponding text fragment based at least in part on the corresponding text fragment comprising a second marker of the same class as the first marker.

3. The method of claim 2, wherein the class of the first marker is a function word class.

4. The method of claim 2, wherein the class of the first marker is selected from the group consisting of one or more punctuation classes, one or more context markup classes and one or more filler classes.

5. The method of claim 2, wherein determining the alignment comprises aligning the second marker with the first marker.

6. The method of claim 1, wherein the comparing comprises selecting the corresponding text fragment based at least in part on a similarity measure between one or more linguistic features of the at least a portion of the input text and the corresponding text fragment.

7. The method of claim 6, wherein the similarity measure is determined based at least in part on a ratio of words that appear in both the at least a portion of the input text and the corresponding text fragment.

8. The method of claim 6, wherein the similarity measure is determined based at least in part on a ratio of words having matching parts of speech between the at least a portion of the input text and the corresponding text fragment.

9. The method of claim 6, wherein the one or more linguistic features comprise one or more features selected from the group consisting of a named entity feature, a verb semantics feature, a noun semantics feature, an adjective semantics feature, an adverb semantics feature, and a syllable structure feature.

10. The method of claim 1, wherein the comparing comprises selecting a sequence of corresponding text fragments for the input text.

11. The method of claim 10, wherein the comparing further comprises:

analyzing the input text to identify a sequence of markers in the input text; and

selecting the sequence of corresponding text fragments from one or more candidate sequences matching the sequence of markers.

12. The method of claim 11, wherein determining the alignment comprises aligning the sequence of markers in the input text with markers in the sequence of corresponding text fragments.

13. The method of claim 11, wherein the comparing further comprises:

computing a join cost for each of the one or more candidate sequences; and

selecting the sequence of corresponding text fragments from the one or more candidate sequences based at least in part on the join cost.

14. The method of claim 10, wherein the comparing further comprises:

inputting the at least a portion of the input text to a statistical model to divide the input text into a sequence of input text fragments; and

selecting the sequence of corresponding text fragments from one or more candidate sequences matching the sequence of input text fragments.

15. The method of claim 10, wherein at least a first text fragment is adjacent in the sequence of corresponding text fragments to a second text fragment, the first text fragment being associated with first spoken audio and the second text fragment being associated with second spoken audio, wherein the first spoken audio was not spoken consecutively with the second spoken audio.

16. The method of claim 1, wherein the spoken audio is aligned with the corresponding text fragment, and the synthesizing comprises extracting prosody from the spoken audio using the alignment of the spoken audio with the corresponding text fragment.

17. The method of claim 1, wherein the synthesizing comprises extracting at least one prosodic feature from the spoken audio, and incorporating the at least one prosodic feature into the synthesized speech, without incorporating the spoken audio into the synthesized speech.

18. The method of claim 1, wherein the extracting comprises specifying prosody for synthesizing the at least a portion of the input text by inputting the corresponding text fragment to a statistical model trained at least partly on the spoken audio.

19. The method of claim 1, wherein the synthesizing comprises specifying at least one prosodic contour for synthesizing the at least a portion of the input text, wherein the at least one prosodic contour is selected from the group consisting of a fundamental frequency contour, an amplitude contour and a duration contour.

20. The method of claim 1, wherein the data set is specific to a domain to which the input text belongs.

21. A system comprising:

at least one memory storing processor-executable instructions; and

at least one processor operatively coupled to the at least one memory, the at least one processor being configured to execute the processor-executable instructions to perform a method comprising;

comparing an input text to a data set of text fragments to select a corresponding text fragment for at least a portion of the input text, the corresponding text fragment being associated with spoken audio, wherein the corresponding text fragment does not exactly match the at least a portion of the input text because at least one word is present in one of the matching text fragment and the at least a portion of the input text, but not in both;

determining an alignment of the corresponding text fragment with the at least a portion of the input text; and

synthesizing speech from the at least a portion of the input text, wherein the synthesizing comprises extracting prosody from the spoken audio and applying the extracted prosody using the alignment of the corresponding text fragment with the at least a portion of the input text.

22. The system of claim 21, wherein selecting the corresponding text fragment comprises:

identifying a first marker included in the at least a portion of the input text;

identifying a class of the first marker; and

selecting the corresponding text fragment based at least in part on the corresponding text fragment comprising a second marker of the same class as the first marker.

23. The system of claim 22, wherein the class of the first marker is a function word class.

24. The system of claim 22, wherein the class of the first marker is selected from the group consisting of one or more punctuation classes, one or more context markup classes and one or more filler classes.

25. The system of claim 22, wherein determining the alignment comprises aligning the second marker with the first marker.

26. The system of claim 21, wherein the comparing comprises selecting the corresponding text fragment based at least in part on a similarity measure between one or more linguistic features of the at least a portion of the input text and the corresponding text fragment.

27. The system of claim 26, wherein the similarity measure is determined based at least in part on a ratio of words that appear in both the at least a portion of the input text and the corresponding text fragment.

28. The system of claim 26, wherein the similarity measure is determined based at least in part on a ratio of words having matching parts of speech between the at least a portion of the input text and the corresponding text fragment.

29. The system of claim 26, wherein the one or more linguistic features comprise one or more features selected from the group consisting of a named entity feature, a verb semantics feature, a noun semantics feature, an adjective semantics feature, an adverb semantics feature, and a syllable structure feature.

30. The system of claim 21, wherein the comparing comprises selecting a sequence of corresponding text fragments for the input text.

31. The system of claim 30, wherein the comparing further comprises:

analyzing the input text to identify a sequence of markers in the input text; and

selecting the sequence of corresponding text fragments from one or more candidate sequences matching the sequence of markers.

32. The system of claim 31, wherein determining the alignment comprises aligning the sequence of markers in the input text with markers in the sequence of corresponding text fragments.

33. The system of claim 31, wherein the comparing further comprises:

computing a join cost for each of the one or more candidate sequences; and

selecting the sequence of corresponding text fragments from the one or more candidate sequences based at least in part on the join cost.

34. The system of claim 30, wherein the comparing further comprises:

inputting the at least a portion of the input text to a statistical model to divide the input text into a sequence of input text fragments; and

selecting the sequence of corresponding text fragments from one or more candidate sequences matching the sequence of input text fragments.

35. The system of claim 30, wherein at least a first text fragment is adjacent in the sequence of corresponding text fragments to a second text fragment, the first text fragment being associated with first spoken audio and the second text fragment being associated with second spoken audio, wherein the first spoken audio was not spoken consecutively with the second spoken audio.

36. The system of claim 21, wherein the spoken audio is aligned with the corresponding text fragment, and the synthesizing comprises extracting prosody from the spoken audio using the alignment of the spoken audio with the corresponding text fragment.

37. The system of claim 21, wherein the synthesizing comprises extracting at least one prosodic feature from the spoken audio, and incorporating the at least one prosodic feature into the synthesized speech, without incorporating the spoken audio into the synthesized speech.

38. The system of claim 21, wherein the extracting comprises specifying prosody for synthesizing the at least a portion of the input text by inputting the corresponding text fragment to a statistical model trained at least partly on the spoken audio.

39. The system of claim 21, wherein the synthesizing comprises specifying at least one prosodic contour for synthesizing the at least a portion of the input text, wherein the at least one prosodic contour is selected from the group consisting of a fundamental frequency contour, an amplitude contour and a duration contour.

40. The system of claim 21, wherein the data set is specific to a domain to which the input text belongs.

41. At least one computer-readable storage medium encoded with a plurality of computer-executable instructions that, when executed, perform a method comprising:

comparing an input text to a data set of text fragments to select a corresponding text fragment for at least a portion of the input text, the corresponding text fragment being associated with spoken audio, wherein the corresponding text fragment does not exactly match the at least a portion of the input text because at least one word is present in one of the matching text fragment and the at least a portion of the input text, but not in both;

determining an alignment of the corresponding text fragment with the at least a portion of the input text; and

synthesizing speech from the at least a portion of the input text, wherein the synthesizing comprises extracting prosody from the spoken audio and applying the extracted prosody using the alignment of the corresponding text fragment with the at least a portion of the input text.

42. The at least one computer-readable storage medium of claim 41, wherein selecting the corresponding text fragment comprises:

identifying a first marker included in the at least a portion of the input text;

identifying a class of the first marker; and

selecting the corresponding text fragment based at least in part on the corresponding text fragment comprising a second marker of the same class as the first marker.

43. The at least one computer-readable storage medium of claim 42, wherein the class of the first marker is a function word class.

44. The at least one computer-readable storage medium of claim 42, wherein the class of the first marker is selected from the group consisting of one or more punctuation classes, one or more context markup classes and one or more filler classes.

45. The at least one computer-readable storage medium of claim 42, wherein determining the alignment comprises aligning the second marker with the first marker.

46. The at least one computer-readable storage medium of claim 41, wherein the comparing comprises selecting the corresponding text fragment based at least in part on a similarity measure between one or more linguistic features of the at least a portion of the input text and the corresponding text fragment.

47. The at least one computer-readable storage medium of claim 46, wherein the similarity measure is determined based at least in part on a ratio of words that appear in both the at least a portion of the input text and the corresponding text fragment.

48. The at least one computer-readable storage medium of claim 46, wherein the similarity measure is determined based at least in part on a ratio of words having matching parts of speech between the at least a portion of the input text and the corresponding text fragment.

49. The at least one computer-readable storage medium of claim 46, wherein the one or more linguistic features comprise one or more features selected from the group consisting of a named entity feature, a verb semantics feature, a noun semantics feature, an adjective semantics feature, an adverb semantics feature, and a syllable structure feature.

50. The at least one computer-readable storage medium of claim 41, wherein the comparing comprises selecting a sequence of corresponding text fragments for the input text.

51. The at least one computer-readable storage medium of claim 50, wherein the comparing further comprises:

analyzing the input text to identify a sequence of markers in the input text; and

selecting the sequence of corresponding text fragments from one or more candidate sequences matching the sequence of markers.

52. The at least one computer-readable storage medium of claim 51, wherein determining the alignment comprises aligning the sequence of markers in the input text with markers in the sequence of corresponding text fragments.

53. The at least one computer-readable storage medium of claim 51, wherein the comparing further comprises:

computing a join cost for each of the one or more candidate sequences; and

selecting the sequence of corresponding text fragments from the one or more candidate sequences based at least in part on the join cost.

54. The at least one computer-readable storage medium of claim 50, wherein the comparing further comprises:

inputting the at least a portion of the input text to a statistical model to divide the input text into a sequence of input text fragments; and

selecting the sequence of corresponding text fragments from one or more candidate sequences matching the sequence of input text fragments.

55. The at least one computer-readable storage medium of claim 50, wherein at least a first text fragment is adjacent in the sequence of corresponding text fragments to a second text fragment, the first text fragment being associated with first spoken audio and the second text fragment being associated with second spoken audio, wherein the first spoken audio was not spoken consecutively with the second spoken audio.

56. The at least one computer-readable storage medium of claim 41, wherein the spoken audio is aligned with the corresponding text fragment, and the synthesizing comprises extracting prosody from the spoken audio using the alignment of the spoken audio with the corresponding text fragment.

57. The at least one computer-readable storage medium of claim 41, wherein the synthesizing comprises extracting at least one prosodic feature from the spoken audio, and incorporating the at least one prosodic feature into the synthesized speech, without incorporating the spoken audio into the synthesized speech.

58. The at least one computer-readable storage medium of claim 41, wherein the extracting comprises specifying prosody for synthesizing the at least a portion of the input text by inputting the corresponding text fragment to a statistical model trained at least partly on the spoken audio.

59. The at least one computer-readable storage medium of claim 41, wherein the synthesizing comprises specifying at least one prosodic contour for synthesizing the at least a portion of the input text, wherein the at least one prosodic contour is selected from the group consisting of a fundamental frequency contour, an amplitude contour and a duration contour.

60. The at least one computer-readable storage medium of claim 41, wherein the data set is specific to a domain to which the input text belongs.