Prosody generating devise, prosody generating method, and program

• US 20070118355A1
• Filed: 01/17/2007
• Published: 05/24/2007
• Est. Priority Date: 03/08/2001
• Status: Active Grant

• Current Assignee: Sovereign Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Inventors: Kamai, Takahiro, Kato, Yumiko
• Granted Patent: US 8,738,381 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G10L 13/10 Prosody rules derived from text; Stress or intonation

Abstract

A prosody generation apparatus capable of suppressing distortion that occurs when generating prosodic patterns and therefore generating a natural prosody is provided. A prosody changing point extraction unit in this apparatus extracts a prosody changing point located at the beginning and the ending of a sentence, the beginning and the ending of a breath group, an accent position and the like. A selection rule and a transformation rule of a prosodic pattern including the prosody changing point is generated by means of a statistical or learning technique and the thus generate rules are stored in a representative prosodic pattern selection rule table and a transformation rule table beforehand. A pattern selection unit selects a representative prosodic pattern from the representative prosodic pattern selection rule table according to the selection rule. A prosody generation unit transforms the selected pattern according to the transformation rule and carries out interpolation with respect to portions other than the prosody changing points so as to generate prosody as a whole.

Claims

1. A prosody generation apparatus that receives phonological information and linguistic information so as to generate prosody, the prosody generation apparatus being operable to refer to (a) a representative prosodic pattern storage unit for accumulating beforehand representative prosodic patterns of portions of speech data, the portions including prosody changing points;

(b) a selection rule storage unit that stores a selection rule predetermined according to attributes concerning phonology or attributes concerning linguistic information of the portions of the speech data including the prosody changing points; and

(c) a transformation rule storage unit that stores a transformation rule predetermined according to attributes concerning the phonology or the linguistic information of the portions of the speech data including the prosody changing points;

comprising;

a prosody changing point setting unit that sets a prosody changing point according to at least any one of the received phonological information and the linguistic information;

a pattern selection unit that selects a representative prosodic pattern from the representative prosodic pattern storage unit according to the selection rule, based on the received phonological information and the linguistic information; and

a prosody generation unit that transforms the representative prosodic pattern selected by the pattern selection unit according to the transformation rule and interpolates a portion that does not include a prosody changing point and located between the thus selected and transformed representative patterns each corresponding to a portion including a prosody changing point.

2. The prosody generation apparatus according to claim 1, wherein the representative prosodic patterns are pitch patterns.

3. The prosody generation apparatus according to claim 1, wherein the representative patterns are power patterns.

4. The prosody generation apparatus according to claim 1, wherein the representative prosodic patterns are patterns generated for each of clusters into which patterns of the portions of the speech data including the prosodic changing points are clustered by means of a statistical technique.

13. The prosody generation apparatus according to claim 1, wherein the prosody changing point includes at least one of a beginning of an accent phrase, an ending of an accent phrase and an accent nucleus.

14. The prosody generation apparatus according to claim 1, wherein assuming that a difference in pitch between adjacent moras or adjacent syllables of the speech data is Δ

P, the prosody changing point is a point where the Δ

P and an immediately following Δ

P are different in sign.

15. The prosody generation apparatus according to claim 13, wherein the prosody changing point is a point where a sum of the Δ

P and the immediately following Δ

P exceeds a predetermined value.

16. The prosody generation apparatus according to claim 1, wherein assuming that a difference in pitch between adjacent moras or adjacent syllables of the speech data is Δ

P, the prosody changing point is a point where the Δ

P and an immediately following Δ

P have a same sign and a ratio between the Δ

P and the immediately following Δ

P exceeds a predetermined value.

17. The prosody generation apparatus according to claim 1, wherein assuming that a difference in pitch between adjacent moras or adjacent syllables of the speech data is Δ

P, the prosody changing point is a point where the Δ

P and an immediately following Δ

P have a same sign and a difference between the Δ

P and the immediately following Δ

P exceeds a predetermined value.

18. The prosody generation apparatus according to claim 17, wherein assuming that the Δ

P is obtained by subtracting a pitch of a preceding mora or syllable from a pitch of a following mora or syllable of the adjacent moras or syllables, the prosody changing point is a point where signs of the Δ

P and the immediately following Δ

P are minus, and a ratio between the Δ

P and the immediately following Δ

P is in a range of 1.5 to 2.5 and exceeds a predetermined value.

19. The prosody generation apparatus according to claim 17, wherein assuming that the Δ

P is obtained by subtracting a pitch of a preceding mora or syllable from a pitch of a following mora or syllable of the adjacent moras or syllables, the prosody changing point is a point where signs of the Δ

P and the immediately following Δ

P are minus, a sign of an immediately preceding Δ

P is plus, and a ratio between the Δ

P and the immediately following Δ

P is in a range of 1.2 to 2.0 and exceeds a predetermined value.

20. The prosody generation apparatus according to claim 1, wherein the prosody changing point setting unit sets the prosody changing point using at least one of the received phonological information and linguistic information, according to a prosody changing point extraction rule predetermined based on attributes concerning the phonology and attributes concerning the linguistic information of the prosody changing point of the speech data.

21. The prosody generation apparatus according to claim 20, wherein the prosody changing point extraction rule is obtained by formulating a relationship between (i) a classification as to whether adjacent moras or syllables of the speech data are a prosody changing point or not and (ii) attributes concerning phonology or attributes concerning linguistic information of the adjacent moras or syllables, by means of a statistical technique or a learning technique so as to predict whether a point is a prosody changing point or not using at least one of the attributes concerning phonology and the attributes concerning linguistic information.

22. The prosody generation apparatus according to claim 1, wherein assuming that a difference in power between adjacent moras or adjacent syllables of the speech data is Δ

A, the prosody changing point is a point where the Δ

A and an immediately following Δ

A are different in sign.

23. The prosody generation apparatus according to claim 22, wherein the prosody changing point is a point where a sum of an absolute value of the Δ

A and an absolute value of the immediately following Δ

A exceeds a predetermined value.

24. The prosody generation apparatus according to claim 1, wherein assuming that a difference in power between adjacent moras or adjacent syllables of the speech data is Δ

A, the prosody changing point is a point where the Δ

A and an immediately following Δ

A have a same sign and a ratio between the Δ

A and the immediately following Δ

A exceeds a predetermined value.

25. The prosody generation apparatus according to claim 1, wherein assuming that a difference in power between adjacent moras or adjacent syllables of the speech data is Δ

A, the prosody changing point is a point where the Δ

A and an immediately following Δ

A have a same sign and a difference between the Δ

A and the immediately following Δ

A exceeds a predetermined value.

26. The prosody generation apparatus according to claim 22, wherein a difference in power of vowels included in the adjacent moras or the adjacent syllables is used as the difference in power between the adjacent moras or the adjacent syllables.

27. The prosody generation apparatus according to claim 1, wherein assuming that a difference between values obtained by standardizing time lengths of adjacent moras, syllables or phonemes of the speech data for each type of phonology is Δ

D, the prosody changing point is a point where the Δ

D exceeds a predetermined value.

28. The prosody generation apparatus according to claim 1, wherein assuming that a difference between values obtained by standardizing time lengths of adjacent moras, syllables or phonemes of the speech data for each type of phonology is Δ

D, the prosody changing point is a point where the Δ

D and an immediately following Δ

D are different in sign.

29. The prosody generation apparatus according to claim 25, wherein the prosody changing point is a point where a sum of an absolute value of the Δ

D and an absolute value of the immediately following Δ

D exceeds a predetermined value.

30. The prosody generation apparatus according to claim 1, wherein assuming that a difference between values obtained by standardizing time lengths of adjacent moras, syllables or phonemes of the speech data for each type of phonology is Δ

D, the prosody changing point is a point where the Δ

D and an immediately following Δ

D have a same sign and a ratio between the Δ

D and the immediately following Δ

D exceeds a predetermined value.

31. The prosody generation apparatus according to claim 1, wherein assuming that a difference between values obtained by standardizing time lengths of adjacent moras, syllables or phonemes of the speech data for each type of phonology is Δ

D, the prosody changing point is a point where the Δ

D and an immediately following Δ

D have a same sign and a difference between the Δ

D and the immediately following Δ

D exceeds a predetermined value.

32. The prosody generation apparatus according to claim 1, wherein the attributes concerning phonology includes one or more of the following attributes:

(1) the number of phonemes, the number of moras, the number of syllables, an accent position, an accent type, an accent strength, a stress pattern or a stress strength of an accent phrase, a clause, a stress phrase, or a word;

(2) the number of moras, the number of syllables or the number of phonemes counted from a beginning of a sentence, a phrase, an accent phrase, a clause, or a word;

(3) the number of moras, the number of syllables, or the number of phonemes counted from an ending of a sentence, a phrase, an accent phrase, a clause, or a word;

(4) the presence or absence of adjacent pauses;

(5) a time length of adjacent pauses;

(6) a time length of a pause located before and the nearest to the prosody changing point;

(7) a time length of a pause located after and the nearest to the prosody changing point;

(8) the number of moras, the number of syllables or the number of phonemes counted from a pause located before and the nearest to the prosody changing point;

(9) the number of moras, the number of syllables or the number of phonemes counted from a pause located after and the nearest to the prosody changing point; and

(10) the number of moras, the number of syllables or the number of phonemes counted from an accent nucleus or a stress position.

33. The prosody generation apparatus according to claim 1, wherein the attributes concerning linguistic information includes one or more of the following attributes:

a part of speech, an attribute concerning a modification structure, a distance to a modifiee, a distance to a modifier, an attribute concerning syntax, prominence, emphasis, or semantic classification of an accent phrase, a clause, a stress phrase, or a word.

34. The prosody generation apparatus according to claim 1, wherein the selection rule is obtained by formulating a relationship between (i) clusters corresponding to the representative patterns and into which prosodic patterns of the speech data are clustered and classified and (ii) attributes concerning phonology or attributes concerning linguistic information of each of the prosodic patterns, by means of a statistical technique or a learning technique so as to predict a cluster to which a prosodic pattern including the prosody changing point belongs, using at least one of the attributes concerning phonology and the attributes concerning linguistic information.

35. The prosody generation apparatus according to claim 1, wherein the transformation is a parallel shifting along a frequency axis of a pitch pattern.

36. The prosody generation apparatus according to claim 1, wherein the transformation is a parallel shifting along a logarithmic axis of a frequency of a pitch pattern.

37. The prosody generation apparatus according to claim 1, wherein the transformation is a parallel shifting along an amplitude axis of a power pattern.

38. The prosody generation apparatus according to claim 1, wherein the transformation is a parallel shifting along a power axis of a power pattern.

39. The prosody generation apparatus according to any claim 1, wherein the transformation is compression or extension in a dynamic range on a frequency axis of a pitch pattern.

40. The prosody generation apparatus according to claim 1, wherein the transformation is compression or extension in a dynamic range on a logarithmic axis of a pitch pattern.

41. The prosody generation apparatus according to claim 1, wherein the transformation is compression or extension in a dynamic range on an amplitude axis of a power pattern.

42. The prosody generation apparatus according to claim 1, wherein the transformation is compression or extension in a dynamic range on a power axis of a power pattern.

43. The prosody generation apparatus according to claim 1, wherein the transformation rule is obtained by clustering prosodic patterns of the speech data into clusters corresponding to the representative patterns so as to produce a representative pattern for each cluster and by formulating a relationship between (i) a distance between each of the prosodic patterns and a representative pattern of a cluster to which the prosodic pattern belongs and (ii) attributes concerning phonology or attributes concerning linguistic information of the prosodic pattern, by means of a statistical technique or a learning technique so as to estimate an amount of transformation of the selected prosodic pattern, using at least one of the attributes concerning phonology and the attributes concerning linguistic information.

44. The prosody generation apparatus according to claim 43, wherein the amount of transformation is one of a shifting amount, a compression rate in a dynamic range and an extension rate in a dynamic range.

49. The prosody generation apparatus according to claim 43, wherein the statistical technique is the Quantification Theory Type I where the shifting amount of a representative prosodic pattern is designated as a criterion variable.

50. The prosody generation apparatus according to claim 43, wherein the statistical technique is the Quantification Theory Type I where a compression rate or an extension rate in a dynamic range of a representative prosodic pattern of a cluster is designated as a criterion variable.

61. The prosody generation apparatus according to claim 21, wherein the statistical technique is a multivariate analysis, a decision tree, or the Quantification Theory Type II where a type of a cluster is designated as a criterion variable.

62. The prosody generation apparatus according to claim 34, wherein the statistical technique is a multivariate analysis, a decision tree, the Quantification Theory Type II where a type of a cluster is designated as a criterion variable, or the Quantification Theory Type I where a distance between a representative prosodic pattern in a cluster and each prosodic data is designated as a criterion variable.

63. The prosody generation apparatus according to claim 43, wherein the statistical technique is the Quantification Theory Type I where a distance between a representative prosodic pattern in a cluster and each prosodic data is designated as a criterion variable.

64. The prosody generation apparatus according to claim 1, wherein the interpolation is a linear interpolation, by means of a spline function, or by means of a sigmoid curve.

65. The prosody generation apparatus according to claim 3, wherein the power is (i) a value obtained by standardizing a power of a mora or a syllable for each type of phonology, or (ii) an amplitude value of a sound source waveform of a mora or a syllable.

66. The prosody generation apparatus according to claim 22, wherein the power is (i) a value obtained by standardizing a power of a mora or a syllable for each type of phonology, or (ii) an amplitude value of a sound source waveform of a mora or a syllable.

67. The prosody generation apparatus according to claim 24, wherein the power is (i) a value obtained by standardizing a power of a mora or a syllable for each type of phonology, or (ii) an amplitude value of a sound source waveform of a mora or a syllable.

68. The prosody generation apparatus according to claim 25, wherein the power is (i) a value obtained by standardizing a power of a mora or a syllable for each type of phonology, or (ii) an amplitude value of a sound source waveform of a mora or a syllable.

5-12. -12. (canceled)

45-48. -48. (canceled)

51-56. -56. (canceled)

57. A prosody generation method by which phonological information and linguistic information are inputted so as to generate prosody, comprising the steps of:

setting a prosody changing point according to at least any one of the inputted phonological information and linguistic information;

selecting a prosodic pattern from representative prosodic patterns for portions including prosody changing points of speech data according to a selection rule predetermined beforehand based on attributes concerning phonology or attributes concerning linguistic information of the portions including the prosodic changing points; and

transforming the selected prosodic pattern according to a transformation rule predetermined beforehand based on attributes concerning the phonology or attributes concerning the linguistic information of the portions including the prosodic changing points, and interpolating a portion that does not include a prosody changing point and located between the thus selected and transformed representative patterns each corresponding to a portion including a prosody changing point.

58. (canceled)

59. A program that has a computer conduct a procedure of receiving phonological information and linguistic information so as to generate prosody, the computer being operable to refer to (a) a representative prosodic pattern storage unit for accumulating beforehand representative prosodic patterns of portions of speech data, the portions including prosody changing points;

(b) a selection rule storage unit that stores a selection rule predetermined according to attributes concerning phonology or attributes concerning linguistic information of the portions of the speech data including the prosody changing points; and

(c) a transformation rule storage unit that stores a transformation rule predetermined according to attributes concerning the phonology or the linguistic information of the portions of the speech data including the prosody changing points;

the program having the computer conduct the steps of;

setting a prosody changing point according to at least any one of the received phonological information and the linguistic information;

selecting a representative prosodic pattern from the representative prosodic pattern storage unit according to the selection rule, based on the received phonological information and the linguistic information; and

transforming the representative prosodic pattern selected by the pattern selection unit according to the transformation rule and interpolating a portion that does not include a prosody changing point and located between the thus selected and transformed representative patterns each corresponding to a portion including a prosody changing point.

60. (canceled)