Prosody generating devise, prosody generating method, and program

• US 8,738,381 B2
• Filed: 01/17/2007
• Issued: 05/27/2014
• Est. Priority Date: 03/08/2001
• Status: Expired due to Fees

• Current Assignee: Sovereign Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Kato, Yumiko, Kamai, Takahiro
• Primary Examiner(s): Armstrong, Angela A
• Application Number:
  US11/654,295
• Publication Number:
  US 20070118355A1
• Time in Patent Office: 2,687 Days
• Field of Search: 704/244, 704246-247, 704/258, 704/260, 704/265, 704/268
• US Class Current: 704/268
• CPC Class Codes: G10L 13/10 Prosody rules derived from text; Stress or intonation

Abstract

A prosody generation apparatus capable of suppressing distortion that occurs when generating prosodic patterns and therefore generating a natural prosody is provided. A prosody changing point extraction unit in this apparatus extracts a prosody changing point located at the beginning and the ending of a sentence, the beginning and the ending of a breath group, an accent position and the like. A selection rule and a transformation rule of a prosodic pattern including the prosody changing point is generated by means of a statistical or learning technique and the thus generate rules are stored in a representative prosodic pattern selection rule table and a transformation rule table beforehand. A pattern selection unit selects a representative prosodic pattern from the representative prosodic pattern selection rule table according to the selection rule. A prosody generation unit transforms the selected pattern according to the transformation rule and carries out interpolation with respect to portions other than the prosody changing points so as to generate prosody as a whole.

Claims

1. A prosody generation apparatus that receives phonological information and linguistic information so as to generate prosody, the prosody generation apparatus being operable to refer to (a) a representative prosodic pattern storage unit for accumulating beforehand representative prosodic patterns of portions of speech data, the portions including prosody changing points;

(b) a selection rule storage unit that stores a selection rule predetermined according to attributes concerning phonology or attributes concerning linguistic information of the portions of the speech data including the prosody changing points; and

(c) a transformation rule storage unit that stores a transformation rule predetermined according to attributes concerning the phonology or the linguistic information of the portions of the speech data including the prosody changing points;

the prosody generation apparatus comprising a computer processing unit and a memory storing a program that are configured to implement;

a prosody changing point setting unit that sets a prosody changing point according to at least any one of the received phonological information and the linguistic information;

a pattern selection unit that selects a representative prosodic pattern from the representative prosodic pattern storage unit according to the selection rule, based on the received phonological information and the linguistic information; and

a prosody generation unit that transforms the representative prosodic pattern selected by the pattern selection unit according to the transformation rule and interpolates the transformed prosodic pattern for a portion between the prosodic patterns corresponding to the prosody changing points,

wherein assuming that a difference in pitch between adjacent moras or adjacent syllables of the speech data is Δ

P, the prosody changing point is a point where the Δ

P and an immediately following Δ

P are different in sign.

2. The prosody generation apparatus according to claim 1, wherein the representative prosodic patterns are pitch patterns.

3. The prosody generation apparatus according to claim 1, wherein the representative patterns are power patterns.

4. The prosody generation apparatus according to claim 3, wherein the power is (i) a value obtained by standardizing a power of a mora or a syllable for each type of phonology, or (ii) an amplitude value of a sound source waveform of a mora or a syllable.

5. The prosody generation apparatus according to claim 1, wherein the representative prosodic patterns are patterns generated for each of clusters into which patterns of the portions of the speech data including the prosodic changing points are clustered by means of a statistical technique.

6. The prosody generation apparatus according to claim 1, wherein the prosody changing point includes at least one of a beginning of an accent phrase, an ending of an accent phrase and an accent nucleus.

7. The prosody generation apparatus according to claim 1, wherein the prosody changing point setting unit sets the prosody changing point using at least one of the received phonological information and linguistic information, according to a prosody changing point extraction rule predetermined based on attributes concerning the phonology and attributes concerning the linguistic information of the prosody changing point of the speech data.

8. The prosody generation apparatus according to claim 1, wherein the attributes concerning phonology includes one or more of the following attributes:

(1) the number of phonemes, the number of moras, the number of syllables, an accent position, an accent type, an accent strength, a stress pattern or a stress strength of an accent phrase, a clause, a stress phrase, or a word;

(2) the number of moras, the number of syllables or the number of phonemes counted from a beginning of a sentence, a phrase, an accent phrase, a clause, or a word;

(3) the number of moras, the number of syllables, or the number of phonemes counted from an ending of a sentence, a phrase, an accent phrase, a clause, or a word;

(4) the presence or absence of adjacent pauses;

(5) a time length of adjacent pauses;

(6) a time length of a pause located before and the nearest to the prosody changing point;

(7) a time length of a pause located after and the nearest to the prosody changing point;

(8) the number of moras, the number of syllables or the number of phonemes counted from a pause located before and the nearest to the prosody changing point;

(9) the number of moras, the number of syllables or the number of phonemes counted from a pause located after and the nearest to the prosody changing point; and

(10) the number of moras, the number of syllables or the number of phonemes counted from an accent nucleus or a stress position.

9. The prosody generation apparatus according to claim 1, wherein the attributes concerning linguistic information includes one or more of the following attributes:

a part of speech, an attribute concerning a modification structure, a distance to a modifiee, a distance to a modifier, an attribute concerning syntax, prominence, emphasis, or semantic classification of an accent phrase, a clause, a stress phrase, or a word.

10. The prosody generation apparatus according to claim 1, wherein the selection rule is obtained by formulating a relationship between (i) clusters corresponding to the representative patterns and into which prosodic patterns of the speech data are clustered and classified and (ii) attributes concerning phonology or attributes concerning linguistic information of each of the prosodic patterns, by means of a statistical technique or a learning technique so as to predict a cluster to which a prosodic pattern including the prosody changing point belongs, using at least one of the attributes concerning phonology and the attributes concerning linguistic information.

11. The prosody generation apparatus according to claim 1, wherein the transformation is a parallel shifting along a frequency axis of a pitch pattern.

12. The prosody generation apparatus according to claim 1, wherein the transformation is a parallel shifting along a logarithmic axis of a frequency of a pitch pattern.

13. The prosody generation apparatus according to claim 1, wherein the transformation is a parallel shifting along an amplitude axis of a power pattern.

14. The prosody generation apparatus according to claim 1, wherein the transformation is a parallel shifting along a power axis of a power pattern.

15. The prosody generation apparatus according to any claim 1, wherein the transformation is compression or extension in a dynamic range on a frequency axis of a pitch pattern.

16. The prosody generation apparatus according to claim 1, wherein the transformation is compression or extension in a dynamic range on a logarithmic axis of a pitch pattern.

17. The prosody generation apparatus according to claim 1, wherein the transformation is compression or extension in a dynamic range on an amplitude axis of a power pattern.

18. The prosody generation apparatus according to claim 1, wherein the transformation is compression or extension in a dynamic range on a power axis of a power pattern.

19. The prosody generation apparatus according to claim 1, wherein the interpolation is a linear interpolation, by means of a spline function, or by means of a sigmoid curve.

20. A prosody generation apparatus that receives phonological information and linguistic information so as to generate prosody, the prosody generation apparatus being operable to refer to (a) a representative prosodic pattern storage unit for accumulating beforehand representative prosodic patterns of portions of speech data, the portions including prosody changing points;

(b) a selection rule storage unit that stores a selection rule predetermined according to attributes concerning phonology or attributes concerning linguistic information of the portions of the speech data including the prosody changing points; and

(c) a transformation rule storage unit that stores a transformation rule predetermined according to attributes concerning the phonology or the linguistic information of the portions of the speech data including the prosody changing points;

the prosody generation apparatus comprising a computer processing unit and a memory storing a program that are configured to implement;

a prosody changing point setting unit that sets a prosody changing point according to at least any one of the received phonological information and the linguistic information;

a pattern selection unit that selects a representative prosodic pattern from the representative prosodic pattern storage unit according to the selection rule, based on the received phonological information and the linguistic information; and

a prosody generation unit that transforms the representative prosodic pattern selected by the pattern selection unit according to the transformation rule and interpolates the transformed prosodic pattern for a portion between the prosodic patterns corresponding to the prosody changing points,

wherein the prosody changing point setting unit sets the prosody changing point using at least one of the received phonological information and linguistic information, according to a prosody changing point extraction rule predetermined based on attributes concerning the phonology and attributes concerning the linguistic information of the prosody changing point of the speech data, and

wherein the prosody changing point extraction rule is obtained by formulating a relationship between (i) a classification as to whether adjacent moras or syllables of the speech data are a prosody changing point or not and (ii) attributes concerning phonology or attributes concerning linguistic information of the adjacent moras or syllables, by means of a statistical technique or a learning technique so as to predict whether a point is a prosody changing point or not using at least one of the attributes concerning phonology and the attributes concerning linguistic information.

21. The prosody generation apparatus according to claim 20, wherein the statistical technique is a multivariate analysis, a decision tree, or the Quantification Theory Type II where a type of a cluster is designated as a criterion variable.

22. A prosody generation apparatus that receives phonological information and linguistic information so as to generate prosody, the prosody generation apparatus being operable to refer to (a) a representative prosodic pattern storage unit for accumulating beforehand representative prosodic patterns of portions of speech data, the portions including prosody changing points;

(b) a selection rule storage unit that stores a selection rule predetermined according to attributes concerning phonology or attributes concerning linguistic information of the portions of the speech data including the prosody changing points; and

(c) a transformation rule storage unit that stores a transformation rule predetermined according to attributes concerning the phonology or the linguistic information of the portions of the speech data including the prosody changing points;

the prosody generation apparatus comprising a computer processing unit and a memory storing a program that are configured to implement;

a prosody changing point setting unit that sets a prosody changing point according to at least any one of the received phonological information and the linguistic information;

a pattern selection unit that selects a representative prosodic pattern from the representative prosodic pattern storage unit according to the selection rule, based on the received phonological information and the linguistic information; and

a prosody generation unit that transforms the representative prosodic pattern selected by the pattern selection unit according to the transformation rule and interpolates the transformed prosodic pattern for a portion between the prosodic patterns corresponding to the prosody changing points,

wherein the transformation rule is obtained by clustering prosodic patterns of the speech data into clusters corresponding to the representative patterns so as to produce a representative pattern for each cluster and by formulating a relationship between (i) a distance between each of the prosodic patterns and a representative pattern of a cluster to which the prosodic pattern belongs and (ii) attributes concerning phonology or attributes concerning linguistic information of the prosodic pattern, by means of a statistical technique or a learning technique so as to estimate an amount of transformation of the selected prosodic pattern, using at least one of the attributes concerning phonology and the attributes concerning linguistic information.

23. The prosody generation apparatus according to claim 22, wherein the amount of transformation is one of a shifting amount, a compression rate in a dynamic range and an extension rate in a dynamic range.

24. The prosody generation apparatus according to claim 22, wherein the statistical technique is the Quantification Theory Type I where the shifting amount of a representative prosodic pattern is designated as a criterion variable.

25. The prosody generation apparatus according to claim 22, wherein the statistical technique is the Quantification Theory Type I where a compression rate or an extension rate in a dynamic range of a representative prosodic pattern of a cluster is designated as a criterion variable.

26. The prosody generation apparatus according to claim 22, wherein the statistical technique is the Quantification Theory Type I where a distance between a representative prosodic pattern in a cluster and each prosodic data is designated as a criterion variable.