Prosody modification device, prosody modification method, and recording medium storing prosody modification program

• US 8,433,573 B2
• Filed: 02/11/2008
• Issued: 04/30/2013
• Est. Priority Date: 03/20/2007
• Status: Active Grant

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: Murase, Kentaro, Katae, Nobuyuki
• Primary Examiner(s): GUERRA-ERAZO, EDGAR X
• Application Number:
  US12/029,316
• Publication Number:
  US 20080235025A1
• Time in Patent Office: 1,905 Days
• Field of Search: 704/260, 704/261, 704/258, 704/254, 704/267, 704/266, 704/268, 704/269, 704/259, 704/255, 704/257, 704/270, 704/231, 704/235, 704/270.1
• US Class Current: 704/260
• CPC Class Codes: G10L 13/033 Voice editing, e.g. manipulating the voice of the synthesiser
• CPC Class Codes: G10L 13/0335 Pitch control
• CPC Class Codes: G10L 21/003 Changing voice quality, e.g. pitch or formants

Abstract

A prosody modification device includes: a real voice prosody input part that receives real voice prosody information extracted from an utterance of a human; a regular prosody generating part that generates regular prosody information having a regular phoneme boundary that determines a boundary between phonemes and a regular phoneme length of a phoneme by using data representing a regular or statistical phoneme length in an utterance of a human with respect to a section including at least a phoneme or a phoneme string to be modified in the real voice prosody information; and a real voice prosody modification part that resets a real voice phoneme boundary by using the generated regular prosody information so that the real voice phoneme boundary and a real voice phoneme length of the phoneme or the phoneme string to be modified in the real voice prosody information are approximate to an actual phoneme boundary and an actual phoneme length of the utterance of the human, thereby modifying the real voice prosody information.

Claims

1. A prosody modification device comprising:

a real voice prosody input part that receives real voice prosody information extracted from an utterance of a human;

a modification section determining part that determines a modification section that includes the phoneme or the phoneme string which are to be modified in the real voice prosody information, based on a kind of a phoneme string of the real voice prosody information;

a regular prosody generating part that generates regular prosody information having a regular phoneme boundary that determines a boundary between phonemes and a regular phoneme length of a phoneme by using data representing a regular or statistical phoneme length in an utterance of a human with respect to the modification section; and

a real voice prosody modification part that resets a real voice phoneme boundary of the phoneme or the phoneme string to be modified in the real voice prosody information by using the regular prosody information generated by the regular prosody generating part so that the real voice phoneme boundary and a real voice phoneme length of the phoneme or the phoneme string to be modified in the real voice prosody information are approximate to an actual phoneme boundary and an actual phoneme length of the utterance of the human, thereby modifying the real voice prosody information.

2. The prosody modification device according to claim 1, wherein the real voice prosody modification part includes a phoneme boundary resetting part that resets the real voice phoneme boundary of the phoneme or the phoneme string to be modified in the real voice prosody information based on a ratio of the regular phoneme length of each phoneme determined by the regular phoneme boundary in the section of the phoneme or the phoneme string to be modified, thereby modifying the real voice prosody information.

3. The prosody modification device according to claim 1, wherein the real voice prosody modification part includes a phoneme boundary resetting part that resets the real voice phoneme boundary of the phoneme or the phoneme string to be modified in the real voice prosody information based on the regular phoneme length of each phoneme of the regular prosody information and a speech rate ratio as a ratio between a rate of speech of the real voice prosody information and a rate of speech of the regular prosody information in the section of the phoneme or the phoneme string to be modified, thereby modifying the real voice prosody information.

4. The prosody modification device according to claim 3, further comprising a speech rate ratio detecting part that calculates, in a speech rate calculation range composed of at least one or more phonemes or morae including the phoneme to be modified in the real voice prosody information, the rate of speech of the real voice prosody information for the phoneme to be modified based on a total sum of the real voice phoneme lengths of respective phonemes determined by the real voice phoneme boundary and the number of phonemes or morae in the speech rate calculation range, as well as the rate of speech of the regular prosody information for the phoneme to be modified based on a total sum of the regular phoneme lengths of the respective phonemes determined by the regular phoneme boundary and the number of phonemes or morae in the speech rate calculation range, and calculates the ratio between the rate of speech of the real voice prosody information and the rate of speech of the regular prosody information as the speech rate ratio,

wherein the phoneme boundary resetting part calculates a modified phoneme length based on the regular phoneme length of each of the phonemes of the regular prosody information and the speech rate ratio calculated by the speech rate ratio detecting part in the section of the phoneme or the phoneme string to be modified, and resets the real voice phoneme boundary of the real voice prosody information so that each real voice phoneme length in the section becomes the modified phoneme length, thereby modifying the real voice prosody information.

5. The prosody modification device according to claim 3, further comprising:

a phoneme length ratio calculating part that calculates a ratio between the real voice phoneme length of each phoneme determined by the real voice phoneme boundary and the regular phoneme length of the phoneme determined by the regular phoneme boundary as a phoneme length ratio of the phoneme in the section of the phoneme or the phoneme string to be modified in the real voice prosody information; and

a speech rate ratio calculating part that smoothes the phoneme length ratio calculated by the phoneme length ratio calculating part, thereby calculating the ratio between the rate of speech of the real voice prosody information and the rate of speech of the regular prosody information as the speech rate ratio,

wherein the phoneme boundary resetting part calculates a modified phoneme length based on the regular phoneme length of the phoneme of the regular prosody information and the speech rate ratio calculated by the speech rate ratio calculating part in the section of the phoneme or the phoneme string to be modified, and resets the real voice phoneme boundary of the real voice prosody information so that each real voice phoneme length in the section becomes the modified phoneme length, thereby modifying the real voice prosody information.

6. The prosody modification device according to claim 1, comprising:

a real voice prosody storing part that stores the real voice prosody information received by the real voice prosody input part or the real voice prosody information modified by the real voice prosody modification part; and

a convergence judging part that writes the real voice prosody information modified by the real voice prosody modification part in the real voice prosody storing part and instructs the real voice prosody modification part to modify the real voice prosody information when a difference between the real voice phoneme length of the real voice prosody information modified by the real voice prosody modification part and the real voice phoneme length of the unmodified real voice prosody information stored in the real voice prosody storing part is not less than a threshold value, as well as outputs the real voice prosody information modified by the real voice prosody modification part when the difference between the real voice phoneme length of the real voice prosody information modified by the real voice prosody modification part and the real voice phoneme length of the unmodified real voice prosody information stored in the real voice prosody storing part is less than the threshold value.

7. A Graphical User Interface device that allows the real voice prosody information modified by the prosody modification device according to claim 1 to be edited.

8. A speech synthesizer that outputs synthetic speech generated based on the real voice prosody information modified by the prosody modification device according to claim 1.

9. A speech synthesizer that outputs synthetic speech generated based on the real voice prosody information edited by the Graphical User Interface device according to claim 7.

10. A prosody modification method comprising:

a real voice prosody input operation in which a real voice prosody input part provided in a computer receives real voice prosody information extracted from an utterance of a human;

a modification section determining operation that determines a modification section that includes the phoneme or the phoneme string which are to be modified in the real voice prosody information, based on a kind of a phoneme string of the real voice prosody information;

a regular prosody generating operation in which a regular prosody generating part provided in the computer generates regular prosody information having a regular phoneme boundary that determines a boundary between phonemes and a regular phoneme length of a phoneme by using data representing a regular or statistical phoneme length in an utterance of a human with respect to the modification section; and

a real voice prosody modifying operation in which a real voice prosody modification part provided in the computer resets a real voice phoneme boundary of the phoneme or the phoneme string to be modified in the real voice prosody information by using the regular prosody information generated in the regular prosody generating operation so that the real voice phoneme boundary and a real voice phoneme length of the phoneme or the phoneme string to be modified in the real voice prosody information are approximate to an actual phoneme boundary and an actual phoneme length of the utterance of the human, thereby modifying the real voice prosody information.

11. A non-transitory recording medium storing a prosody modification program that allows a computer to execute:

a real voice prosody input process of receiving real voice prosody information extracted from an utterance of a human;

a modification section determination process of determining the section that includes the phoneme or the phoneme string which are to be modified in the real voice prosody information, based on a kind of a phoneme string of the real voice prosody information;

a regular prosody generation process of generating regular prosody information having a regular phoneme boundary that determines a boundary between phonemes and a regular phoneme length of a phoneme by using data representing a regular or statistical phoneme length in an utterance of a human with respect to the modification section; and

a real voice prosody modification process of resetting a real voice phoneme boundary of the phoneme or the phoneme string to be modified in the real voice prosody information by using the regular prosody information generated in the regular prosody generation process so that the real voice phoneme boundary and a real voice phoneme length of the phoneme or the phoneme string to be modified in the real voice prosody information are approximate to an actual phoneme boundary and an actual phoneme length of the utterance of the human, thereby modifying the real voice prosody information.