Voice quality conversion device and voice quality conversion method for converting voice quality of an input speech using target vocal tract information and received vocal tract information corresponding to the input speech

• US 8,898,055 B2
• Filed: 05/08/2008
• Issued: 11/25/2014
• Est. Priority Date: 05/14/2007
• Status: Expired due to Fees

• Current Assignee: Sovereign Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Panasonic Intellectual Property Corporation of America (Panasonic Holdings Corporation)
• Inventors: Hirose, Yoshifumi, Kamai, Takahiro, Kato, Yumiko
• Primary Examiner(s): Desir, Pierre-Louis
• Assistant Examiner(s): KOVACEK, DAVID M
• Application Number:
  US12/307,021
• Publication Number:
  US 20090281807A1
• Time in Patent Office: 2,392 Days
• Field of Search: 704200-203, 704231-242, 704246-257, 704258-260, 704/261, 704262-269, 704/278, 704E17001-E17016, 704E19001-E19049, 704E13001-E1304
• US Class Current: 704/203
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units
• CPC Class Codes: G10L 2021/0135 Voice conversion or morphing
• CPC Class Codes: G10L 21/00 Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility G10L19/00 takes precedence
• CPC Class Codes: G10L 21/003 Changing voice quality, e.g. pitch or formants

Abstract

A voice quality conversion device including: a target vowel vocal tract information hold unit holding target vowel vocal tract information of each vowel indicating target voice quality; a vowel conversion unit (i) receiving vocal tract information with phoneme boundary information of the speech including information of phonemes and phoneme durations, (ii) approximating a temporal change of vocal tract information of a vowel in the vocal tract information with phoneme boundary information applying a first function, (iii) approximating a temporal change of vocal tract information of the same vowel held in the target vowel vocal tract information hold unit applying a second function, (iv) calculating a third function by combining the first function with the second function, and (v) converting the vocal tract information of the vowel applying the third function; and a synthesis unit synthesizing a speech using the converted information.

Claims

1. A voice quality conversion device that converts voice quality of an input speech using information corresponding to the input speech, said voice quality conversion device comprising:

a target vowel vocal tract information hold unit configured to hold target vowel vocal tract information of each vowel, the target vowel vocal tract information indicating target voice quality;

a vowel conversion unit configured to

(i) receive vocal tract information with phoneme boundary information which is vocal tract information that corresponds to the input speech and that is added with information of (1) a phoneme in the input speech and (2) a duration of the phoneme,

(ii) approximate, as a first polynomial expression, a temporal change of received vocal tract information of a vowel included in the received vocal tract information with phoneme boundary information,

(iii) approximate, as a second polynomial expression, a temporal change of target vocal tract information of the vowel, the target vocal tract information being included in the target vowel vocal tract information held in said target vowel vocal tract information hold unit,

(iv) approximate, as a third polynomial expression, interpolated vocal tract information of the vowel by combining (1) the first polynomial expression approximating the temporal change of the received vocal tract information of the vowel with (2) the second polynomial expression approximating the temporal change of the target vocal tract information of the vowel, and

(v) convert the received vocal tract information of the vowel using the third polynomial expression approximating the interpolated vocal tract information of the vowel; and

a synthesis unit configured to synthesize a speech using the converted vocal tract information of the vowel converted by said vowel conversion unit,

wherein (i) the first polynomial expression approximates a change in the received vocal tract information of the vowel over time, (ii) the second polynomial expression approximates a change in the target vocal tract information of the vowel over time, and (iii) the third polynomial expression approximates a change in the interpolated vocal tract information of the vowel over time,

wherein the first polynomial expression approximating the temporal change of the received vocal tract information of the vowel and the second polynomial expression approximating the temporal change of the target vocal tract information of the vowel have a same time period that overlaps over the entire time period of the vowel, and

wherein said vowel conversion unit is configured to generate the third polynomial expression by adding the first polynomial expression with the second polynomial expression based on a predetermined conversion ratio.

2. The voice quality conversion device according to claim 1, further comprising

a consonant vocal tract information derivation unit configured to (i) receive the vocal tract information with phoneme boundary information, and (ii) derive vocal tract information of each consonant held in the vocal tract information with phoneme boundary information, from pieces of vocal tract information of consonants having voice quality which is not the target voice quality,

wherein said synthesis unit is configured to synthesize the speech using (i) the converted vocal tract information for the vowel converted by said vowel conversion unit and (ii) the derived vocal tract information of each consonant that is derived by said consonant vocal tract information derivation unit.

3. The voice quality conversion device according to claim 2,

wherein said consonant vocal tract information derivation unit includes:

a consonant vocal tract information hold unit configured to hold, for each consonant held in the vocal tract information with phoneme boundary information, pieces of vocal tract information extracted from speeches of a plurality of speakers; and

a consonant selection unit configured to (i) receive the vocal tract information with phoneme boundary information, and (ii) select vocal tract information of a consonant held in the vocal tract information with phoneme boundary information, from among the pieces of vocal tract information for each consonant held in the vocal tract information with phoneme boundary information, the selected vocal tract information being suitable for vocal tract information converted by said vowel conversion unit for a vowel positioned at a vowel section prior or subsequent to the consonant.

4. The voice quality conversion device according to claim 3,

wherein said consonant selection unit is configured to select the vocal tract information of the consonant based on continuity between a value of the selected vocal tract information and a value of the vocal tract information converted by said vowel conversion unit for the vowel positioned at the vowel section prior to or subsequent to the each consonant.

5. The voice quality conversion device according to claim 3, further comprising

a consonant transformation unit configured to transform the vocal tract information of the consonant selected by said consonant selection unit so as to improve continuity between a value of the selected vocal tract information and a value of the vocal tract information converted by said vowel conversion unit for the vowel positioned at the vowel section prior to or subsequent to the consonant.

6. The voice quality conversion device according to claim 1, further comprising

a conversion ratio receiving unit configured to receive a conversion ratio representing a degree of conversion to the target voice quality,

wherein said vowel conversion unit is configured to

(i) receive the conversion ratio received by said conversion ratio unit and

(ii) approximate, as the third polynomial expression, the interpolated vocal tract information by combining (1) the first polynomial expression approximating the temporal change of the received vocal tract information of the vowel with (2) the second polynomial expression approximating the temporal change of the target vocal tract information of the vowel at the conversion ratio received by said conversion ratio receiving unit.

7. The voice quality conversion device according to claim 6,

wherein said vowel conversion unit is configured to:

(i) approximate, for each order of the first polynomial expression, the temporal change of the received vocal tract information of the vowel included in the received vocal tract information with phoneme boundary information,

(ii) approximate, for each order of the second polynomial expression, a temporal change of target vocal tract information of the vowel, the target vocal tract information being included in the target vowel vocal tract information held in said target vowel vocal tract information hold unit,

(iii) approximate, for each order of the third polynomial expression, the interpolated vocal tract information by combining (1) the first polynomial expression approximating the temporal change of the received vocal tract information of the vowel with (2) the second polynomial expression approximating the temporal change of the target vocal tract information of the vowel at the conversion ratio.

8. The voice quality conversion device according to claim 1,

wherein said vowel conversion unit is further configured to interpolate vocal tract information of a first vowel and vocal tract information of a second vowel to be continuously connected to each other at a vowel boundary, the vocal tract information of the first vowel and the vocal tract information of the second vowel being included in a glide section that is a predetermined time period including the vowel boundary which is a temporal boundary between the vocal tract information of the first vowel and the vocal tract information of the second vowel.

9. The voice quality conversion device according to claim 8,

wherein the predetermined time period is set to be longer as a duration of the first vowel and the second vowel which are positioned prior and subsequent to the vowel boundary is longer.

10. The voice quality conversion device according to claim 1,

wherein the vocal tract information is one of a Partial Auto Correlation (PARCOR) coefficient and a reflection coefficient of a vocal tract acoustic tube model.

11. The voice quality conversion device according to claim 10,

wherein each of the PARCOR coefficient and the reflection coefficient of the vocal tract acoustic tube model is calculated according to a polynomial expression of an all-pole model which is generated by applying Linear Predictive Coding (LPC) analysis to the input speech.

12. The voice quality conversion device according to claim 10,

wherein each of the PARCOR coefficient and the reflection coefficient of the vocal tract acoustic tube model is calculated according to a polynomial expression of an all-pole model which is generated by applying Autoregressive Exogenous (ARX) analysis to the input speech.

13. The voice quality conversion device according to claim 1,

wherein the vocal tract information with phoneme boundary information is generated from a synthetic speech generated from a text.

14. The voice quality conversion device according to claim 1, further comprising:

a stable vowel section extraction unit configured to detect a stable vowel section from a speech having the target voice quality; and

a target vocal tract information generation unit configured to extract, from the stable vowel section, the vocal tract information as the target vowel vocal tract information,

wherein said target vowel vocal tract information hold unit is configured to hold the target vowel vocal tract information that is generated by said stable vowel extraction unit and said target vocal tract information generation unit.

15. The voice quality conversion device according to claim 14,

wherein said stable vowel section extraction unit includes:

a phoneme recognition unit configured to recognize a phoneme in the speech having the target voice quality; and

a stable section extraction unit configured to extract, as the stable vowel section, a vowel section having a likelihood greater than a threshold value from vowel sections in the phonemes recognized by said phoneme recognition unit, the likelihood being determined by the recognition of said phoneme recognition unit.

16. The voice quality conversion device according to claim 1, wherein said target vowel vocal tract information hold unit is configured to only hold the target vowel vocal tract information for each vowel.

17. The voice quality conversion device according to claim 1, wherein the vocal tract information is one of a reflection coefficient, a linear prediction coefficient, and a line spectrum pairs coefficient.

18. A voice quality conversion method of converting voice quality of an input speech using information corresponding to the input speech, said voice quality conversion method comprising:

receiving vocal tract information with phoneme boundary information which is vocal tract information that corresponds to the input speech and that is added with information of (i) a phoneme in the input speech and (ii) a duration of the phoneme;

approximating, as a first polynomial expression, a temporal change of received vocal tract information of a vowel included in the received vocal tract information with phoneme boundary information;

approximating, as a second polynomial expression, a temporal change of target vocal tract information of the vowel, the target vocal tract information of the vowel indicating target voice quality;

approximating, as a third polynomial expression, interpolated vocal tract information of the vowel by combining (i) the first polynomial expression

approximating the temporal change of the received vocal tract information of the vowel with (ii) the second polynomial expression approximating the temporal change of the target vocal tract information of the vowel;

converting the received vocal tract information of the vowel using the third polynomial expression approximating the interpolated vocal tract information of the vowel; and

synthesizing a speech using the converted vocal tract information of the vowel converted in said converting,

wherein (i) the first polynomial expression approximates a change in the received vocal tract information of the vowel over time, (ii) the second polynomial expression approximates a change in the target vocal tract information of the vowel over time, and (iii) the third polynomial expression approximates a change in the interpolated vocal tract information of the vowel over time,

wherein the first polynomial expression approximating the temporal change of the received vocal tract information of the vowel and the second polynomial expression approximating the temporal change of the target vocal tract information of the vowel have a same time period that overlaps over the entire time period of the vowel, and

wherein said approximating, as the third polynomial expression, the interpolated vocal tract information of the vowel includes generating the third polynomial expression by adding the first polynomial expression with the second polynomial expression based on a predetermined conversion ratio.

19. A non-transitory computer readable recording medium having stored thereon a program for converting voice quality of an input speech using information corresponding to the input speech, wherein, when executed by a computer, said program causes the computer to perform a method comprising:

receiving vocal tract information with phoneme boundary information which is vocal tract information that corresponds to the input speech and that is added with information of (i) a phoneme in the input speech and (ii) a duration of the phoneme;

approximating, as a first polynomial expression, a temporal change of received vocal tract information of a vowel included in the received vocal tract information with phoneme boundary information;

approximating, as a second polynomial expression, a temporal change of target vocal tract information of the vowel, the target vocal tract information of the vowel indicating target voice quality;

approximating, as a third polynomial expression, interpolated vocal tract information by combining (i) the first polynomial expression approximating the temporal change of the received vocal tract information of the vowel with (ii) the second polynomial expression approximating the temporal change of the target vocal tract information of the vowel; and

converting the received vocal tract information of the vowel using the third polynomial expression approximating the interpolated vocal tract information of the vowel; and

synthesizing a speech using the converted vocal tract information of the vowel converted in said converting,

wherein (i) the first polynomial expression approximates a change in the received vocal tract information of the vowel over time, (ii) the second polynomial expression approximates a change in the target vocal tract information of the vowel over time, and (iii) the third polynomial expression approximates a change in the interpolated vocal tract information of the vowel over time,

wherein the first polynomial expression approximating the temporal change of the received vocal tract information of the vowel and the second polynomial expression approximating the temporal change of the target vocal tract information of the vowel have a same time period that overlaps over the entire time period of the vowel, and

wherein said approximating, as the third polynomial expression, the interpolated vocal tract information of the vowel includes generating the third polynomial expression by adding the first polynomial expression with the second polynomial expression based on a predetermined conversion ratio.

20. A voice quality conversion system that converts voice quality of an original speech to be converted using information corresponding to the original speech, said voice quality conversion system comprising:

a server; and

a terminal connected to said server via a network,

wherein said server includes;

a target vowel vocal tract information hold unit configured to hold target vowel vocal tract information of each vowel, the target vowel vocal tract information indicating target voice quality;

a target vowel vocal tract information sending unit configured to send the target vowel vocal tract information held in said target vowel vocal tract information hold unit to said terminal via the network;

an original speech hold unit configured to hold original speech information that is information corresponding to the original speech; and

an original speech information sending unit configured to send the original speech information held in said original speech hold unit to said terminal via the network, wherein said terminal includes;

a target vowel vocal tract information receiving unit configured to receive the target vowel vocal tract information from said target vowel vocal tract information sending unit;

an original speech information receiving unit configured to receive the original speech information from said original speech information sending unit;

a vowel conversion unit configured to

(i) approximate, as a first polynomial expression, a temporal change of received vocal tract information of a vowel included in the received original speech information received by said original speech information receiving unit,

(ii) approximate a second polynomial expression, a temporal change of target vocal tract information for the vowel, the target vocal tract information for the vowel being included in the target vowel vocal tract information received by said target vowel vocal tract information receiving unit,

(iii) approximate, as a third polynomial expression, interpolated vocal tract information by combining (i) the first polynomial expression approximating the temporal change of the received vocal tract information of the vowel with (ii) the second polynomial expression approximating the temporal change of the target vocal tract information of the vowel, and

(iv) convert the vocal tract information of the vowel using the third polynomial expression approximating the interpolated vocal tract information; and

a synthesis unit configured to synthesize a speech using the converted vocal tract information of the vowel converted by said vowel conversion unit,

wherein (i) the first polynomial expression approximates a change in the received vocal tract information of the vowel over time, (ii) the second polynomial expression approximates a change in the target vocal tract information of the vowel over time, and (iii) the third polynomial expression approximates a change in the interpolated vocal tract information of the vowel over time,

wherein the first polynomial expression approximating the temporal change of the received vocal tract information of the vowel and the second polynomial expression approximating the temporal change of the target vocal tract information of the vowel have a same time period that overlaps over the entire time period of the vowel, and

wherein said vowel conversion unit is configured to generate the third polynomial expression by adding the first polynomial expression with the second polynomial expression based on a predetermined conversion ratio.

21. A voice quality conversion system that converts voice quality of an original speech to be converted using information corresponding to the original speech, said voice quality conversion system comprising:

a terminal; and

a server connected to said terminal via a network,

wherein said terminal includes;

a target vowel vocal tract information generation unit configured to generate target vowel vocal tract information of each vowel, the target vowel vocal tract information indicating target voice quality;

a target vowel vocal tract information sending unit configured to send the target vowel vocal tract information generated by said target vowel vocal tract information generation unit to said server via the network;

a voice quality conversion speech receiving unit configured to receive a speech with converted voice quality; and

a reproduction unit configured to reproduce the speech with the converted voice quality received by said voice quality conversion speech receiving unit, wherein said server includes;

an original speech hold unit configured to hold original speech information that is information corresponding to the original speech;

a target vowel vocal tract information receiving unit configured to receive the target vowel vocal tract information from said target vowel vocal tract information sending unit;

a vowel conversion unit configured to

(i) approximate, as a first polynomial expression, a temporal change of received vocal tract information of a vowel included in the original speech information held in said original speech information hold unit,

(ii) approximate, as a second polynomial expression, a temporal change of target vocal tract information of the vowel, the target vocal tract information being included in the target vowel vocal tract information received by said target vowel vocal tract information receiving unit,

(iii) approximate, as a third polynomial expression, interpolated vocal tract information by combining (i) the first polynomial expression approximating the temporal change of the received vocal tract information of the vowel with (ii) the second polynomial expression approximating the temporal change of the target vocal tract information of the vowel, and

(iv) convert the received vocal tract information of the vowel using the third polynomial expression approximating the interpolated vocal tract information;

a synthesis unit configured to synthesize a speech using the converted vocal tract information for the vowel converted by said vowel conversion unit; and

a synthetic speech sending unit configured to send, as the speech with the converted voice quality, the speech synthesized by said synthesis unit to said voice quality conversion speech receiving unit via the network,

wherein (i) the first polynomial expression approximates a change in the received vocal tract information of the vowel over time, (ii) the second polynomial expression approximates a change in the target vocal tract information of the vowel over time, and (iii) the third polynomial expression approximates a change in the interpolated vocal tract information of the vowel over time,

wherein the first polynomial expression approximating the temporal change of the received vocal tract information of the vowel and the second polynomial expression approximating the temporal change of the target vocal tract information of the vowel have a same time period that overlaps over the entire time period of the vowel, and

wherein said vowel conversion unit is configured to generate the third polynomial expression by adding the first polynomial expression with the second polynomial expression based on a predetermined conversion ratio.