PHONETIC POSTERIORGRAMS FOR MANY-TO-ONE VOICE CONVERSION

• US 20180012613A1
• Filed: 06/09/2017
• Published: 01/11/2018
• Est. Priority Date: 07/11/2016
• Status: Active Grant

• Current Assignee: Chinese University of Hong Kong
• Original Assignee: Chinese University of Hong Kong
• Inventors: Sun, Lifa, Li, Kun, Wang, Hao, Kang, Shiyin, Meng, Mei Ling Helen
• Granted Patent: US 10,176,819 B2
• Time in Patent Office: Days
• CPC Class Codes: G10L 13/033 Voice editing, e.g. manipulating the voice of the synthesiser
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/16 using artificial neural networks
• CPC Class Codes: G10L 15/187 Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
• CPC Class Codes: G10L 2021/0135 Voice conversion or morphing
• CPC Class Codes: G10L 21/007 characterised by the process used
• CPC Class Codes: G10L 25/24 the extracted parameters being the cepstrum

Abstract

A method for converting speech using phonetic posteriorgrams (PPGs). A target speech is obtained and a PPG is generated based on acoustic features of the target speech. Generating the PPG may include using a speaker-independent automatic speech recognition (SI-ASR) system for equalizing different speakers. The PPG includes a set of values corresponding to a range of times and a range of phonetic classes, the phonetic classes corresponding to senones. A mapping between the PPG and one or more segments of the target speech is generated. A source speech is obtained, and the source speech are converted into a converted speech based on the PPG and the mapping.

Claims

1. A computer-implemented method comprising:

obtaining a target speech;

obtaining a source speech;

generating a phonetic posteriorgram (PPG) based on acoustic features of the target speech, the PPG including a set of values corresponding to a range of times and a range of phonetic classes;

generating a mapping between the PPG and the acoustic features of the target speech; and

converting the source speech into a converted speech based on the PPG and the mapping.

2. The computer-implemented method of claim 1, wherein the range of phonetic classes correspond to a range of senones.

3. The computer-implemented method of claim 1, wherein the set of values correspond to posterior probabilities of each of the range of phonetic classes for each of the range of times, and wherein the PPG comprises a matrix.

4. The computer-implemented method of claim 1, wherein the source speech is different than the target speech.

5. The computer-implemented method of claim 1, wherein generating the PPG includes using a speaker-independent automatic speech recognition (SI-ASR) system for equalizing different speakers.

6. The computer-implemented method of claim 5, wherein the SI-ASR system is trained for PPGs generation using a multi-speaker ASR corpus, an input being an MFCC feature vector of t^th frame, denoted as X_t, and an output being a vector of posterior probabilities defined by P_t=(p(s|X_t)|s=1, 2, . . . , C) where p(s|X_t) is the posterior probability of each phonetic class s.

7. The computer-implemented method of claim 5, wherein generating the mapping between the PPG and the acoustic features of the target speech includes using a bidirectional long short-term memory based recurrent neural network (DBLSTM) structure to model the mapping, and wherein the mapping is between the PPG and corresponding acoustic features of the target speech.

8. The computer-implemented method of claim 7, wherein converting the source speech into the converted speech includes converting the PPG into one or more mel-cepstral coefficients (MCEPs) using a trained DBLSTM model.

9. A non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to perform operations including:

obtaining a target speech;

obtaining a source speech;

generating a phonetic posteriorgram (PPG) based on acoustic features of the target speech, the PPG including a set of values corresponding to a range of times and a range of phonetic classes;

generating a mapping between the PPG and the acoustic features of the target speech; and

converting the source speech into a converted speech based on the PPG and the mapping.

10. The non-transitory computer-readable medium of claim 9, wherein the range of phonetic classes correspond to a range of senones.

11. The non-transitory computer-readable medium of claim 9, wherein the set of values correspond to posterior probabilities of each of the range of phonetic classes for each of the range of times, and wherein the PPG comprises a matrix.

12. The non-transitory computer-readable medium of claim 9, wherein the source speech is different than the target speech.

13. The non-transitory computer-readable medium of claim 9, wherein generating the PPG includes using a speaker-independent automatic speech recognition (SI-ASR) system for equalizing different speakers.

14. The non-transitory computer-readable medium of claim 13, wherein generating the mapping between the PPG and the acoustic features of the target speech includes using a bidirectional long short-term memory based recurrent neural network (DBLSTM) structure to model the mapping, and wherein the mapping is between the PPG and corresponding acoustic features of the target speech.

15. The non-transitory computer-readable medium of claim 14, wherein converting the source speech into the converted speech includes converting the PPG into one or more mel-cepstral coefficients (MCEPs) using a trained DBLSTM model.

16. A system comprising:

a processor;

a computer-readable medium in data communication with the processor, the computer-readable medium comprising instructions that, when executed by the processor, cause the processor to perform operations including;

obtaining a target speech;

obtaining a source speech;

generating a phonetic posteriorgram (PPG) based on acoustic features of the target speech, the PPG including a set of values corresponding to a range of times and a range of phonetic classes;

generating a mapping between the PPG and the acoustic features of the target speech; and

converting the source speech into a converted speech based on the PPG and the mapping.

17. The system of claim 16, wherein the range of phonetic classes correspond to a range of senones.

18. The system of claim 16, wherein the set of values correspond to posterior probabilities of each of the range of phonetic classes for each of the range of times, and wherein the PPG comprises a matrix.

19. The system of claim 16, wherein generating the PPG includes using a speaker-independent automatic speech recognition (SI-ASR) system for equalizing different speakers.

20. The system of claim 19, wherein generating the mapping between the PPG and the acoustic features of the target speech includes using a bidirectional long short-term memory based recurrent neural network (DBLSTM) structure to model the mapping, and wherein the mapping is between the PPG and corresponding acoustic features of the target speech.