Artificial intelligence-based text-to-speech system and method

• US 10,319,364 B2
• Filed: 05/17/2018
• Issued: 06/11/2019
• Est. Priority Date: 05/18/2017
• Status: Active Grant

• Current Assignee: Telepathy Labs, Inc.
• Original Assignee: Telepathy Labs, Inc.
• Inventors: Reber, Martin, Avijeet, Vijeta
• Primary Examiner(s): Jackson, Jakieda R
• Application Number:
  US15/982,326
• Publication Number:
  US 20180336881A1
• Time in Patent Office: 390 Days
• Field of Search: 7042001, 704232, 704259
• CPC Class Codes: G06F 18/10 Pre-processing; Data cleansing
• CPC Class Codes: G06F 18/2135 based on approximation criteria, e.g. principal component analysis
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 3/042 Knowledge-based neural networks; Logical representations of neural networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G10L 13/04 Details of speech synthesis systems, e.g. synthesiser structure or memory management
• CPC Class Codes: G10L 13/08 Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
• CPC Class Codes: G10L 19/00 Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis in musical instruments G10H

Abstract

A technique improves training and speech quality of a text-to-speech (TTS) system having an artificial intelligence, such as a neural network. The TTS system is organized as a front-end subsystem and a back-end subsystem. The front-end subsystem is configured to provide analysis and conversion of text into input vectors, each having at least a base frequency, f₀, a phenome duration, and a phoneme sequence that is processed by a signal generation unit of the back-end subsystem. The signal generation unit includes the neural network interacting with a pre-existing knowledgebase of phenomes to generate audible speech from the input vectors. The technique applies an error signal from the neural network to correct imperfections of the pre-existing knowledgebase of phenomes to generate audible speech signals. Speech signal specific modelling techniques in combination with applied psychoacoustic principles drive training efficiency of neural networks with positive impact on quality of generated speech signals.

Claims

1. A text-to-speech (TTS) system comprising:

a front-end subsystem configured to provide analysis and conversion of text into an input vector having a base frequency for a phoneme, a phenome duration, and a phoneme sequence; and

a back-end subsystem coupled to the front-end subsystem and configured to convert the input vector of the base frequency, the phoneme duration and the phoneme sequence into an intermediate vector for processing by a signal generation unit of the back-end subsystem, the signal generation unit having a neural network interacting with a pre-existing knowledgebase of phonemes, wherein the signal generation unit is configured to use the neural network interacting with the pre-existing knowledgebase of phonemes to apply an error signal to correct for speech signal distortions of the pre-existing knowledgebase of phonemes to generate the speech signal.

2. The TTS system of claim 1 further comprising a transformation unit that converts a frequency domain signal combined from the neural network and the pre-existing knowledgebase into the speech signal.

3. The TTS system of claim 1 wherein the pre-existing knowledgebase of phonemes comprises average basic acoustic signal data of how a speaker speaks derived from averaging of hours of recorded audible speech.

4. The TTS system of claim 1 wherein the neural network is configured to correct for psychoacoustic perceived speech signal distortions of the pre-existing knowledgebase of phonemes.

5. The TTS system of claim 1 wherein the back-end subsystem is further configured to upsample a frequency of the input vector provided to the neural network and the pre-existing knowledgebase to another frequency of the intermediate vector.

6. The TTS system of claim 5 wherein the upsampling unit includes a pitch normalization to normalize a pitch length of the input vector, and wherein the back-end subsystem includes an inverse pitch normalization unit to normalize the speech signal from the transformation unit.

7. The TTS system of claim 1 wherein the neural network is configured to correct errors of voiced phonemes of the pre-existing knowledgebase of phonemes based on principal component analysis.

8. The TTS system of claim 7 wherein the principal component analysis is based on lossy modelling.

9. The TTS system of claim 7 wherein the back-end subsystem includes another neural network configured to correct errors of unvoiced phonemes of the pre-existing knowledgebase of phonemes based on noise band modelling.

10. The TTS system of claim 1 wherein the neural network is configured based on psychoacoustic modeling of phonemes.

11. A method of processing text-to-speech comprising:

receiving an input vector having a base frequency for a phoneme, a phenome duration for the phoneme, and a phoneme sequence;

upsampling the input vector of the base frequency, the phoneme duration and the phoneme sequence into an intermediate vector;

generating a speech signal from the intermediate vector using a pre-existing knowledgebase of phonemes; and

applying an error signal from a neural network to correct for speech signal distortions of the speech signal based on an interaction between the neural network and the pre-existing knowledgebase.

12. The method of processing text-to-speech of claim 11 further comprising converting a frequency domain signal combined from the error signal and the generated speech signal of the pre-existing knowledgebase into a time domain speech signal.

13. The method of processing text-to-speech of claim 11 wherein the pre-existing knowledgebase of phonemes comprises average basic acoustic signal data of how a speaker speaks derived from averaging of hours of recorded audible speech.

14. The method of processing text-to-speech of claim 11 wherein the neural network is configured to correct for psychoacoustic perceived speech signal distortions of the pre-existing knowledgebase of phonemes.

15. The method of processing text-to-speech of claim 11 further comprising applying an output of another neural network to correct for speech signal distortions of unvoiced phonemes of the speech signal based on noise band modelling.

16. The method of processing text-to-speech of claim 11 wherein the upsampling further includes normalizing a pitch length of the input vector, and where the time domain signal is normalized using inverse pitch normalization.

17. The method of processing text-to-speech of claim 11 wherein the neural network configured to correct the signal distortions of the speech signal for voiced phonemes of the pre-existing knowledgebase of phonemes based on principal component analysis.

18. The method of processing text-to-speech of claim 17 wherein the principal component analysis is based on lossy modelling.

19. The method of processing text-to-speech of claim 11 wherein the neural network is configured based on psychoacoustic modeling of phonemes.

20. A non-transitory computer-readable medium having program instructions which, when executed across one or more processors, causes at least a portion of the one or more processors to perform operations comprising:

receiving an input vector having a base frequency for a phoneme, a phenome duration for the phoneme, and a phoneme sequence;

upsampling the input vector of the base frequency, the phoneme duration and the phoneme sequence into an intermediate vector;

generating a speech signal from the intermediate vector using a pre-existing knowledgebase of phonemes; and

applying an error signal from a neural network to correct for speech signal distortions of the speech signal based on interactions between the pre-existing knowledgebase.