Hyper-structure recurrent neural networks for text-to-speech

• US 10,127,901 B2
• Filed: 06/13/2014
• Issued: 11/13/2018
• Est. Priority Date: 06/13/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Zhao, Pei, Leung, Max, Yao, Kaisheng, Yan, Bo, Zhao, Sheng, Alleva, Fileno A.
• Primary Examiner(s): Shah, Bharatkumar S
• Application Number:
  US14/303,969
• Publication Number:
  US 20150364128A1
• Time in Patent Office: 1,614 Days
• Field of Search: 704259
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G10L 13/08 Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
• CPC Class Codes: G10L 13/10 Prosody rules derived from text; Stress or intonation

Abstract

The technology relates to converting text to speech utilizing recurrent neural networks (RNNs). The recurrent neural networks may be implemented as multiple modules for determining properties of the text. In embodiments, a part-of-speech RNN module, letter-to-sound RNN module, a linguistic prosody tagger RNN module, and a context awareness and semantic mining RNN module may all be utilized. The properties from the RNN modules are processed by a hyper-structure RNN module that determine the phonetic properties of the input text based on the outputs of the other RNN modules. The hyper-structure RNN module may generate a generation sequence that is capable of being converting to audible speech by a speech synthesizer. The generation sequence may also be optimized by a global optimization module prior to being synthesized into audible speech.

Claims

1. A method for converting text to speech, the method comprising:

receiving text input into a plurality of first level recurrent neural networks;

determining, by a first recurrent neural network in the plurality of first level recurrent neural networks, one or more properties of the text input from the group consisting of;

part-of-speech properties, phonemes, linguistic prosody properties, contextual properties, and semantic properties;

determining, by a second recurrent neural network in the plurality of first level recurrent neural networks, one or more properties of the text input from the group consisting of;

part-of-speech properties, phonemes, linguistic prosody properties, contextual properties, and semantic properties, wherein the determined one or properties by the second recurrent neural network is different from the determined one or more properties by the first recurrent neural network;

receiving, by a recurrent neural network in a second level, the determined properties from the first recurrent neural network in the plurality of first level recurrent neural networks and the second recurrent neural network in the plurality of first level recurrent neural networks;

determining by the recurrent neural network in the second level, phonetic properties for the text input based on the properties received from the first recurrent neural network in the plurality of first level recurrent neural networks and the second neural network in the plurality of first level recurrent neural networks, wherein the recurrent neural network in the second level is different from the first recurrent neural network in the plurality of first level recurrent neural networks and the second recurrent neural network in the plurality of first level recurrent neural networks; and

based on the determined phonetic properties, generating a generation sequence for synthetization by an audio synthesizer.

2. The method of claim 1, wherein the one or more properties received are the part-of-speech properties and phonemes.

3. The method of claim 1, wherein the one or more properties received are the linguistic prosody properties, the contextual properties, and the semantic properties.

4. The method of claim 1, wherein the one or more properties received are the phonemes, the contextual properties, and the semantic properties.

5. The method of claim 1, further comprising optimizing the generation sequence.

6. The method of claim 1, further comprising synthesizing the generation sequence into audible speech.

7. The method of claim 1, wherein the one or more properties are received as a dense auxiliary input.

8. The method of claim 1, wherein the text input and the one or more properties are received as a dense auxiliary input.

9. The method of claim 1, wherein the recurrent neural network in the second level is a part of a hyper-structure module.

10. The method of claim 1, wherein the one or more properties are received by a hidden layer and an output layer of the recurrent neural network in the second level.

11. A computer storage device, having computer-executable instructions that, when executed by at least one processor, perform a method for converting text-to-speech, the method comprising:

receiving text input into a plurality of first level recurrent neural networks;

determining, by a first recurrent neural network in the plurality of first level recurrent neural networks, one or more properties of the text input from the group consisting of;

part-of-speech properties, phonemes, linguistic prosody properties, contextual properties, and semantic properties;

determining, by a second recurrent neural network in the plurality of first level recurrent neural networks, one or more properties of the text input from the group consisting of;

part-of-speech properties, phonemes, linguistic prosody properties, contextual properties, and semantic properties, wherein the determined one or properties by the second recurrent neural network is different from the determined one or more properties by the first recurrent neural network;

receiving, by a recurrent neural network in a second level, the determined properties from the first recurrent neural network in the plurality of first level recurrent neural networks and the second recurrent neural networks in the plurality of first level recurrent neural networks;

determining by the recurrent neural network in the second level, phonetic properties for the text input based on the properties received from the first recurrent neural network in the plurality of first level recurrent neural networks and the second neural network in the plurality of first level recurrent neural networks, wherein the recurrent neural network in the second level is different from the first recurrent neural network in the plurality of first level recurrent neural networks and the second recurrent neural network in the plurality of first level recurrent neural networks; and

based on the determined phonetic properties, generating a generation sequence for synthetization by an audio synthesizer.

12. The computer storage device of claim 11, wherein the one or more properties received are the part-of-speech properties and phonemes.

13. The computer storage device of claim 11, wherein the one or more properties received are the phonemes, the contextual properties, and the semantic properties.

14. The computer storage device of claim 11, wherein the method further comprises optimizing the generation sequence.

15. The computer storage device of claim 11, wherein the method further comprises synthesizing the generation sequence into audible speech.

16. The computer storage device of claim 11, wherein the one or more properties are received as a dense auxiliary input.

17. The computer storage device of claim 11, wherein the text input and the one or more properties are received as a dense auxiliary input.

18. The computer storage device of claim 11, wherein the recurrent neural network in the second level is a part of a hyper-structure module.

19. The computer storage device of claim 11, wherein the one or more properties are received by a hidden layer and an output layer of the recurrent neural network in the second level.

20. A system for converting text-to-speech comprising:

at least one processor; and

memory encoding computer executable instructions that, when executed by at least one processor, perform a method for converting text to speech, the method comprising;

receiving text input into a plurality of first level recurrent neural networks;

determining, by a first recurrent neural network in the plurality of first level recurrent neural networks, one or more properties of the text input from the group consisting of;

part-of-speech properties, phonemes, linguistic prosody properties, contextual properties, and semantic properties;

determining, by a second recurrent neural network in the plurality of first level recurrent neural networks, one or more properties of the text input from the group consisting of;

part-of-speech properties, phonemes, linguistic prosody properties, contextual properties, and semantic properties, wherein the determined one or properties by the second recurrent neural network is different from the determined one or more properties by the first recurrent neural network;

receiving, by a recurrent neural network in a second level, the determined properties from the first recurrent neural network in the plurality of first level recurrent neural networks and the second recurrent neural network in the plurality of first level recurrent neural networks;

determining by the recurrent neural network in the second level, phonetic properties for the text input based on the properties received from the first recurrent neural network in the plurality of first level recurrent neural networks and second neural network in the plurality of first level recurrent neural networks wherein the recurrent neural network in the second level is different from the first recurrent neural network in the plurality of first level recurrent neural networks and the second recurrent neural network in the plurality of first level recurrent neural networks; and

based on the determined phonetic properties, generating a generation sequence for synthetization by an audio synthesizer.