SPEECH SYNTHESIS FROM DETECTED SPEECH ARTICULATOR MOVEMENT

• US 20170263237A1
• Filed: 09/10/2015
• Published: 09/14/2017
• Est. Priority Date: 09/16/2014
• Status: Abandoned Application

• Current Assignee: University of HULL
• Original Assignee: University of HULL
• Inventors: Green, Philip Duncan, Moore, Roger Kenneth, Gilbert, James Michael, Lopez, Jose Andres Gonzalez
• Application Number:
  US15/511,784
• Publication Number:
  US 20170263237A1
• Time in Patent Office: Days
• CPC Class Codes: G10L 13/04 Details of speech synthesis systems, e.g. synthesiser structure or memory management
• CPC Class Codes: G10L 13/08 Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features

Abstract

The present invention relates to methods and systems for generating synthesized speech from detected speech articulator movement and to methods and systems for generating models for generating synthesized speech from detected articulator movement.

Claims

1. A method of synthesising speech, comprising

inputting articulator movement data to a conversion model, said conversion model comprising parameters representing a relationship between speech articulator movements and speech sound;

transforming by the conversion model said articulator movement data into synthesised speech sound data in accordance with the parameters, wherein

the parameters of said conversion model are estimated based on training articulator movement data and training speech sound data, said training speech sound data associated with training speech sound and said training articulator movement data associated with articulator movement of a user produced in accordance with the training speech sound.

2. A method according to claim 1, wherein the training speech sound is produced by the user and said training articulator movement data is associated with articulator movement produced when the user produces the training speech sound.

3. A method according to claim 1, wherein the training articulator movement data is associated with articulator movement produced by the user when the user mouths along with the training speech sound.

4. A method according to claim 1, wherein the articulator movement data input to the conversion model corresponds to input articulator movement feature vectors.

5. A method according to claim 4, wherein the synthesised speech sound data comprises output speech feature vectors produced by the conversion model when transforming the input articulator movement feature vectors.

6. A method according to claim 5, wherein the output speech feature vectors are estimated from the articulator movement feature vectors by applying a non-linear transformation that depends on the conversion model parameters.

7. A method according to claim 1, further comprising converting the synthesised speech sound data to an audible time domain signal.

8. A method of generating a conversion model for synthesising speech, said method comprising:

receiving training articulator movement data and training speech sound data, said training speech sound data associated with training speech sound and said training articulator movement data associated with articulator movement of a user produced in accordance with the training speech sound, and

estimating parameters of the conversion model based on the training articulator movement data and training speech sound data, said parameters representing a relationship between articulator movement and speech sound.

9. A method according to claim 8, wherein the training speech sound is produced by the user and said training articulator movement data is associated with articulator movement produced when the user produces the training speech sound.

10. A method according to claim 8, wherein, the training articulator movement data is associated with articulator movement produced by the user when the user mouths along with the training speech sound.

11. A method according to claim 9, further comprising

extracting the training speech sound data from speech sound signals captured when the user makes the training speech sound, and

extracting the training articulator movement data from articulator movement signals captured when the user produces the training speech sound.

12. A method according to claim 11, wherein capturing the speech sound signals and capturing the articulator movement signals occurs substantially simultaneously.

13. A method according to claim 11, wherein the training speech sound data comprises training speech feature vectors, and the training articulator movement data comprises training articulator movement feature vectors.

14. A method according to claim 11, further comprising

using a training procedure to estimate the parameters given the training speech feature vectors and the training articulator movement feature vectors, the parameters defining a joint or conditional probability distribution associating articulator movement feature vectors input to the conversion model with output speech feature vectors output from the conversion model.

15. A method according to claim 14, wherein applying the conditional probability distribution is represented either as a recurrent neural network or as a statistical mixture model comprising a number of probability distributions weighted by corresponding mixture weights.

16. A method according to claim 15, wherein the method according to claim 15, wherein the recurrent neural network is a long short term memory (LSTM) recurrent neural network.

17. A method according to claim 15, wherein the statistical mixture model is a mixture of factor analysers (MFA).

19. A method according to claim 11, comprising extracting the training speech feature vectors from the speech sound signals using linear predictive coding (LPC).

20. A method according to claim 11, comprising extracting the training articulator movement feature vectors from the articulator movement signals using principal component analysis (PCA).

18. (canceled)

21. A system for synthesising speech, comprising a conversion model implemented on a data processor, said conversion model comprising parameters representing a relationship between speech articulator movements and speech sound, said conversion model arranged

to receive input articulator movement data, and responsive to receiving the input articulator movement data,

to transform the input articulator movement data into synthesised speech sound data in accordance with the parameters, wherein

the parameters of the conversion model are estimated based on training articulator movement data and training speech sound data, said training speech sound data associated with training speech sound and said training articulator movement data associated with articulator movement of a user produced in accordance with the training speech sound.

22. -26. (canceled)