Coarticulation Method for Audio-Visual Text-to-Speech Synthesis

• US 20100076762A1
• Filed: 11/30/2009
• Published: 03/25/2010
• Est. Priority Date: 09/07/1999
• Status: Active Grant

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Graf, Hans Peter, Cosatto, Eric, Schroeter, Juergen
• Granted Patent: US 8,078,466 B2
• Time in Patent Office: Days
• US Class Current: 704/235
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads

Abstract

A method for generating animated sequences of talking heads in text-to-speech applications wherein a processor samples a plurality of frames comprising image samples. The processor reads first data comprising one or more parameters associated with noise-producing orifice images of sequences of at least three concatenated phonemes which correspond to an input stimulus. The processor reads, based on the first data. second data comprising images of a noise-producing entity. The processor generates an animated sequence of the noise-producing entity.

Claims

1. A method of synchronizing synthesized speech and animation, the method causing a computing device to perform steps comprising:

associating a received stimulus with a phoneme having corresponding mouth parameters;

selecting a parameter set corresponding to the mouth parameters from an animation library, the parameter set representing frame segments; and

generating, via a noise producing entity, speech associated with the stimulus that is synchronized with the frame segments.

2. The method of claim 1, wherein generating speech further comprises overlaying frame segments on a larger entity to synthesize a whole animated image.

3. The method of claim 1, wherein the stimulus is text.

4. The method of claim 3, wherein the text stimulus is derived from speech recognition.

5. The method of claim 1, wherein the speech is output using a phoneme transcript stored in a coarticulation library.

6. The method of claim 1, further comprising iteratively applying the method to phoneme sequences in the stimulus to form a complete animation.

7. The method of claim 1, wherein the parameter set is associated with images of at least three concatenated phonemes with correspond to the stimulus.

16. The method of claim 1, wherein generating speech further comprises overlaying frame segments on a larger entity to synthesize a whole animated image.

17. The method of claim 1, wherein the stimulus is text.

18. The method of claim 3, wherein the text stimulus is derived from speech recognition.

19. The method of claim 1, wherein the speech is output using a phoneme transcript stored in a coarticulation library.

20. The method of claim 1, further comprising iteratively applying the method to phoneme sequences in the stimulus to form a complete animation.

8. A system for synchronizing synthesized speech and animation, the system comprising:

a processor;

a module controlling the processor to associate a received stimulus with a phoneme having corresponding mouth parameters;

a module controlling the processor to select a parameter set corresponding to the mouth parameters from an animation library, the parameter set representing frame segments; and

a module controlling the processor to generate, via a noise producing entity, speech associated with the stimulus that is synchronized with the frame segments.

9. The system of claim 8, wherein the module controlling the processor to generate speech further controls the processor to overlay frame segments on a larger entity to synthesize a whole animated image.

10. The system of claim 8, wherein the stimulus is text.

11. The system of claim 10, wherein the text stimulus is derived from speech recognition.

12. The system of claim 8, wherein the speech is output using a phoneme transcript stored in a coarticulation library.

13. The system of claim 8, further comprising a module controlling the processor to iteratively apply the method to phoneme sequences in the stimulus to form a complete animation.

14. The system of claim 8, wherein the parameter set is associated with images of at least three concatenated phonemes with correspond to the stimulus.

15. A method of synchronizing synthesized speech and animation, the method causing a computing device to perform steps comprising:

associating a received stimulus with a phoneme having corresponding mouth parameters;

selecting a parameter set corresponding to the mouth parameters from an animation library, the parameter set representing frame segments; and

generating, via a noise producing entity, speech associated with the stimulus that is synchronized with the frame segments.