COARTICULATION METHOD FOR AUDIO-VISUAL TEXT-TO-SPEECH SYNTHESIS

• US 20080221904A1
• Filed: 05/19/2008
• Published: 09/11/2008
• Est. Priority Date: 09/07/1999
• Status: Active Grant

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Graf, Hans Peter, Cosatto, Eric, Schroeter, Juergen
• Granted Patent: US 7,630,897 B2
• Time in Patent Office: Days
• US Class Current: 704/276
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads

Abstract

A method for generating animated sequences of talking heads in text-to-speech applications wherein a processor samples a plurality of frames comprising image samples. The processor reads first data comprising one or more parameters associated with noise-producing orifice images of sequences of at least three concatenated phonemes which correspond to an input stimulus. The processor reads, based on the first data. second data comprising images of a noise-producing entity. The processor generates an animated sequence of the noise-producing entity.

Claims

1. A method of generating a noise-producing entity, the method comprising:

receiving a stimulus representing data that the noise-producing entity will track;

associating the stimulus with at least one phoneme or phoneme sequence recalling all mouth parameters corresponding to the associated at least one phoneme or phoneme sequence;

selecting a parameter set from an animation library, the parameter set representing frame segments, the selected parameter set corresponding to the recalled mouth parameters; and

outputting speech associated with the stimulus in synchronization with outputting frame segments associated with the parameter set.

2. The method of claim 1, wherein outputting frame segment further comprises overlaying frame segments on a larger entity to synthesize a whole animated image.

3. The method of claim 1, wherein the stimulus is text.

4. The method of claim 1, wherein the speech is output using a phoneme transcript stored in a coarticulation library.

5. The method of claim 1, wherein the method is iteratively applied to phoneme sequences in the stimulus to form a complete animation.

6. The method of claim 1, wherein the parameter set is associated with images of at least three concatenated phonemes which correspond to the stimulus.

7. A system for generating a noise-producing entity, the system comprising:

a processor;

a module configured to control the processor to receive a stimulus representing data that the noise-producing entity will track;

a module configured to control the processor to associate the stimulus with at least one phoneme or phoneme sequence;

a module configured to control the processor to recall all mouth parameters corresponding to the associated at least one phoneme or phoneme sequence;

a module configured to control the processor to select a parameter set from an animation library, the parameter set representing frame segments, the selected parameter set corresponding to the recalled mouth parameters; and

a module configured to control the processor to output speech associated with the stimulus in synchronization with outputting frame segments associated with the parameter set.

8. The system of claim 7, wherein the module configured to control the processor to output frame segments further overlays frame segments on a larger entity to synthesize a whole animated image.

9. The system of claim 7, wherein the stimulus is text.

10. The system of claim 7, wherein the speech is output using a phoneme transcript stored in a coarticulation library.

11. The system of claim 7, wherein the modules configured to control the processor iteratively operate on phoneme sequences in the stimulus to form a complete animation.

12. The system of claim 7, wherein the parameter set is associated with images of at least three concatenated phonemes which correspond to the stimulus.

13. A computer-readable medium storing instructions for controlling a computing device to generate a noise-producing entity, the instructions comprising:

receiving a stimulus representing data that the noise-producing entity will track;

associating the stimulus with at least one phoneme or phoneme sequence recalling all mouth parameters corresponding to the associated at least one phoneme or phoneme sequence;

selecting a parameter set from an animation library, the parameter set representing frame segments, the selected parameter set corresponding to the recalled mouth parameters; and

outputting speech associated with the stimulus in synchronization with outputting frame segments associated with the parameter set.

14. The computer-readable medium of claim 13, wherein outputting frame segment further comprises overlaying frame segments on a larger entity to synthesize a whole animated image.

15. The computer-readable medium of claim 13, wherein the stimulus is text.

16. The computer-readable medium of claim 13, wherein the speech is output using a phoneme transcript stored in a coarticulation library.

17. The computer-readable medium of claim 13, wherein the instructions are iteratively applied to phoneme sequences in the stimulus to form a complete animation.

18. The computer-readable medium of claim 13, wherein the parameter set is associated with images of at least three concatenated phonemes which correspond to the stimulus.