Method and apparatus for synthetic speech in facial animation

• US 5,878,396 A
• Filed: 02/05/1998
• Issued: 03/02/1999
• Est. Priority Date: 01/21/1993
• Status: Expired due to Term

• Current Assignee: Apple Computer Incorporated (Apple Inc.)
• Original Assignee: Apple Computer Incorporated (Apple Inc.)
• Inventors: Henton, Caroline G.
• Primary Examiner(s): Hudspeth, David R.
• Assistant Examiner(s): Opsasnick, Michael N.
• Application Number:
  US09/019,514
• Time in Patent Office: 390 Days
• Field of Search: 704/270, 704/276, 704/235
• US Class Current: 704/276
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads
• CPC Class Codes: H04N 19/20 using video object coding

Abstract

The present invention utilizes a novel approach to facial imaging synchronized with synthetic speech. Mapping viseme images to a diphone requires the same `transitioning` in that the imaging associated with a diphone is not a static image, but rather, a series of images which dynamically depict, with lip, teeth and tongue positioning, the sound transition occurring in the relevant diphone. Each series of lip, teeth, and tongue positioning transitions is referred to herein as a `diseme.` A diseme (like a diphone) thus begins somewhere during one viseme (phone) and ends somewhere during a following viseme (phone). Due to lip, teeth and tongue position imaging commonality, phones are grouped into archiphonic families. A single diseme, which depicts the transition from a phone in one archiphonic family to another phone in a different archiphonic family, can be used for displaying the transition between any phone in the first archiphonic family to any phone in the second archiphonic family. In this way, the approximately 1800 diphones in General American English can be visually depicted by a relatively small number of disemes, again, due to their similarity in lip, teeth, and tongue image positioning. This results in a mapping between synthetic speech and facial imaging which more accurately reflects the speech transitional movements of a realistic speaker image.

Claims

1. A method for synchronizing facial images to synthetic speech comprising:

a) storing a sequence of facial images from a viseme in a first archiphonic group to a viseme in a second archiphonic group;

b) displaying the sequence of facial images from the viseme in the first archiphonic group to the viseme in the second archiphonic group if a phone from the first archiphonic group is followed by a phone from the second archiphonic group in the synthetic speech.

2. An apparatus for synchronizing facial images to synthetic speech comprising:

a) means for storing a sequence of facial images from a viseme in a first archiphonic group to a viseme in a second archiphonic group;

b) means for displaying the sequence of facial images from the viseme in the first archiphonic group to the viseme in the second archiphonic group if a phone from the first archiphonic group is followed by a phone from the second archiphonic group in the synthetic speech.

3. A method for synchronizing facial images to synthetic speech in an electronic system wherein the synthetic speech can be divided into a set of phonemes comprising distinctive phonetic features and wherein each of the phonemes of the set of phonemes has associated with it a viseme comprising an image of visible articulatory facial features, said method comprising:

a) dividing the phonemes of the set of phonemes into multiple families according to the distinctive phonetic features of the phonemes of the set of phonemes;

b) designating one phonemes of each of the multiple families as an archiphone for that family;

c) storing a diseme comprising a sequence of images beginning with the viseme of the archiphone of 1 of the multiple families and ending with the viseme of the archiphone of one of the other of the multiple families;

d) displaying the stored diseme if in the synthetic speech any phonemes from the one of the multiple families is followed by any phonemes from the one of the other of the multiple families.

4. The system for synchronizing facial images to synthetic speech in an electronic system wherein the synthetic speech can be divided into a set of phonemes comprising distinctive phonetic features and wherein each of the phonemes of the set of phonemes has associated with it a viseme comprising an image of visible articulatory facial features, said apparatus comprising:

a) means for dividing the phonemes of the set of phonemes into multiple families according to the distinctive phonetic features of the phonemes of the set of phonemes;

b) means for designating one phony of each of the multiple families as an archiphone for that family;

c) means for storing a diseme comprising a sequence of images beginning with the viseme of the archiphone of one of the multiple families and ending with the viseme of the archiphone of one of the other of the multiple families;

d) means for displaying be stored diseme if in the synthetic speech any phonemes from the one of the multiple families is followed by any phonemes from the one of the other of the multiple families.

5. A method for synchronizing facial images to synthetic speech comprising:

a) storing a sequence of facial images from a first viseme to a second viseme; and

b) displaying the sequence of facial image from the first viseme to the second viseme if a first phone is followed by a second phone in the synthetic speech.

6. The method according to claim 5 wherein the first viseme is an element in a first archiphonic group and the second viseme is an element in a second archiphonic group.

7. The method according to claim 5 wherein the first phone is an element in a first archiphonic group and the second phone is an element in a second archiphonic group.

8. A computer-readable medium comprising program instructions for the synchronizing of facial images to synthetic speech by performing the steps of:

a) storing a sequence of facial images from a viseme in a first archiphonic group to a viseme in a second archiphonic group; and

b) displaying the sequence of facial images from the viseme in the first archiphonic group to the viseme in the second archiphonic group if a phone from the first archiphonic group is followed by a phone from the second archiphonic group in the synthetic speech.