Method for generating photo-realistic animated characters

• US 5,995,119 A
• Filed: 06/06/1997
• Issued: 11/30/1999
• Est. Priority Date: 06/06/1997
• Status: Expired due to Term

• Current Assignee: AT&T Corporation (AT&T, Inc.)
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Graf, Hans Peter, Cosatto, Eric
• Primary Examiner(s): Nguyen, Phu K.
• Assistant Examiner(s): Vo, Cliff N.
• Application Number:
  US08/869,531
• Time in Patent Office: 907 Days
• Field of Search: 345/419, 345/435, 345/473, 345/474
• US Class Current: 345/473
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings

Abstract

A method for generating photo-realistic characters wherein one or more pictures of an individual are decomposed into a plurality of parameterized facial parts. The facial parts are stored in memory. To create animated frames, the individual facial parts are recalled from memory in a defined manner and overlaid onto a base face to form a whole face, which, in turn, may be overlaid onto a background image to form an animated frame.

Claims

1. A method for generating an animated image of a photo-realistic face, comprising the steps of

decomposing one or more pictures of a face into a hierarchy of parameterized facial parts;

storing the parameterized facial parts into a memory;

loading, in a designated manner, one or more of the parameterized facial parts from the memory; and

overlaying the one or more parameterized facial parts onto a base face to form a whole face.

2. The method of claim 1, wherein said loading step and said overlaying step are performed in real-time.

3. The method of claim 1, wherein said decomposing step comprises photographic extraction.

4. The method of claim 3, wherein said photographic extraction step further comprises warping.

5. The method of claim 3, wherein said photographic extraction step further comprises morphing.

6. The method of claim 3, wherein said photographic extraction step further comprises interpolation between reference views.

7. The method of claim 1, wherein said decomposing step comprises warping.

8. The method of claim 1, wherein the parameterized facial parts comprise parameters in memory.

9. The method of claim 8, wherein the parameters for each parameterized facial part comprise control points positioned on a bitmap.

10. The method of claim 1, wherein said overlaying step further comprises the step of:

overlaying the whole face with one or more secondary effects.

11. The method of claim 1, wherein the parameterized facial parts comprise a mouth, eyes, a nose, cheeks, a chin, a forehead, teeth, a tongue, and a mouth cavity.

12. The method of claim 1, further comprising the step of:

overlaying the whole face onto a background image to form a frame; and

outputting the frame to a screen.

13. The method of claim 12, wherein said loading step, both overlaying steps, and said outputting step are performed in real-time.

14. A method for generating animated frames of photo-realistic characters, comprising the steps of

decomposing one or more pictures of a face into a group of parameterized facial parts;

storing the group of parameterized facial parts into a model library;

overlaying designated parameterized facial parts of the group onto a base face to form a whole face; and

overlaying the whole face onto a background image to form an animated frame.

15. The method of claim 14, wherein both overlaying steps are performed in real-time on a computer.

16. The method of claim 14, wherein the designated parameterized facial parts of the group are selected based on a designated viseme.

17. The method of claim 14, wherein said decomposing step comprises photographic extraction.

18. The method of claim 14, wherein said decomposing step comprises warping.

19. The method of claim 14, wherein the designated viseme is obtained from a memory by mapping a specified phoneme.

20. The method of claim 14, further comprising the step of:

outputting the animated frame to a screen.

21. The method of claim 20, wherein both overlaying steps and said outputting step are performed in real-time on a computer.

22. The method of claim 14, further comprising the step of:

outputting the animated frame to a file.

23. The method of claim 19, further comprising the step of synchronizing the frame with the specified phoneme.

24. The method of claim 14, further comprising the steps of:

decomposing the one or more pictures into a group of parameterized facial parts with hair; and

overlaying designated parameterized facial parts with hair onto the base face.

25. A method for generating an animated sequence of frames of photo-realistic faces, comprising the steps of:

decomposing one or more pictures of a character into a plurality of parameterized facial parts, each parameterized facial part corresponding to a specific expression and each parameterized facial part comprising parameters in memory;

loading from memory, for each of a plurality of desired facial expressions, a set of parameterized facial parts;

overlaying, for each of the plurality of desired facial expressions, the set of parameterized facial parts onto a base face to form a whole face having the facial expression, wherein parameters of each parameterized facial part of the set are determined based on the desired facial expression; and

overlaying, for each of the plurality of desired facial expressions, the whole face onto a background to form a sequence of animated frames.

26. The method of claim 25, further comprising the step of:

synchronizing, using a speech synthesizer, each frame of the sequence with a phoneme corresponding to the facial expression used in the frame.

27. A method for generating an animated frame of a speaking photo-realistic face, comprising the steps of:

obtaining from memory a viseme predetermined to correspond with a specified phoneme input;

selecting from memory a set of one or more parameterized facial parts based on parameters corresponding to the viseme;

overlaying the set of selected one or more parameterized facial parts onto a base face to form a whole face; and

overlaying the whole face onto a background to form a frame.

29. The method according to claim 27, further comprising the step of:

synchronizing the frame with a sound corresponding to the phoneme.

28. The method of step 27, wherein the specified phoneme comprises the output of a text-to-speech synthesizer used to input text to be spoken concurrently with the animated frame.

30. A method for generating a sequence of animated frames of photo-realistic characters, comprising the steps of

decomposing one or more pictures of a face into parameterized facial parts comprising a plurality of distinct expressions;

storing the plurality of parameterized facial parts into a memory;

loading, for each of a plurality of frames, a designated set of parameterized facial parts from the memory;

overlaying, for each of the plurality of frames, the designated parameterized facial parts onto a base face to form an animated image.

31. The method of claim 30, wherein said decomposing step comprises photographic extraction.

32. The method of claim 30, wherein said decomposing step comprises warping.

33. The method of claim 30, wherein

said loading and said overlaying steps are performed in real-time on a computer.

34. The method of claim 33, wherein the computer is a personal computer.

35. The method of claim 30, wherein each designated set of parameterized facial parts is determined based on a viseme.

36. A method for synthesizing an animated frame of a photo-realistic character, comprising the steps of:

obtaining a viseme from memory based on a phoneme input;

selecting one or more parameterized facial parts based on parameters corresponding to the viseme;

calculating coarticulation effects based on the parameters corresponding to the viseme;

producing modified parameters based at least on said calculating step;

selecting modified parameterized facial parts based on the modified parameters; and

generating an animated frame by overlaying facial parts onto a base face overlaid on a background image.

37. The method of claim 36, wherein said step of producing modified parameters is based on input comprising facial expression information.

38. A method for developing information to enable a program to generate an animated image comprising the steps of:

defining a base face and storing the definition of said base face in a memory;

decomposing said base face into facial parts;

for at least one of said facial parts, developing a plurality of different realizations of said at least one of said facial parts, and storing said different realizations in said memory;

where some of said different realizations of said at least one of said facial parts correspond to different visemes.

39. The method of claim 38 where some of said different realizations of said at least one of said facial parts correspond to different emotional expressions.

40. The method of claim 38 where said facial parts are parameterized, and the parameters are adapted for use in combining diverse facial parts into an image of a face.

41. A method for generating a sequence of animated frames comprising the steps of:

identifying a phoneme to be synthesized;

selecting from a memory a set of facial parts and their associated parameters, which facial parts combine to form a face image, where at least one of the facial parts that is selected for inclusion in said set is selected from a subset of like facial parts in said memory, based on said phoneme;

retrieving from memory a base face definition; and

overlaying said facial parts selected for said set onto said base face with the aid of said parameters.