Method of modeling objects to synthesize three-dimensional, photo-realistic animations

• US 6,504,546 B1
• Filed: 02/08/2000
• Issued: 01/07/2003
• Est. Priority Date: 02/08/2000
• Status: Expired due to Term

• Current Assignee: Rakuten, Inc. (Rakuten Group, Inc.)
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Graf, Hans Peter, Cosatto, Eric
• Primary Examiner(s): Jankus, Almis R.
• Application Number:
  US09/500,334
• Time in Patent Office: 1,064 Days
• Field of Search: 345/473-475
• US Class Current: 345/473
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads

Abstract

A method for modeling three-dimensional objects to create photo-realistic animations using a data-driven approach. The three-dimensional object is defined by a set of separate three-dimensional planes, each plane enclosing an area of the object that undergoes visual changes during animation. Recorded video is used to create bitmap data to populate a database for each three-dimensional plane. The video is analyzed in terms of both rigid movements (changes in pose) and plastic deformation (changes in expression) to create the bitmaps. The modeling is particularly well-suited for animations of a human face, where an audio track generated by a text-to-speech synthesizer can be added to the animation to create a photo-realistic “talking head”.

Claims

1. A method of creating a three-dimensional model of a three-dimensional object for generating photo-realistic animation, the method comprising the steps of:

a) defining a set of three-dimensional planes approximating an overall shape of the three-dimensional object and its surrounding area;

b) recording a set of images of said three-dimensional object as it undergoes both rigid movement, defined as changes in pose, and plastic deformation, defined as changes in appearances;

c) creating a bitmap for each recorded image;

d) storing the created bitmaps in a database, the database used to create the three-dimensional model of said three-dimensional object;

e) preparing an empty frame to receive a synthesized view of said three-dimensional object;

f) choosing an arbitrary pose for the synthesis of said three-dimensional object and a desired appearance;

g) selecting, for a three-dimensional plane from the set of three-dimensional planes, a bitmap with the desired appearance, chosen in step f), from those stored in the database and determining the pose of said three-dimensional object as it appears on that bitmap;

h) re-projecting the three-dimensional plane of said three-dimensional object from its pose on the recorded bitmap onto its new pose on the empty frame created in step e); and

i) repeating steps g) and h) for each three-dimensional plane in the set of three-dimensional planes.

2. The method as defined in claim 1 wherein the method further comprises the step of:

j) repeating steps e) through i) for each frame of an animation.

3. The method as defined in claim 1, wherein in performing step h), the following steps are performed:

1) obtaining, for a selected plane from the set of three-dimensional planes, a first quadrilateral boundary marking the plane'"'"'s perspective projection on the recorded image, using the pose of the object on said image;

2) obtaining, for the selected plane, a second quadrilateral boundary on the synthesized frame, marking the plane'"'"'s perspective projection using the new chosen pose of the object; and

3) warping the pixels of the image that are bound by the first quadrilateral into the pixels of the synthesized frame bound by the second quadrilateral.

4. The method as defined in claim 3 wherein in performing step h3), bilinear interpolation is used in the warping process.

5. The method as defined in claim 3 wherein in performing step h3), pixel warping includes the method of alpha-blending using a transparency mask.

6. The method as defined by claim 1 wherein in performing step g), a feature vector is used for selecting the bitmap.

7. The method as defined in claim 6 wherein the feature vector contains one or more of the following features:

the pose of the object;

measurements of recognizable parts of the object obtained by computer vision algorithms from the image;

statistics calculated directly from the pixels of the image;

normalized bitmaps from which pixels are obtained by warping the pixels of an area of the image corresponding to the perspective projection of a plane of the object'"'"'s model using the pose of the object;

measurements of recognizable parts on a normalized bitmap; and

statistics calculated directly from the pixels of a normalized bitmap.

8. The method as defined in claim 7 wherein the feature vectors are pre-computed from the recorded bitmaps of step b) and stored in the database, before executing step g).

9. The method as defined in claim 1 wherein the object is a human face and in performing step a), the set of three-dimensional planes approximating the overall shape of the face comprises:

forehead, eyes, cheeks, mouth, chin, jaw and the entire head.

10. The method as defined in claim 9 wherein in performing step f), the pose is computed from the positions of a set of facial parts, namely:

the four eye corners and the pair of nostrils.