Rapid computer modeling of faces for animation

• US 20040208344A1
• Filed: 05/14/2004
• Published: 10/21/2004
• Est. Priority Date: 03/09/2000
• Status: Active Grant

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Zhengyou, Liu, Zicheng, Cohen, Michael F., Jacobs, Charles E.
• Granted Patent: US 6,950,537 B2
• Time in Patent Office: Days
• US Class Current: 382/118
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/503 : involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109

Abstract

Described herein is a technique for creating a 3D face model using images obtained from an inexpensive camera associated with a general-purpose computer. Two still images of the user are captured, and two video sequences. The user is asked to identify five facial features, which are used to calculate a mask and to perform fitting operations. Based on a comparison of the still images, deformation vectors are applied to a neutral face model to create the 3D model. The video sequences are used to create a texture map. The process of creating the texture map references the previously obtained 3D model to determine poses of the sequential video images.

Claims

1. One or more computer-readable media containing a program that is executable by a computer to create a three-dimensional model of a face, the program comprising the following actions:

constructing a three-dimensional face model from two or more two-dimensional face images, wherein the constructed face model comprises a plurality of points specified in three dimensions;

applying a combination of deformation vectors to a pre-specified face model to deform the pre-specified face model approximately to the constructed face model, wherein the pre-specified face model comprises a plurality points specified in three dimensions and each deformation vector specifies a plurality of displacements corresponding respectively to the plurality of points of the pre-specified face model.

2. One or more computer-readable media as recited in claim 1, wherein the applying action comprises calculating a linear combination of the deformation vectors and the pre-specified face model.

3. One or more computer-readable media as recited in claim 1, wherein the applying action comprises calculating a linear combination of the deformation vectors and the pre-specified face model, the actions further comprising:

multiplying each deformation vector by a respective multiplicative constant before when calculating the linear combination; and

selecting the multiplicative constants to minimize a summation of distances between the deformed face model and the points of the constructed face model.

4. One or more computer-readable media as recited in claim 1, wherein the applying action comprises calculating a linear combination of the deformation vectors and the pre-specified face model, the actions further comprising:

identifying one or more points of the constructed face model and corresponding points of the pre-specified face model;

multiplying each deformation vector by a respective multiplicative constant before when calculating the linear combination; and

selecting the multiplicative constants to minimize a summation of the distances between the identified points of the constructed face model and the corresponding points of the pre-specified face model.

5. One or more computer-readable media as recited in claim 1, wherein the applying action comprises calculating a linear combination of the deformation vectors and the pre-specified face model, the actions further comprising:

identifying one or more points of the constructed face model and corresponding points of the pre-specified face model;

multiplying each deformation vector by a respective multiplicative constant before when calculating the linear combination; and

selecting the multiplicative constants to minimize (a) a summation of the distances between the identified points of the constructed face model and the corresponding points of the pre-specified face model, and (b) a summation of the distances between the identified points of the constructed face model and the corresponding points of the pre-specified face model.

6. A method of creating a three-dimensional model of a face, comprising applying a combination of deformation vectors to a pre-specified face model to deform the pre-specified face model approximately to a reconstructed face model, wherein the pre-specified face model comprises a plurality of points specified in three dimensions and each deformation vector specifies a plurality of displacements corresponding respectively to the plurality of three dimensional points of the pre-specified face model.

7. A method as recited in claim 6, wherein the applying comprises calculating a linear combination of the deformation vectors and the pre-specified face model.

8. A method as recited in claim 6, wherein the applying comprises calculating a linear combination of the deformation vectors and the pre-specified face model, the actions further comprising:

multiplying each deformation vector by a respective multiplicative constant before when calculating the linear combination; and

selecting the multiplicative constants to minimize a summation of distances between the deformed face model and the points of the constructed face model.

9. A method as recited in claim 6, wherein the applying comprises calculating a linear combination of the deformation vectors and the pre-specified face model, the actions further comprising:

identifying one or more points of the constructed face model and corresponding points of the pre-specified face model;

multiplying each deformation vector by a respective multiplicative constant before when calculating the linear combination; and

selecting the multiplicative constants to minimize a summation of distances between the identified points of the constructed face model and the corresponding points of the pre-specified face model.

10. A method as recited in claim 6, wherein the applying comprises calculating a linear combination of the deformation vectors and the pre-specified face model, the actions further comprising:

identifying one or more points of the constructed face model and corresponding points of the pre-specified face model;

multiplying each deformation vector by a respective multiplicative constant before when calculating the linear combination; and

selecting the multiplicative constants to minimize (a) a summation of distances between the deformed face model and the points of the constructed face model, and (b) a summation of distances between the identified points of the constructed face model and the corresponding points of the pre-specified face model.

11. A method of creating a three-dimensional model of a face, comprising:

specifying a neutral face model in terms of three dimensional points;

specifying a plurality of deformation vectors, wherein each deformation vector indicates a plurality of displacements corresponding respectively to the points of the neutral face model, and wherein each deformation vector deforms the neutral face model in a particular way when applied to the neutral face model;

specifying degree of deformation for each deformation vector in terms of a multiplicative constant;

deforming the neutral face model by calculating a linear combination of the neutral face model and the deformation vectors, each deformation vector being multiplied by its multiplicative constant; and

selecting the multiplicative constants to minimize a summation of distances between the deformed face model and the points of the constructed face model.

12. A method as recited in claim 11, further comprising:

identifying one or more points of the constructed face model and corresponding points of the pre-specified face model;

further selecting the multiplicative constants to minimize a summation of distances between the identified points of the constructed face model and the corresponding points of the pre-specified face model.

13. A method as recited in claim 11, further comprising:

accepting input from a human user identifying one or more points of the constructed face model that correspond to particular points of the pre-specified face model;

further selecting the multiplicative constants to minimize a summation of distances between the identified points of the constructed face model and the corresponding points of the pre-specified face model.