Rapid computer modeling of faces for animation

• US 7,142,698 B2
• Filed: 12/27/2005
• Issued: 11/28/2006
• Est. Priority Date: 03/09/2000
• Status: Expired due to Term

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Zhengyou, Liu, Zicheng, Cohen, Michael F., Jacobs, Charles E.
• Primary Examiner(s): SHERALI, ISHRAT I
• Application Number:
  US11/275,340
• Publication Number:
  US 20060126924A1
• Time in Patent Office: 336 Days
• Field of Search: 382115-118, 382/154, 382/100, 382/103, 382/107, 382/190, 382/209, 382216-218, 382/285, 382/276, 382294-297, 382/236, 382162-165, 345418-428, 345581-589, 345473-475, 345/441, 348 141- 1416, 348 77- 78, 340 582- 583
• US Class Current: 382/118
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/503 : involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109

Abstract

Described herein is a technique for creating a 3D face model using images obtained from an inexpensive camera associated with a general-purpose computer. Two still images of the user are captured, and two video sequences. The user is asked to identify five facial features, which are used to calculate a mask and to perform fitting operations. Based on a comparison of the still images, deformation vectors are applied to a neutral face model to create the 3D model. The video sequences are used to create a texture map. The process of creating the texture map references the previously obtained 3D model to determine poses of the sequential video images.

Claims

1. One or more computer-readable media containing a program that is executable by a computer to create a three-dimensional model of a face, the program comprising the following actions:

capturing at least two images of the face;

matching a plurality of points between the images;

estimating rotation and translation of the face based on the images;

determining 3D positions of the matched points based on the estimated rotation and translation to create a constructed face model, the constructed face model comprising a plurality of points specified in three dimensions; and

applying a combination of deformation vectors to a pre-specified face model to deform the pre-specified face model approximately to the constructed face model, wherein the pre-specified face model comprises a plurality points specified in three dimensions and each deformation vector specifies a plurality of displacements corresponding respectively to the plurality of points of the pre-specified face model.

2. One or more computer-readable media as recited in claim 1, wherein said estimating is based on matched points within the images.

3. One or more computer-readable media as recited in claim 1, wherein the applying action comprises calculating a linear combination of the deformation vectors and the pre-specified face model.

4. One or more computer-readable media as recited in claim 1, wherein the applying action comprises calculating a linear combination of the deformation vectors and the pre-specified face model, the actions further comprising:

multiplying each deformation vector by a respective multiplicative constant before when calculating the linear combination; and

selecting the multiplicative constants to minimize a summation of distances between the deformed face model and the points of the constructed face model.

5. One or more computer-readable media as recited in claim 1, wherein the applying action comprises calculating a linear combination of the deformation vectors and the pre-specified face model, the actions further comprising:

identifying one or more points of the constructed face model and corresponding points of the pre-specified face model;

multiplying each deformation vector by a respective multiplicative constant before when calculating the linear combination; and

selecting the multiplicative constants to minimize a summation of the distances between the identified points of the constructed face model and the corresponding points of the pre-specified face model.

6. One or more computer-readable media as recited in claim 1, wherein the applying action comprises calculating a linear combination of the deformation vectors and the pre-specified face model, the actions further comprising:

identifying one or more points of the constructed face model and corresponding points of the pre-specified face model;

multiplying each deformation vector by a respective multiplicative constant before when calculating the linear combination; and

selecting the multiplicative constants to minimize (a) a summation of the distances between the identified points of the constructed face model and the corresponding points of the pre-specified face model, and (b) a summation of the distances between the identified points of the constructed face model and the corresponding points of the pre-specified face model.

7. A method of creating a three-dimensional model of a face, comprising:

capturing at least two images of the face;

matching a plurality of points between the images;

estimating rotation and translation of the face based the images;

determining 3D positions of the matched points based on the estimated rotation and translation to create a constructed face model, the constructed face model comprising a plurality of points specified in three dimensions; and

applying a combination of deformation vectors to a pre-specified face model to deform the pre-specified face model approximately to the constructed face model, wherein the pre-specified face model comprises a plurality of points specified in three dimensions and each deformation vector specifies a plurality of displacements corresponding respectively to the plurality of three dimensional points of the pre-specified face model.

8. A method as recited in claim 7, wherein said estimating is based on matched points within the images.

9. A method as recited in claim 7, wherein the applying comprises calculating a linear combination of the deformation vectors and the pre-specified face model.

10. A method as recited in claim 7, wherein the applying comprises calculating a linear combination of the deformation vectors and the pre-specified face model, the actions further comprising:

multiplying each deformation vector by a respective multiplicative constant before when calculating the linear combination; and

selecting the multiplicative constants to minimize a summation of distances between the deformed face model and the points of the constructed face model.

11. A method as recited in claim 7, wherein the applying comprises calculating a linear combination of the deformation vectors and the pre-specified face model, the actions further comprising:

identifying one or more points of the constructed face model and corresponding points of the pre-specified face model;

multiplying each deformation vector by a respective multiplicative constant before when calculating the linear combination; and

selecting the multiplicative constants to minimize a summation of distances between the identified points of the constructed face model and the corresponding points of the pre-specified face model.

12. A method as recited in claim 7, wherein the applying comprises calculating a linear combination of the deformation vectors and the pre-specified face model, the actions further comprising:

identifying one or more points of the constructed face model and corresponding points of the pre-specified face model;

multiplying each deformation vector by a respective multiplicative constant before when calculating the linear combination; and

selecting the multiplicative constants to minimize (a) a summation of distances between the deformed face model and the points of the constructed face model, and (b) a summation of distances between the identified points of the constructed face model and the corresponding points of the pre-specified face model.

13. An apparatus that creates a three-dimensional model of a face, the apparatus comprising:

means for capturing at least two images of the face;

means for matching a plurality of points between the images;

means for estimating rotation and translation of the face based on the images;

means for determining 3D positions of the matched points based on the estimated rotation and translation to create a constructed face model, the constructed face model comprising a plurality of points specified in three dimensions; and

means for applying a combination of deformation vectors to a pre-specified face model to deform the pre-specified face model approximately to the constructed face model, wherein the pre-specified face model comprises a plurality points specified in three dimensions and each deformation vector specifies a plurality of displacements corresponding respectively to the plurality of points of the pre-specified face model.

14. An apparatus as recited in claim 13, wherein said estimating means performs estimation based on matched points within the images.

15. An apparatus as recited in claim 13, wherein the applying means is capable of calculating a linear combination of the deformation vectors and the pre-specified face model.

16. An apparatus as recited in claim 13, wherein the applying means is capable of calculating a linear combination of the deformation vectors and the pre-specified face model, the apparatus further comprising:

means for multiplying each deformation vector by a respective multiplicative constant before when calculating the linear combination; and

means for selecting the multiplicative constants to minimize a summation of distances between the deformed face model and the points of the constructed face model.

17. An apparatus as recited in claim 13, wherein the applying means is capable of calculating a linear combination of the deformation vectors and the pre-specified face model, the apparatus further comprising:

means for identifying one or more points of the constructed face model and corresponding points of the pre-specified face model;

means for multiplying each deformation vector by a respective multiplicative constant before when calculating the linear combination; and

means for selecting the multiplicative constants to minimize a summation of the distances between the identified points of the constructed face model and the corresponding points of the pre-specified face model.

18. An apparatus as recited in claim 13, wherein the applying means is capable of calculating a linear combination of the deformation vectors and the pre-specified face model, the apparatus further comprising:

means for identifying one or more points of the constructed face model and corresponding points of the pre-specified face model;

means for multiplying each deformation vector by a respective multiplicative constant before when calculating the linear combination; and

means for selecting the multiplicative constants to minimize (a) a summation of the distances between the identified points of the constructed face model and the corresponding points of the pre-specified face model, and (b) a summation of the distances between the identified points of the constructed face model and the corresponding points of the pre-specified face model.