Rapid computer modeling of faces for animation

• US 20050190962A1
• Filed: 04/20/2005
• Published: 09/01/2005
• Est. Priority Date: 03/09/2000
• Status: Active Grant

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Zhengyou, Liu, Zicheng, Cohen, Michael F., Jacobs, Charles E.
• Granted Patent: US 7,065,233 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/503 : involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109

Abstract

Described herein is a technique for creating a 3D face model using images obtained from an inexpensive camera associated with a general-purpose computer. Two still images of the user are captured, and two video sequences. The user is asked to identify five facial features, which are used to calculate a mask and to perform fitting operations. Based on a comparison of the still images, deformation vectors are applied to a neutral face model to create the 3D model. The video sequences are used to create a texture map. The process of creating the texture map references the previously obtained 3D model to determine poses of the sequential video images.

Claims

1. One or more computer-readable media containing a program that is executable by a computer to create a 3D model of a face, the program comprising the following actions:

capturing at least two images of the face;

finding matched points in the two images based on locations identified by a human user on at least one of the images;

estimating rotation and translation of the face based on the matched points in the images; and

determining 3D positions of the matched points based on the estimated rotation and translation to create a 3D representation of the face.

2. One or more computer-readable media as recited in claim 1, further comprising:

fitting a neutral face model to the 3D representation by applying deformation vectors to the neutral face model.

3. One or more computer-readable media as recited in claim 1, wherein the identified locations are distinct facial features.

4. One or more computer-readable media as recited in claim 1, wherein the identified locations correspond to eyes, nose and mouth.

5. One or more computer-readable media as recited in claim 1, wherein the identified locations corresponding to one or more of the following:

eye corners, mouth ends, and nose tip.

6. One or more computer-readable media as recited in claim 1, the actions further comprising:

capturing a succession of 2D images containing the face in a range of poses;

determining poses of the face by performing the following actions for each 2D image in succession;

matching points of the object in the 2D image with corresponding points of a previous 2D image whose pose is already known, wherein the matched points of the images have corresponding 3D positions in fitted neutral face model, the 3D positions being determined by the poses of the images;

calculating a pose for the 2D image that minimizes differences between projections of 3D positions of matched points of the previous image onto the given image and the corresponding matched points of the given image.

7. A method to create a 3D model of a face, the method comprising the following actions:

capturing at least two images of the face;

finding matched points in the two images based on locations identified by a human user on at least one of the images;

estimating rotation and translation of the face based on the matched points in the images; and

determining 3D positions of the matched points based on the estimated rotation and translation to create a 3D representation of the face.

8. A method as recited in claim 7, further comprising:

fitting a neutral face model to the 3D representation by applying deformation vectors to the neutral face model;

9. A method as recited in claim 7, further comprising:

capturing a succession of 2D images containing the face in a range of poses;

determining poses of the face in each of the 2D images;

creating a texture map from the succession of 2D images in conjunction with the determined poses of the 2D images.

10. A method as recited in claim 7, wherein the identified locations correspond to one or more of the following:

eye corners, mouth ends, and nose tip.

11. A method as recited in claim 7, wherein the identified locations correspond to eye corners, mouth ends, and nose tip.

12. A device that creates a 3D model of a face, the device being configured to perform actions comprising:

capturing at least two images of the face;

finding matched points in the two images based on locations identified by a human user on at least one of the images;

estimating rotation and translation of the face based on the matched points in the images; and

determining 3D positions of the matched points based on the estimated rotation and translation to create a 3D representation of the face.

13. A device as recited in claim 12, further comprising:

fitting a neutral face model to the 3D representation by applying deformation vectors to the neutral face model.

14. A device as recited in claim 12, wherein the identified locations correspond to one or more of the following:

eye corners, mouth ends, and nose tip.

15. A device as recited in claim 12, wherein the identified locations correspond to eye corners, mouth ends, and nose tip.

16. A device as recited in claim 12, the actions further comprising:

capturing a succession of 2D images containing the face in a range of poses;

determining poses of the face by performing the following actions for each 2D image in succession;

matching points of the object in the 2D image with corresponding points of a previous 2D image whose pose is already known, wherein the matched points of the images have corresponding 3D positions in fitted neutral face model, the 3D positions being determined by the poses of the images;

calculating aose for the 2D image that minimizes differences between projections of 3D positions of matched points of the previous image onto the given image and the corresponding matched points of the given image.

17. A device as recited in claim 12, the actions further comprising:

capturing a succession of 2D images containing the face in a range of poses;

determining poses of the face by performing the following actions for each 2D image in succession;

matching points of the object in the 2D image with corresponding points of a previous 2D image whose pose is already known, wherein the matched points of the images have corresponding 3D positions in fitted neutral face model, the 3D positions being determined by the poses of the images;

calculating a pose for the 2D image that minimizes differences between projections of 3D positions of matched points of the previous image onto the given image and the corresponding matched points of the given image; and

creating a texture map from the succession of 2D images in conjunction with the calculated poses of the 2D images.

18. A device as recited in claim 12, the actions further comprising:

capturing a succession of 2D images containing the face in a range of poses; and

creating a texture map from the succession of 2D images.