Rapid computer modeling of faces for animation

• US 7,212,656 B2
• Filed: 01/26/2006
• Issued: 05/01/2007
• Est. Priority Date: 03/09/2000
• Status: Expired due to Term

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Zhengyou, Liu, Zicheng, Cohen, Michael F., Jacobs, Charles E.
• Primary Examiner(s): SHERALI, ISHRAT I
• Application Number:
  US11/275,755
• Publication Number:
  US 20060104491A1
• Time in Patent Office: 460 Days
• Field of Search: 382115-118, 382/154, 382/100, 382/103, 382/107, 382/190, 382/209, 382216-218, 382/285, 382/276, 382294-297, 382/236, 382162-165, 345418-428, 345581-589, 345473-475, 345/441, 348 141- 1416, 348 77- 78, 340 582- 583, 700/700, 700/704, 700719-723
• US Class Current: 382/118
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/503 : involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109

Abstract

Described herein is a technique for creating a 3D face model using images obtained from an inexpensive camera associated with a general-purpose computer. Two still images of the user are captured, and two video sequences. The user is asked to identify five facial features, which are used to calculate a mask and to perform fitting operations. Based on a comparison of the still images, deformation vectors are applied to a neutral face model to create the 3D model. The video sequences are used to create a texture map. The process of creating the texture map references the previously obtained 3D model to determine poses of the sequential video images.

Claims

1. A system capable of determining the pose of an object in a given 2D image relative to a 3D model of the object, comprising:

means for matching points of the object in the given image with corresponding points of a base image, wherein each matched point has a corresponding 3D position in the 3D model, the 3D positions being determined by the poses of the images, wherein the pose of the base image is known; and

means for calculating the pose of the given 2D image such that projections of 3D positions of matched points of the base image onto the given 2D image coincide approximately with the corresponding matched points of the given 2D image.

2. A system as recited in claim 1, wherein the calculating performed by the calculating means comprises minimizing differences between the projections of 3D positions of matched points of the base image onto the given 2D image and the corresponding matched points of the given 2D image.

3. A system as recited in claim 1, wherein the matching performed by the matching means comprises detecting and matching corners of the 2D images.

4. One or more computer-readable media containing code that is executable by a computer to determine poses of an object in a succession of 2D images, relative to a 3D model of the object, the code comprising:

code for matching points of the object in the current 2D image with corresponding points of a previous 2D image whose pose is already known, wherein the matched points of the previous 2D image has corresponding 3D positions in the 3D model;

code for calculating a pose for the current 2D image such that the projections of 3D positions of matched points of the previous 2D image onto the current 2D image coincide approximately with the corresponding matched points of the current 2D image.

5. One or more computer-readable media as recited in claim 4, wherein the matching comprises detecting and matching corners of the 2D images.

6. One or more computer-readable media as recited in claim 4, wherein the calculating comprises minimizing differences between the projections of 3D positions of matched points of the previous 2D image onto the current 2D image and the corresponding matched points of the current 2D image.

7. A user interface usable to determine poses of an object in a succession of 2D images, relative to a 3D model of the object, comprising:

means for matching points of the object in the current 2 D image with corresponding points of a previous 2D image whose pose is already known, wherein the matched points of the previous 2D image has corresponding 3D positions in the 3D model;

means for calculating a pose for the current 2D image such that the projections of 3D positions of matched points of the previous 2D image onto the current 2D image coincide approximately with the corresponding matched points of the current 2D image.

8. A user interface as recited in claim 7, wherein the means for matching comprises detecting and matching corners of the 2D images.

9. A user interface as recited in claim 7, wherein the means for calculating comprises minimizing differences between the projections of 3D positions of matched points of the previous 2D image onto the current 2D image and the corresponding matched points of the current 2D image.