Rapid computer modeling of faces for animation

• US 20020012454A1
• Filed: 01/04/2001
• Published: 01/31/2002
• Est. Priority Date: 03/09/2000
• Status: Active Grant

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Zhengyou, Liu, Zicheng, Cohen, Michael F., Jacobs, Charles E.
• Granted Patent: US 6,807,290 B2
• Time in Patent Office: Days
• US Class Current: 382/118
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/503 : involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109

Abstract

Described herein is a technique for creating a 3D face model using images obtained from an inexpensive camera associated with a general-purpose computer. Two still images of the user are captured, and two video sequences. The user is asked to identify five facial features, which are used to calculate a mask and to perform fitting operations. Based on a comparison of the still images, deformation vectors are applied to a neutral face model to create the 3D model. The video sequences are used to create a texture map. The process of creating the texture map references the previously obtained 3D model to determine poses of the sequential video images.

Claims

1. A method of processing two or more images to distinguish a head in the images from other portions of the images, comprising the following actions:

predicting portions of the images that form facial portions of the images;

calculating one or more skin colors by sampling the images at the predicted portions;

computing a first mask image that marks any image pixels whose colors are different in the two or more images;

creating a second mask image that marks any image pixels whose colors correspond to the calculated one or more skin colors;

combining the first and second mask images to create a final mask image.

2. A method as recited in claim 1, wherein the combining comprises intersecting.

3. A method as recited in claim 1, wherein the combining comprises joining.

4. A method as recited in claim 1, wherein the predicting is based on points supplied by a human user.

5. A method as recited in claim 1, wherein the predicting is based on point supplied by a human user, the points corresponding to a plurality of distinct facial features.

6. One or more computer-readable media containing a program that is executable by a computer to process two or more images for distinguishing a head in the images from other portions of the images, the program comprising the following actions:

identifying locations of a plurality of distinct facial features in the images;

predicting an outer area that corresponds to the head, based on the indicated locations of facial features;

predicting an inner area within the outer area that corresponds to a face portion of the head, based on the indicated locations of facial features;

calculating one or more skin colors by sampling the images at locations that are specified relative to the indicated locations of facial features;

creating a first mask image that marks any image pixels whose colors are different in the two images;

creating a second mask image that marks any image pixels whose colors correspond to the calculated one or more skin colors;

within the inner area, noting all of the marked pixels on the first mask image and also noting any unmarked pixels of the first mask image that correspond in location to marked pixels in the second mask image;

forming a final mask image that marks the noted pixels as being part of the head.

7. One or more computer-readable media as recited in claim 6, the actions further comprising:

predicting a lower area of the image that corresponds to a chin portion of the head;

within the lower area, noting any marked pixels in the first mask image that correspond in location to marked pixels in the second mask image.

8. One or more computer-readable media as recited in claim 6, wherein the identifying comprises accepting input from a human user.

9. One or more computer-readable media as recited in claim 6, wherein the identified locations correspond to eyes, nose and mouth.

10. One or more computer-readable media as recited in claim 6, wherein the identified locations comprise eye corners, mouth ends, and nose tip.

11. One or more computer-readable media as recited in claim 6, wherein the inner and outer areas are defined by inner and outer ellipses, and the outer ellipse is approximately 25% larger than the inner ellipse.

12. One or more computer-readable media as recited in claim 6, wherein:

the sizes of the ellipses are calculated based on spacial relationships of the identified locations.

13. One or more computer-readable media as recited in claim 6, wherein:

the identified locations comprise the inner eye corners and the mouth ends;

the inner and outer areas are defined by inner and outer ellipses; and

the sizes of the ellipses are calculated based on spacial relationships of the identified locations.

14. One or more computer-readable media as recited in claim 6, wherein:

the identified locations comprise the inner eye corners and the mouth ends;

the inner and outer areas are defined by inner and outer ellipses;

the width of the inner ellipse is five times the distance between the inner eye corners; and

the height of the inner ellipse is three times the vertical distance between the eye corners and the mouth ends.

15. One or more computer-readable media as recited in claim 6, wherein:

the identified locations comprise the inner eye corners and the mouth ends;

the inner and outer areas are defined by inner and outer ellipses;

the width of the inner ellipse is five times the distance between the inner eye corners;

the height of the inner ellipse is three times the vertical distance between the eye corners and the mouth ends; and

the outer ellipse is 25% larger than the inner ellipse.

16. One or more computer-readable media containing a program that is executable by a computer to create a three-dimensional model of a face, the program comprising the following actions:

constructing a three-dimensional face model from two or more two-dimensional face images, wherein the constructed face model comprises a plurality of points specified in three dimensions;

applying a combination of deformation vectors to a pre-specified face model to deform the pre-specified face model approximately to the constructed face model, wherein the pre-specified face model comprises a plurality points specified in three dimensions and each deformation vector specifies a plurality of displacements corresponding respectively to the plurality of points of the pre-specified face model.

17. One or more computer-readable media as recited in claim 16, wherein the applying action comprises calculating a linear combination of the deformation vectors and the pre-specified face model.

18. One or more computer-readable media as recited in claim 16, wherein the applying action comprises calculating a linear combination of the deformation vectors and the pre-specified face model, the actions further comprising:

multiplying each deformation vector by a respective multiplicative constant before when calculating the linear combination; and

selecting the multiplicative constants to minimize a summation of distances between the deformed face model and the points of the constructed face model.

19. One or more computer-readable media as recited in claim 16, wherein the applying action comprises calculating a linear combination of the deformation vectors and the pre-specified face model, the actions further comprising:

identifying one or more points of the constructed face model and corresponding points of the pre-specified face model;

multiplying each deformation vector by a respective multiplicative constant before when calculating the linear combination; and

selecting the multiplicative constants to minimize a summation of the distances between the identified points of the constructed face model and the corresponding points of the pre-specified face model.

20. One or more computer-readable media as recited in claim 16, wherein the applying action comprises calculating a linear combination of the deformation vectors and the pre-specified face model, the actions further comprising:

identifying one or more points of the constructed face model and corresponding points of the pre-specified face model;

multiplying each deformation vector by a respective multiplicative constant before when calculating the linear combination; and

selecting the multiplicative constants to minimize (a) a summation of the distances between the identified points of the constructed face model and the corresponding points of the pre-specified face model, and (b) a summation of the distances between the identified points of the constructed face model and the corresponding points of the pre-specified face model.

21. A method of creating a three-dimensional model of a face, comprising applying a combination of deformation vectors to a pre-specified face model to deform the pre-specified face model approximately to a reconstructed face model, wherein the pre-specified face model comprises a plurality of points specified in three dimensions and each deformation vector specifies a plurality of displacements corresponding respectively to the plurality of three dimensional points of the pre-specified face model.

22. A method as recited in claim 21, wherein the applying comprises calculating a linear combination of the deformation vectors and the pre-specified face model.

23. A method as recited in claim 21, wherein the applying comprises calculating a linear combination of the deformation vectors and the pre-specified face model, the actions further comprising:

multiplying each deformation vector by a respective multiplicative constant before when calculating the linear combination; and

selecting the multiplicative constants to minimize a summation of distances between the deformed face model and the points of the constructed face model.

24. A method as recited in claim 21, wherein the applying comprises calculating a linear combination of the deformation vectors and the pre-specified face model, the actions further comprising:

identifying one or more points of the constructed face model and corresponding points of the pre-specified face model;

multiplying each deformation vector by a respective multiplicative constant before when calculating the linear combination; and

selecting the multiplicative constants to minimize a summation of distances between the identified points of the constructed face model and the corresponding points of the pre-specified face model.

25. A method as recited in claim 21, wherein the applying comprises calculating a linear combination of the deformation vectors and the pre-specified face model, the actions further comprising:

identifying one or more points of the constructed face model and corresponding points of the pre-specified face model;

multiplying each deformation vector by a respective multiplicative constant before when calculating the linear combination; and

selecting the multiplicative constants to minimize (a) a summation of distances between the deformed face model and the points of the constructed face model, and (b) a summation of distances between the identified points of the constructed face model and the corresponding points of the pre-specified face model.

26. A method of creating a three-dimensional model of a face, comprising:

specifying a neutral face model in terms of three dimensional points;

specifying a plurality of deformation vectors, wherein each deformation vector indicates a plurality of displacements corresponding respectively to the points of the neutral face model, and wherein each deformation vector deforms the neutral face model in a particular way when applied to the neutral face model;

specifying degree of deformation for each deformation vector in terms of a multiplicative constant;

deforming the neutral face model by calculating a linear combination of the neutral face model and the deformation vectors, each deformation vector being multiplied by its multiplicative constant; and

selecting the multiplicative constants to minimize a summation of distances between the deformed face model and the points of the constructed face model.

27. A method as recited in claim 26, further comprising:

identifying one or more points of the constructed face model and corresponding points of the pre-specified face model;

further selecting the multiplicative constants to minimize a summation of distances between the identified points of the constructed face model and the corresponding points of the pre-specified face model.

28. A method as recited in claim 26, further comprising:

accepting input from a human user identifying one or more points of the constructed face model that correspond to particular points of the pre-specified face model;

further selecting the multiplicative constants to minimize a summation of distances between the identified points of the constructed face model and the corresponding points of the pre-specified face model.

29. A method of determining the pose of an object in a given 2D image relative to a 3D model of the object, comprising:

matching points of the object in the given image with corresponding points of a base image, wherein each matched point has a corresponding 3D position in the 3D model, the 3D positions being determined by the poses of the images, wherein the pose of the base image is known; and

calculating the pose of the given image such that projections of 3D positions of matched points of the base image onto the given image coincide approximately with the corresponding matched points of the given image.

30. A method as recited in claim 29 wherein the calculating comprises minimizing differences between the projections of 3D positions of matched points of the base image onto the given image and the corresponding matched points of the given image.

31. A method as recited in claim 29 wherein the calculating comprises:

summing the differences between the projections of 3D positions of matched points of the base image onto the given image and the corresponding matched points of the given image; and

calculating a pose that minimizes the summed differences.

32. A method as recited in claim 29 wherein the matching comprises detecting and matching corners of the images.

35. One or more computer-readable media as recited in claim 29 wherein the matching comprises detecting and matching corners of the images.

33. One or more computer-readable media containing a program that is executable by a computer to determine poses of an object in a succession of 2D images, relative to a 3D model of the object;

the program comprising the following actions, for each 2D image in succession;

matching points of the object in the 2D image with corresponding points of a previous 2D image whose pose is already known, wherein the matched points of the previous image has corresponding 3D positions in the 3D model;

calculating a pose for the 2D image that minimizes differences between the projections of 3D positions of matched points of the previous image onto the given image and the corresponding matched points of the given image.

34. One or more computer-readable media as recited in claim 33 wherein the calculating comprises:

summing the differences between the projections of 3D positions of matched points of the previous image onto the given image and the corresponding matched points of the given image; and

calculating the pose to minimize the summed differences.

36. A method for creating a 3D model of a face, comprising:

capturing at least two 2D images of the face;

identifying locations of a plurality of distinct facial features in the two images;

identifying head regions of the images, based on the identified positions of the plurality of distinct facial features;

matching points of the head regions between the images;

estimating rotation and translation of the head regions based on the matched points in the images;

determining 3D positions of the matched points based on the estimated rotation and translation to create a 3D representation of the face; and

fitting a neutral face model to the 3D representation by applying deformation vectors to the neutral face model.

37. A method as recited in claim 36, wherein the identifying comprises accepting input from a human user.

38. A method as recited in claim 36, wherein the identified locations correspond to eyes, nose and mouth.

39. A method as recited in claim 36, wherein the identified locations comprise eye corners, mouth ends, and nose tip.

40. A method as recited in claim 36, further comprising:

capturing a succession of 2D images containing the head in a range of poses;

determining poses of the head by performing the following actions for each 2D image in succession;

matching points of the object in the 2D image with corresponding points of a previous 2D image whose pose is already known, wherein the matched points of the images have corresponding 3D positions in fitted neutral face model, the 3D positions being determined by the poses of the images;

calculating a pose for the 2D image that minimizes differences between projections of 3D positions of matched points of the previous image onto the given image and the corresponding matched points of the given image.

41. A method as recited in claim 36, further comprising:

capturing a succession of 2D images containing the head in a range of poses;

determining poses of the face in each of the 2D images by performing the following actions for each 2D image in succession;

matching points of the object in the 2D image with corresponding points of a previous 2D image whose pose is already known, wherein the matched points of the images have corresponding 3D positions in fitted neutral face model, the 3D positions being determined by the poses of the images;

calculating a pose for the face of the 2D image that minimizes differences between the projections of 3D positions of matched points of the previous image onto the given image and the corresponding matched points of the given image

creating a texture map from the succession of 2D images in conjunction with the calculated poses of the 2D images.

42. One or more computer-readable media containing a program that is executable by a computer to create a 3D model of a face, the program comprising the following actions:

capturing at least two 2D images of the face;

identifying locations of a plurality of distinct facial features in the two images;

identifying head regions of the images, based on the identified positions of the plurality of distinct facial features;

matching points of the head regions between the images;

estimating rotation and translation of the head regions based on the matched points in the images;

determining 3D positions of the matched points based on the estimated rotation and translation to create a 3D representation of the face;

fitting a neutral face model to the 3D representation by applying deformation vectors to the neutral face model;

capturing a succession of 2D images containing the face in a range of poses;

determining poses of the face in each of the 2D images by performing the following actions for each 2D image in succession;

matching points of the object in the 2D image with corresponding points of a previous 2D image whose pose is already known, wherein the matched points of the images have corresponding 3D positions in fitted neutral face model, the 3D positions being determined by the poses of the images;

calculating a pose for the face of the 2D image that minimizes differences between the projections of 3D positions of matched points of the previous image onto the given image and the corresponding matched points of the given image; and

creating a texture map from the succession of 2D images in conjunction with the calculated poses of the 2D images.

43. One or more computer-readable media as recited in claim 42, wherein the identifying comprises accepting input from a human user.

44. One or more computer-readable media as recited in claim 42, wherein the identified locations correspond to eyes, nose and mouth.

45. One or more computer-readable media as recited in claim 42, wherein the identified locations comprise eye corners, mouth ends, and nose tip.