Rapid computer modeling of faces for animation

• US 7,149,329 B2
• Filed: 10/19/2004
• Issued: 12/12/2006
• Est. Priority Date: 03/09/2000
• Status: Expired due to Term

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Zhengyou, Liu, Zicheng, Cohen, Michael F., Jacobs, Charles E.
• Primary Examiner(s): Sherali, Ishrat
• Application Number:
  US10/968,553
• Publication Number:
  US 20050053277A1
• Time in Patent Office: 784 Days
• Field of Search: 382115-118, 382/154, 382/100, 382/103, 382/107, 382/190, 382/209, 382216-218, 382/282, 382/276, 382/288, 382/283, 382162-165, 345418-428, 345581-589, 345473-475, 345/441, 348 141- 1416, 348 77- 78, 340 582- 583
• US Class Current: 382/118
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/503 : involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109

Abstract

Described herein is a technique for creating a 3D face model using images obtained from an inexpensive camera associated with a general-purpose computer. Two still images of the user are captured, and two video sequences. The user is asked to identify five facial features, which are used to calculate a mask and to perform fitting operations. Based on a comparison of the still images, deformation vectors are applied to a neutral face model to create the 3D model. The video sequences are used to create a texture map. The process of creating the texture map references the previously obtained 3D model to determine poses of the sequential video images.

Claims

1. One or more computer-readable media containing instructions to process two or more images for distinguishing a head in the images from other portions of the images, the instructions being executable to perform actions comprising:

predicting portions of the images that form facial portions of the images;

calculating one or more skin colors by sampling the images at the predicted portions;

computing a first mask image that marks any image pixels whose colors are different in the two or more images;

creating a second mask image that marks any image pixels whose colors correspond to the calculated one or more skin colors;

combining the first and second mask images to create a final mask image.

2. One or more computer-readable media as recited in claim 1, wherein the combining comprises intersecting.

3. One or more computer-readable media as recited in claim 1, wherein the combining comprises joining.

4. One or more computer-readable media as recited in claim 1, wherein the predicting is based on points supplied by a human user.

5. One or more computer-readable media as recited in claim 1, wherein the predicting is based on point supplied by a human user, the points corresponding to a plurality of distinct facial features.

6. An apparatus configured to process two or more images for distinguishing a head in the images from other portions of the images, the apparatus being configured to perform actions comprising:

means for predicting portions of the images that form facial portions of the images;

means for calculating one or more skin colors by sampling the images at the predicted portions;

means for computing a first mask image that marks any image pixels whose colors are different in the two or more images;

means for creating a second mask image that marks any image pixels whose colors correspond to the calculated one or more skin colors;

means for combining the first and second mask images to create a final mask image.

7. An apparatus as recited in claim 6, wherein the means for combining comprises intersecting.

8. An apparatus as recited in claim 6, wherein the means for combining comprises joining.

9. An apparatus as recited in claim 6, wherein the means for predicting utilizes points supplied by a human user.

10. An apparatus as recited in claim 6, wherein the means for predicting utilizes points supplied by a human user, the points corresponding to a plurality of distinct facial features.

11. A method to process two or more images for distinguishing a head in the images from other portions of the images, the method comprising the following actions:

identifying locations of a plurality of distinct facial features in the images;

predicting an outer area that corresponds to the head, based on the indicated locations of facial features;

predicting an inner area within the outer area that corresponds to a face portion of the head, based on the indicated locations of facial features;

calculating one or more skin colors by sampling the images at locations that are specified relative to the indicated locations of facial features;

creating a first mask image that marks any image pixels whose colors are different in the two images;

creating a second mask image that marks any image pixels whose colors correspond to the calculated one or more skin colors;

within the inner area, noting marked pixels on the first mask image and also noting unmarked pixels of the first mask image that correspond in location to marked pixels in the second mask image;

forming a final mask image that marks the noted pixels as being part of the head.

12. A method as recited in claim 11, the actions further comprising:

predicting a lower area of the image that corresponds to a chin portion of the head;

within the lower area, noting marked pixels in the first mask image that correspond in location to marked pixels in the second mask image.

13. A method recited in claim 11, wherein the identifying comprises accepting input from a human user.

14. A method as recited in claim 11, wherein the identified locations correspond to eyes, nose and mouth.

15. A method as recited in claim 11, wherein the identified locations comprise eye corners, mouth ends, and nose tip.

16. A method as recited in claim 11, wherein the inner and outer areas are defined by inner and outer ellipses, and the outer ellipse is approximately 25% larger than the inner ellipse.

17. A method as recited in claim 11, wherein:

the sizes of the ellipses are calculated based on spacial relationships of the identified locations.

18. A method as recited in claim 11, wherein:

the identified locations comprise the inner eye corners and the mouth ends;

the inner and outer areas are defined by inner and outer ellipses; and

the sizes of the ellipses are calculated based on spacial relationships of the identified locations.

19. A method as recited in claim 11, wherein:

the identified locations comprise the inner eye corners and the mouth ends;

the inner and outer areas are defined by inner and outer ellipses;

the width of the inner ellipse is five times the distance between the inner eye corners; and

the height of the inner ellipse is three times the vertical distance between the eye corners and the mouth ends.

20. A method as recited in claim 11, wherein:

the identified locations comprise the inner eye corners and the mouth ends;

the inner and outer areas are defined by inner and outer ellipses;

the width of the inner ellipse is five times the distance between the inner eye corners;

the height of the inner ellipse is three times the vertical distance between the eye corners and the mouth ends; and

the outer ellipse is 25% larger than the inner ellipse.

21. An apparatus for processing two or more images for distinguishing a head in the images from other portions of the images, comprising:

means for identifying locations of a plurality of distinct facial features in the images;

means for predicting an outer area that corresponds to the head, based on the indicated locations of facial features;

means for predicting an inner area within the outer area that corresponds to a face portion of the head, based on the indicated locations of facial features;

means for calculating one or more skin colors by sampling the images at locations that are specified relative to the indicated locations of facial features;

means for creating a first mask image that marks any image pixels whose colors are different in the two images;

means for creating a second mask image that marks any image pixels whose colors correspond to the calculated one or more skin colors;

means for noting, within the inner area, marked pixels on the first mask image and unmarked pixels of the first mask image that correspond in location to marked pixels in the second mask image;

means for forming a final mask image that marks the noted pixels as being part of the head.

22. An apparatus as recited in claim 21, further comprising:

means for predicting a lower area of the image that corresponds to a chin portion of the head;

means for noting, within the lower area, marked pixels in the first mask image that correspond in location to marked pixels in the second mask image.

23. An apparatus as recited in claim 21, wherein the means for identifying comprises means for accepting input from a human user.

24. An apparatus as recited in claim 21, wherein the identified locations correspond to eyes, nose and mouth.

25. An apparatus as recited in claim 21, wherein the identified locations comprise eye corners, mouth ends, and nose tip.

26. An apparatus as recited in claim 21, wherein the inner and outer areas are defined by inner and outer ellipses, and the outer ellipse is approximately 25% larger than the inner ellipse.

27. An apparatus as recited in claim 21, wherein:

the sizes of the ellipses are calculated based on spacial relationships of the identified locations.

28. An apparatus as recited in claim 21, wherein:

the identified locations comprise the inner eye corners and the mouth ends;

the inner and outer areas are defined by inner and outer ellipses; and

the sizes of the ellipses are calculated based on spacial relationships of the identified locations.

29. An apparatus as recited in claim 21, wherein:

the identified locations comprise the inner eye corners and the mouth ends;

the inner and outer areas are defined by inner and outer ellipses;

the width of the inner ellipse is five times the distance between the inner eye corners; and

the height of the inner ellipse is three times the vertical distance between the eye corners and the mouth ends.

30. An apparatus as recited in claim 21, wherein:

the identified locations comprise the inner eye corners and the mouth ends;

the inner and outer areas are defined by inner and outer ellipses;

the width of the inner ellipse is five times the distance between the inner eye corners;

the height of the inner ellipse is three times the vertical distance between the eye corners and the mouth ends; and

the outer ellipse is 25% larger than the inner ellipse.