Facial image processing methods and systems

• US 7,356,447 B2
• Filed: 04/30/2004
• Issued: 04/08/2008
• Est. Priority Date: 08/30/2000
• Status: Expired due to Term

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Corporation
• Inventors: Kang, Sing Bing, Olynyk, Kirk, Guenter, Brian K., Raghupathy, Sashi, Marschner, Stephen
• Primary Examiner(s): Jones; Hugh
• Application Number:
  US10/836,281
• Publication Number:
  US 20040201586A1
• Time in Patent Office: 1,439 Days
• Field of Search: 703/2, 382/275, 382/294
• US Class Current: 703/2
• CPC Class Codes: G06T 15/506 Illumination models
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06V 10/60 relating to illumination properties, e.g. using a reflectance or lighting model
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

In the described embodiment, methods and systems for processing facial image data for use in animation are described. In one embodiment, a system is provided that illuminates a face with illumination that is sufficient to enable the simultaneous capture of both structure data, e.g. a range or depth map, and reflectance properties, e.g. the diffuse reflectance of a subject'"'"'s face. This captured information can then be used for various facial animation operations, among which are included expression recognition and expression transformation.

Claims

1. A computer-implemented facial image processing method comprising:

illuminating a subject'"'"'s head with one or more light sources that are selected to suppress specular reflection;

capturing digital images from a plurality of positions wound the subject'"'"'s head while the subject'"'"'s head is illuminated, each of the digital images representing the subject'"'"'s head at an angle from which the digital image is captured at one of the plurality of positions;

computing a surface normal to form a three-dimensional surface for each of the digital images;

computing an albedo map based on the computed three-dimensional surface for each of the digital images, the albedo map each providing a reflectance for each point of the three-dimensional surface that represents the subject'"'"'s head; and

combining two or more of the computed albedo maps for the digital images to provide a single albedo map for the subject'"'"'s head, the combining comprising at least combining two or more three-dimensional surfaces to form a complete three-dimensional surface and combining two or more albedo maps for the corresponding three-dimensional surfaces to form a complete albedo map for the complete three-dimensional surface of the subject'"'"'s head.

2. The facial image processing method of claim 1, wherein the light sources provide polarized light.

3. The facial image processing method of claim 1, wherein said capturing comprises using a digital camera that has a complementary polarizer configured to remove the specularity.

4. The facial image processing method of claim 1, wherein said combining comprises computing a weighted average of individual albedo maps.

5. The facial image processing method of claim 4, wherein said computing of the weighted average comprises using a weighting function that gives higher weights to pixels that are viewed and/or illuminated from directions nearly normal to the surface of the subject.

6. The facial image processing method of claim 1, wherein said computing comprises:

for each pixel in a texture map;

computing a surface normal;

computing the irradiance;

computing the viewing direction; and

computing coordinates in image space; and

computing the Lambertian reflectance for one or more of the pixels.

7. The facial image processing method of claim 1, wherein said computing comprises, prior to computing an albedo for a particular pixel, verifying that the pixel is visible and suitably illuminated.

8. The facial image processing method of claim 7 further comprising designating each pixel as having different degrees of visibility and illumination and computing an albedo for a pixel only if the pixel is fully visible, fully illuminated by at least one light source, and not partially illuminated by any light source.

9. A facial image-processing system comprising:

a camera;

multiple light sources that produce light selected to suppress the specular reflection of a subject'"'"'s head that is viewed by the camera; and

an image processor configured to;

receive multiple digital images of a subject'"'"'s head that are produced by the camera, each of the multiple digital images representing the subject'"'"'s head at an angle from which the digital image is captured;

compute a surface normal to form a three-dimensional surface for each of the digital images;

compute an albedo map based on the computed three-dimensional surface for each image, the albedo map each providing a reflectance for each point of the three-dimensional surface that represents the subject'"'"'s head;

combine albedo maps for all of the images to provide a single albedo map for the subject'"'"'s head, wherein to combine albedo maps for all of the images comprises to combine each of the computed three-dimensional surfaces to form a complete three-dimensional surface and to combine each of the computed albedo maps for the corresponding three-dimensional surface to form a complete albedo map for the complete three-dimensional surface of the subject'"'"'s head.

10. The facial image processing system of claim 9, wherein the image processor combines the albedo maps by computing a weighted average of the individual albedo maps.

11. The facial image processing system of claim 10, wherein the image processor computes the weighted average of the individual albedo maps by using a weighting function that gives higher weights to pixels that are viewed and/or illuminated from directions neatly normal to the surface of the subject.