Facial image processing methods and systems

• US 20040201586A1
• Filed: 04/30/2004
• Published: 10/14/2004
• Est. Priority Date: 08/30/2000
• Status: Active Grant

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Corporation
• Inventors: Kang, Sing Bing, Olynyk, Kirk, Guenter, Brian K., Raghupathy, Sashi, Marschner, Stephen
• Granted Patent: US 7,356,447 B2
• Time in Patent Office: Days
• US Class Current: 345/426
• CPC Class Codes: G06T 15/506 Illumination models
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06V 10/60 relating to illumination properties, e.g. using a reflectance or lighting model
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

In the described embodiment, methods and systems for processing facial image data for use in animation are described. In one embodiment, a system is provided that illuminates a face with illumination that is sufficient to enable the simultaneous capture of both structure data, e.g. a range or depth map, and reflectance properties, e.g. the diffuse reflectance of a subject'"'"'s face. This captured information can then be used for various facial animation operations, among which are included expression recognition and expression transformation.

Claims

1. A computer-implemented facial image processing method comprising:

illuminating a subject'"'"'s head with one or more light sources that are selected to suppress specular reflection;

capturing digital images from a plurality of positions around the subject'"'"'s head while the subject'"'"'s head is illuminated;

computing an albedo map for each of the digital images; and

combining two or more of the computed albedo maps for the digital images to provide a single albedo map for the subject'"'"'s head.

2. The facial image processing method of claim 1, wherein the light sources provide polarized light.

3. The facial image processing method of claim 1, wherein said capturing comprises using a digital camera that has a complementary polarizer configured to remove the specularity.

4. The facial image processing method of claim 1, wherein said combining comprises computing a weighted average of individual albedo maps.

5. The facial image processing method of claim 4, wherein said computing of the weighted average comprises using a weighting function that gives higher weights to pixels that are viewed and/or illuminated from directions nearly normal to the surface of the subject.

6. The facial image processing method of claim 1, wherein said computing comprises:

for each pixel in a texture map;

computing a surface normal;

computing the irradiance;

computing the viewing direction; and

computing coordinates in image space; and

computing the Lambertian reflectance for one or more of the pixels.

7. The facial image processing method of claim 1, wherein said computing comprises, prior to computing an albedo for a particular pixel, verifying that the pixel is visible and suitably illuminated.

8. The facial image processing method of claim 7 further comprising designating each pixel as having different degrees of visibility and illumination and computing an albedo for a pixel only if the pixel is fully visible, fully illuminated by at least one light source, and not partially illuminated by any light source.

9. A facial image-processing system comprising:

a camera;

multiple light sources that produce light selected to suppress the specular reflection of a subject'"'"'s head that is viewed by the camera; and

an image processor configured to;

receive multiple digital images of a subject'"'"'s head that are produced by the camera;

compute an albedo map for each image;

combine albedo maps for all of the images to provide a single albedo map for the subject'"'"'s head.

10. The facial image processing system of claim 9, wherein the image processor combines the albedo maps by computing a weighted average of the individual albedo maps.

11. The facial image processing system of claim 10, wherein the image processor computes the weighted average of the individual albedo maps by using a weighting function that gives higher weights to pixels that are viewed and/or illuminated from directions nearly normal to the surface of the subject.