Recovering parameters from a sub-optimal image

• US 8,009,880 B2
• Filed: 05/11/2007
• Issued: 08/30/2011
• Est. Priority Date: 05/11/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wang, Yang, Zhang, Zhengyou, Liu, Zicheng, Hua, Gang
• Primary Examiner(s): Ahmed; Samir A
• Assistant Examiner(s): HU, FRED H.
• Application Number:
  US11/747,695
• Publication Number:
  US 20080279423A1
• Time in Patent Office: 1,572 Days
• Field of Search: None
• US Class Current: 382/118
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/60 relating to illumination properties, e.g. using a reflectance or lighting model
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

A subregion-based image parameter recovery system and method for recovering image parameters from a single image containing a face taken under sub-optimal illumination conditions. The recovered image parameters (including albedo, illumination, and face geometry) can be used to generate face images under a new lighting environment. The method includes dividing the face in the image into numerous smaller regions, generating an albedo morphable model for each region, and using a Markov Random Fields (MRF)-based framework to model the spatial dependence between neighboring regions. Different types of regions are defined, including saturated, shadow, regular, and occluded regions. Each pixel in the image is classified and assigned to a region based on intensity, and then weighted based on its classification. The method decouples the texture from the geometry and illumination models, and then generates an objective function that is iteratively solved using an energy minimization technique to recover the image parameters.

Claims

1. A method for processing an image containing a face, comprising:

using a computing device to perform the following;

dividing a portion of the image containing the face into a plurality of regions;

generating an albedo morphable model for each of the plurality of regions;

using smoothness constraints to model spatial dependence between neighboring ones of the plurality of regions; and

recovering image parameters associated with the image using each of the albedo morphable models.

2. The method of claim 1, wherein the recovered image parameters include at least one of:

(a) an albedo of the portion of the image containing the face;

(b) a surface normal of the portion of the image containing the face;

(c) an illumination parameter of the portion of the image containing the face.

3. The method of claim 1, further comprising:

detecting face feature points in the image using a face feature detection method to obtain a set of detected feature points;

obtaining pre-selected feature points from the 3D morphable models; and

recovering pose parameters and shape parameters using the set of detected feature points and the pre-selected feature points to estimate an initial shape of the face.

4. The method of claim 3, further comprising using the initial shape estimate of the face to divide the portion of the image containing the face into a plurality of regions.

5. The method of claim 4, wherein dividing the portion of the image containing the face into a plurality of regions further comprises:

identifying a following type of regions in the image;

(a) a saturated region;

(b) a shadow region;

(c) a regular region;

(d) an occluded region; and

dividing the image into the plurality of regions based on the type of region identified.

6. The method of claim 5, further comprising dividing each of the regular regions into a plurality of regular subregions.

7. The method of claim 5, further comprising:

compute a constant albedo scale factor by averaging intensity values of the image; and

estimating an initial illumination coefficient using the constant albedo scale factor and the initial shape estimate of the face.

8. The method of claim 7, further comprising computing an initial albedo at each pixel in the portion of the image containing the face.

9. The method of claim 8, further comprising constructing an objective function that is an energy minimization problem containing the illumination parameter, the albedo, texture Principal Component Analysis (PCA) coefficients, shape PCA coefficients, and a surface normal.

10. The method of claim 9, further comprising:

fixing the illumination parameter and solving for the albedo, the texture PCA coefficients, the shape PCA coefficients, and the surface normal; and

fixing the albedo and the surface normal and solving for the illumination parameter.

11. A computer-readable storage medium having computer-executable instructions stored thereon for changing an illumination condition of an image containing a face, comprising:

inputting the image that was taken under harsh lighting conditions and identifying a portion of the image containing the face as a face image;

segmenting the face image into a plurality of regions;

generating an albedo morphable model for each of the plurality of regions;

holding an illumination constant and solving for albedo and geometry in an iterative manner;

generating a geometry morphable model for each of the plurality of regions such that the geometry morphable model and the albedo morphable model are two separate morphable models; and

holding the albedo and geometry constant and solving for illumination in an iterative manner.

12. The computer-readable storage medium of claim 11, further comprising determining that a first set of morphable model coefficients for a first region are within a variance from a second set of morphable model coefficients for a second region that is neighboring region of the first region.

13. The computer-readable storage medium of claim 12, further comprising computing the variance using exemplar texture data in the albedo morphable models of the first region and the second region.

14. The computer-readable storage medium of claim 13, further comprising allowing a smaller difference between values of the first set of morphable model coefficients and the second set of morphable model coefficients when the variance is small and allowing a larger difference between the values if the variance is large.

15. The computer-readable storage medium of claim 14, wherein segmenting the face image into a plurality of regions further comprises identifying saturated regions, shadow regions, regular regions, and occluded regions in the face image.

16. A computer-implemented process for improving an illumination of a single image containing a face and taken under sub-optimal illumination conditions, comprising:

computing an initial shape estimation of the face to generate a face image, wherein the face image is the portion of the image containing the face;

classifying each pixel in the face image as one of;

(a) a saturated pixel;

(b) a shadow pixel;

(c) a regular pixel;

(d) an occluded pixel;

weighting each pixel in the face image based on its classification;

assigning each pixel to one of a plurality of regions in the face image based on the pixel classification;

generating using the computer an albedo morphable model for each of the plurality of regions;

generating using the computer a geometry morphable model for each of the plurality of regions such that the geometry morphable model and the albedo morphable model are two separate morphable models; and

obtaining image parameters from the morphable models that are used to improve the illumination of the single image containing the face.

17. The computer-implemented process of claim 16, wherein classifying each pixel in the face image further comprises:

determining an intensity value for each pixel; and

classifying each pixel based on its intensity value.

18. The computer-implemented process of claim 17, further comprising modeling spatial dependence between neighboring regions of the plurality of regions using smoothness constraints.

19. The computer-implemented process of claim 18, further comprising:

computing an albedo scale factor by averaging intensity values for the face image;

estimating an initial illumination coefficient at each pixel based on the albedo scale factor; and

computing an initial albedo at each pixel using the corresponding initial illumination coefficient for that pixel.

20. The computer-implemented process of claim 19, further comprising:

using an iterative minimization technique to obtain the image parameters that include an albedo and an illumination coefficient; and

changing the illumination coefficient to improve illumination of the image while changing the albedo to preserve a quality of the image.