Automated face enhancement

• US 20070189627A1
• Filed: 02/14/2006
• Published: 08/16/2007
• Est. Priority Date: 02/14/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Cohen, Michael, Wang, Jue
• Granted Patent: US 7,634,108 B2
• Time in Patent Office: Days
• US Class Current: 382/254
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation

Abstract

An automated face enhancement system and process which can automatically improve faces in videos or other images by applying cosmetic effects, given only a small amount of user interaction for initialization. For an input video, the system will first track the face and eye locations, and classify pixels in the face into different facial components based on local color models. The classification results of different frames are temporally smoothed to ensure temporal coherence. A set of cosmetic filters are then applied to different facial components.

Claims

1. A process for enhancing the appearance of a face in an image in an image sequence, comprising the process actions of:

(a) identifying facial components in an initial image frame of the image sequence and determining local color models for each of these facial components;

(b) inputting a subsequent image frame of the image sequence as the current image frame;

(c) locating face and eye locations in the current image frame;

(d) classifying pixels of the face into different facial components using the local color models and the face and eye locations to produce an initial classification map for the current image frame;

(e) temporally smoothing the pixels of the initial classification map of the current image frame using a final classification map of the immediately previous image frame of the image sequence to create a final classification map of the current image frame; and

(f) applying cosmetic filters to facial components defined in the final classification map of the current image frame to produce an enhanced image of the face.

2. The process of claim 1 wherein process actions (b) through (f) are repeated on subsequent image frames of the image sequence.

3. The process of claim 1 wherein the process action for locating the face and eye location comprises;

locating the face with a face detector;

determining skin and non-skin regions in the location of the face; and

searching the non skin regions of the located face using an eye template for each eye to determine the location of the eyes.

4. The process of claim 3 wherein the process action for determining the skin and non-skin regions of the face comprises;

dividing the face location into small regions;

determining a mean color for each small region and using this mean color as a representative color for the corresponding small region;

calculating a skin color probability for each small region using global statistical skin color models;

sorting the small regions based on their skin color probabilities;

determining an upper probability threshold;

determining a lower probability threshold;

designating all pixels whose skin color probabilities are greater than the upper probability threshold as skin pixels;

designating all pixels whose skin color probabilities are less than the lower probability threshold as non-skin pixels;

designating all other pixels as unknown pixels;

training a positive Gaussian Mixture Model based on skin pixels;

training a negative Gaussian Mixture Model based on non-skin pixels;

re-examining every pixel of the face location using the positive and negative Gaussian Mixture Models to determine if that pixel is a skin or non-skin pixel.

5. The process of claim 1 wherein the process action of locating the eye locations comprises, for each eye, the process actions of:

defining an image patch that defines an eye template in an initial image frame;

for the eye template, computing a gray scale image, a horizontal edge map and a vertical edge map;

creating three one dimensional (1 D) signals of the eye template using the grayscale image, the horizontal edge map, and the vertical edge map;

inputting a subsequent frame as the current frame and for every possible eye location in the current frame extracting an eye image patch;

for each eye image patch in the current frame,

computing a gray scale image, a horizontal edge map and a vertical edge map;

creating three 1D signals of the each eye image patch using the grayscale image, the horizontal edge map, and the vertical edge map;

comparing the eye template with the eye image patch to determine the amount of correlation using the three 1D signals; and

choosing as the current eye location the current image patch that is most closely correlated to the eye template patch.

6. The process of claim 1 wherein the local color models are manually specified by a user for each of the facial components.

7. The process of claim 1 wherein the local color model for each facial component is trained using positive and negative Gaussian Mixture Models.

8. The process of claim 1 wherein the process action of temporally smoothing is performed using a moving averages regression technique.

9. The process of claim 1 wherein a given facial component comprises an eye, eyebrows, under eye shadows or lips.

10. The process of claim 1 wherein the process action of applying cosmetic filters is carried out by blending a target color with each pixel of a given facial component in the final classification map.

11. A computer-readable medium having computer-executable instructions for performing the process recited in claim 2.

12. A system for improving the visual quality of a face in an image frame, comprising:

a general purpose computing device;

a computer program comprising program modules executable by the general purpose computing device, wherein the computing device is directed by the program modules of the computer program to,

input an image frame of an image sequence;

determine the location of the face in the image frame;

determine the position of the eyes and mouth in the image frame;

classify each pixel of the face into face components using the location of the face and eyes and color models for each of the face components;

temporally smooth the classified pixels using classified pixels of a previous image frame; and

apply one or more cosmetic filters to the smoothed classified pixels to output an enhanced image frame.

13. The system of claim 12 further comprising a program module to bilaterally filter the input image prior to determining the location of the face.

14. The system of claim 12 wherein the location of the eyes is determined by:

defining an eye template;

for the eye template the one dimensional signals;

inputting a subsequent image frame and for every possible eye location in the subsequent frame extracting an eye image patch;

for each eye image patch in the current frame,

computing three 1D signals;

comparing the eye template with the eye image patch to determine the amount of correlation using the three 1D signals; and

choosing as the current eye location the eye image patch that is most closely correlated to the eye template.

15. The system of claim 12 wherein the one or more cosmetic filters are applied to a combination of the eyes, eye brows, lips or shadows around the eyes.

16. The system of claim 12 wherein the program module to apply one or more cosmetic filters to the smoothed classified pixels, comprises sub-modules to:

determine a filter strength weight based on a maximum filter strength and the probability that a pixel is classified as a given component;

blending each smoothed classified pixel with a target color for the given component using the filter strength weight.

17. The system of claim 16 wherein the target color is different for different facial components.

18. The system of claim 17 wherein:

for a facial component comprising an eye, the target color is the color of the eye after applying a sharpening filter;

for a facial component comprising an eyebrow, the target color is black;

for a facial component comprising lips, the target color is a constant pre-defined color;

for a skin region, the target color is an average color of skin; and

for an under eye region, the target color is an average color of the skin.

19. A computer-implemented process for enhancing a face in a video frame in a video sequence, comprising the process actions of:

describing color models for different components of the face;

inputting a video frame;

locating face and eye locations in the video frame;

classifying pixels of the face into different facial components for the video frame using the color models and the face and eye locations;

applying cosmetic filters to different facial components in the video frame to produce an enhanced image of the face.

20. The computer-implemented process of claim 19 further comprising:

temporally smoothing the classified pixels of the face with the classified pixel of the face in a previous frame prior to applying cosmetic filters to different facial components.