Facial feature detection on mobile devices

• US 20070154096A1
• Filed: 12/31/2005
• Published: 07/05/2007
• Est. Priority Date: 12/31/2005
• Status: Active Grant

• Current Assignee: ArcSoft, Inc.
• Original Assignee: ArcSoft, Inc.
• Inventors: Wang, Jin, Cao, Jiangen, Li, Shu, Zhang, Huaqi
• Granted Patent: US 7,643,659 B2
• Time in Patent Office: Days
• US Class Current: 382/190
• CPC Class Codes: G06V 10/446 using Haar-like filters, e.g. using integral image techniques
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

Locating an eye includes generating an intensity response map by applying a 3-rectangle filter and applying K-mean clustering to the map to determine the eye. Locating an eye corner includes applying logarithm transform and grayscale stretching to generate a grayscale eye patch, generating a binary map of the patch by using a threshold based on a histogram of the patch, and estimating the eye corner by averaging coordinates weighted by minimal eigenvalues of spatial gradient matrices in a search region based on the binary map. Locating a mouth corner includes generating another intensity response map and generating another binary map using another threshold based on another histogram of the intensity response map. Locating a chin or a cheek includes applying angle constrained gradient analysis to reject locations that cannot be the chin or cheek. Locating a cheek further includes removing falsely detected cheeks by parabola fitting curves through the cheeks.

Claims

1. A method for locating facial features on a facial area, comprising:

locating an eye location, comprising;

generating an intensity response map by applying a 3-rectangle filter to pixels in an eye slice from the facial area;

applying K-mean clustering to the intensity response map to determine the eye location;

locating an eye corner location, comprising;

preprocessing pixels in an eye patch from the facial area by applying logarithm transform and grayscale stretching to generate a grayscale eye patch;

generating a binary map by binarizing the grayscale eye patch using a threshold based on a histogram of the grayscale eye patch;

estimating the eye corner location by averaging coordinates weighted by minimal eigenvalues of spatial gradient matrices in an eye corner search region based on the binary map;

locating a mouth corner location, comprising;

generating another intensity response map by applying another 3-rectangle filter to pixels in a mouth patch from the facial area;

generating another binary map by binarizing the another intensity response map using another threshold based on another histogram of the another intensity response map;

locating a chin location, comprising;

applying angle constrained gradient analysis to reject locations in a chin search region from the facial area that cannot be the chin location;

locating a cheek location, comprising;

applying another angle constrained gradient analysis to reject locations in a cheek search region from the facial area that cannot be the cheek location;

removing falsely detected cheek location candidates by parabola fitting curves through the chin location and cheek location candidates.

2. The method of claim 1, wherein said locating an eye location further comprises:

splitting the eye slice based on the eye locations to form eye patches;

applying an AdaBoost filter to the eye patches to identify eye candidates;

for each eye patch;

grouping eye candidates into groups based on their locations, each group having a group candidate;

for each group, calculating a sum of minimum differences to a threshold of the AdaBoost filter of the eye candidates in said each group; and

selecting a group candidate of a group with a largest sum of minimum differences as the eye.

3. The method of claim 2, after said splitting the eye slice and before said applying an AdaBoost filter, further comprising:

generating a mirror image of one of the eye patches, wherein the AdaBoost filter is trained with only samples of one of a right eye and a left eye.

4. The method of claim 2, wherein in said locating an eye location, said applying a 3-rectangle filter comprises:

for each pixel in the eye slice, calculating a sum of (1) a first difference between corresponding pixel intensities in a middle rectangle and a top rectangle in the facial area, and (2) a second difference between corresponding pixel intensities in the middle rectangle and a bottom rectangle in the facial area, wherein the top rectangle is atop the middle rectangle and the middle rectangle is atop the bottom rectangle.

5. The method of claim 4, wherein said locating an eye location further comprises:

creating an integral image of the color image, wherein said calculating a sum comprises using the integral image to calculate the sum.

6. The method of claim 1, wherein said locating an eye location further comprises:

estimating the eye slice on the facial area, comprising;

ULx=W*EyeX−

W*EyeW/2,
ULy=H*EyeY−

H*EyeH/2,
DRx=W*(1−

EyeX)+W*EyeW/2,
DRy=H*EyeY+H*EyeH/2,

where “

W” and

“

H”

are respective width and height of the facial area normalized to 1, “

ULx” and

“

ULy”

are respective x and y coordinates of a top left corner of the eye slice, “

EyeW”

is a width of one eye set equal to 0.45, “

EyeH”

is a height of one eye set equal to 0.3, “

EyeX”

is an x coordinate of a center of a left eye set equal to 0.315, “

EyeY”

is a y coordinate of the center of the left eye set equal to 0.344, and “

DRx” and

“

DRy”

are respective x and y coordinates of a bottom right corner of the eye slice.

7. The method of claim 1, wherein in said locating an eye corner location, said applying logarithm transform comprises:

Y(x)=log(I(x)+1),

where “

x”

represents a pixel, and “

I” and

“

Y”

represent the intensities before and after the transformation, respectively.

8. The method of claim 1, wherein in said locating an eye corner location, said applying grayscale stretching comprises:

determining a lower bound “

L”

intensity value as follows;

$L=\underset{i}{\mathrm{argmin}}<br><br>\left\{\underset{i}{\sum <br><br>}<br><br>H<br><br>\left(i\right)><br><br>N*R\right\},$

determining an upper bound “

H”

intensity value as follows;

$H=\underset{i}{\mathrm{argmin}}<br><br>\left\{\underset{i}{\sum <br><br>}<br><br>H<br><br>\left(i\right)><br><br>N*\left(1-R\right)\right\},$

determining a new intensity value “

I”

as follows;

I′

(x)=(Y′

(x)−

L)*255/(H−

L),

where “

H(i)”

is a histogram value, “

N”

is a number of pixels in the eye patch, “

i”

represents an intensity value, “

arg min”

is a minimum of all values that meet a condition specified between { }, “

R”

is a grayscale stretching ratio, and “

x”

represents coordinates of a pixel.

9. The method of claim 1, wherein in said locating an eye corner location, said binarizing the grayscale eye patch comprises:

computing the histogram of the grayscale eye patch;

determining a threshold for said binarization, comprising;

$\mathrm{thr}=\underset{i}{\mathrm{argmin}}<br><br>\left\{\underset{j<<br><br>=i}{\sum <br><br>}<br><br>H<br><br>\left(j\right)><br><br>=p*N\right\},$

where “

i”

is an intensity value ranging from 0 to 255, “

j”

is an intensity value ranging from 0 to 255, “

H(j)”

is a histogram value at the intensity j, “

p”

is a percentage, and “

N”

is a number of pixels in the grayscale eye patch.

10. The method of claim 1, wherein in said locating an eye corner location, said estimating the eye corner location comprises:

defining an eye corner search region on the grayscale eye patch;

for each pixel in the search region, calculating a spatial gradient matrix;

for each spatial gradient matrix, calculating a minimum eigenvalue;

determining a maximum of minimum eigenvalues of spatial gradient matrices;

setting a threshold proportional to the maximum of minimum eigenvalues;

for each minimum eigenvalue that is less than the threshold, incrementing (1) coordinates of the eye corner location by coordinates of a corresponding pixel of the minimum eigenvalue, and (2) a number of pixels having eigenvalues greater than the threshold; and

dividing the coordinates of the eye corner location by the number of pixels.

11. The method of claim 10, wherein the eye corner search region is centered at one of a left most and a right most position of a white region in the binary map, said defining eye corner search region comprises:

w=Elen*0.2,
h=Elen*0.3,

where “

w” and

“

h”

are respective width and the height of the eye corner search region, and “

ELen”

is a distance between two eye locations.

12. The method of claim 1, wherein said locating a mouth corner location, prior to said generating another intensity response map, further comprising:

estimating a mouth search region;

applying an AdaBoost filter to the mouth search region to identify mouth candidates;

grouping the mouth candidates into groups based on their locations, each group having a group candidate;

for each group, calculating a sum of minimum differences to a threshold of the AdaBoost filter of the mouth candidates in said each group; and

selecting a group candidate of a group with a largest sum of minimum differences as the mouth patch.

13. The method of claim 12, wherein said estimating a mouth search region comprises:

L=W*0.15,
R=W*0.85,
U=Elen*0.5,
D=Elen*1.8,

where “

L”

is a left margin from the mouth search region to a left edge of the facial area, “

W”

is a width of the facial area, “

R”

is a right margin from the mouth search region to a right edge of the facial area, “

U”

is a distance from a midpoint of two eye locations to the top of the mouth search region, “

Elen”

is the distance between the two eye locations, and “

D”

is a distance from the midpoint to the bottom of the mouth search region.

14. The method of claim 1, wherein in said locating a mouth corner location, said applying another 3-rectangle filter comprises:

for each pixel in the mouth patch, calculating a sum of (1) a first difference between corresponding pixel intensities in a middle rectangle and a top rectangle in the facial area, and (2) a second difference between corresponding pixel intensities in the middle rectangle and a bottom rectangle in the facial area, wherein the top rectangle is atop the middle rectangle and the middle rectangle is atop the bottom rectangle.

15. The method of claim 14, wherein said locating a mouth corner location further comprises:

creating an integral image of the color image, wherein said calculating a sum comprises using the integral image to calculate the sum.

16. The method of claim 1, wherein said locating a mouth corner location further comprises:

applying gray level erosion to the mouth patch prior to said applying another 3-rectangle filter.

17. The method of claim 1, wherein in said locating a mouth corner location, said binarizing the another intensity response map comprises:

computing the histogram of the another intensity response map;

determining a threshold for said binarization the another intensity response map, comprising;

$\mathrm{thr}=\underset{i}{\mathrm{argmin}}<br><br>\left\{\underset{j<<br><br>=i}{\sum <br><br>}<br><br>H<br><br>\left(j\right)><br><br>=p*N\right\},$

where “

i”

is an intensity value ranging from 0 to 255, “

j”

is an intensity value ranging from 0 to 255, “

H(j)”

is a histogram value at the intensity j, “

p”

is a percentage, and “

N”

is a number of pixels in the grayscale eye patch.

18. The method of claim 1, wherein in said locating a chin location, said applying angle constrained gradient analysis comprises:

estimating the chin search region;

generating a first scalar map of the chin search region by calculating angled constrained gradients in a first color space;

generating a second scalar map of the chin search region by calculating angled constrained gradients in a second color space;

merging the first and the second scalar maps to form a merged scalar map;

generating a score map by weighting pixel values from the merged scalar map and the chin search region; and

selecting a location of a pixel having a largest score from the score map as the chin location.

19. The method of claim 18, wherein said estimating the chin search region comprises:

setting left and the right boundaries of the chin search region by a width of two mouth corners;

setting top and the bottom boundaries of the chin search region as follows;

U=Elen*0.45,
D=Elen*0.85,

where “

U”

is a distance from the two mouth corners to the top boundary of the chin search region, “

D”

is a distance from the two mouth corners to the bottom boundary of the chin search region, and “

Elen”

is the distance between two eye locations.

20. The method of claim 18, wherein said merging the first and the second scalar maps comprises selecting larger of corresponding values in the first and the second scalar maps to form the merged scalar map.

21. The method of claim 18, wherein said generating a first scalar map comprises:

initializing the first scalar map;

for each pixel in the chin search region;

computing a gradient in Y space;

computing a benchmark vector from a nose tip to said each pixel;

determining an angle between the gradient and the benchmark vector;

determining if the angle is less than a threshold;

if the angle is less than a threshold, projecting the gradient in the direction of the benchmark vector to determine a value of a scalar gradient at said each pixel.

22. The method of claim 18, wherein said generating a second scalar map comprises:

initializing the second scalar map;

for each pixel in the chin search region;

computing a gradient in Cr−

Cb space;

computing a benchmark vector from a nose tip to said each pixel;

determining an angle between the gradient and the benchmark vector;

determining if the angle is less than a threshold;

if the angle is less than a threshold, projecting the gradient in the direction of the benchmark vector to determine a value of a scalar gradient at said each pixel.

23. The method of claim 18, wherein said locating a chin location further comprises:

blurring the chin search region prior to said generating a first scalar map of the chin search region.

24. The method of claim 1, wherein in said locating a cheek location, said applying angle constrained gradient analysis comprises:

estimating the cheek search region;

generating a first scalar map of the cheek search region by calculating angled constrained gradients in a first color space;

generating a second scalar map of the cheek search region by calculating angled constrained gradients in a second color space;

merging the first and the second scalar maps to form a merged scalar map;

generating a score map by weighting pixel values from the merged scalar map and the cheek search region;

selecting a location of a pixel having a largest score from the score map as the cheek location.

25. The method of claim 24, wherein in said locating a cheek location, said estimating the check search region comprises:

H=0.1*NEdis,
L=0.3*Elen,
R=0.85*Elen,

where “

H”

is a vertical distance between a midpoint of two mouth corners to a center of the cheek search region, “

NEdis”

is a vertical distance between the midpoint of the two mouth corners and a midpoint between two eye locations, “

L”

is a horizontal distance from one mouth corner to a left boundary of the check search region, “

Elen”

is a distance between the two eye locations, and “

R”

is a horizontal distance from the one mouth corner to a right boundary of the check search region.

26. The method of claim 24, wherein said merging the first and the second scalar maps comprises selecting larger of corresponding values in the first and the second scalar maps to form the merged scalar map.

27. The method of claim 24, wherein said generating a first scalar map comprises:

initializing the first scalar map;

for each pixel in the cheek search region;

computing a gradient in Y space;

computing a benchmark vector from a nose tip to said each pixel;

determining an angle between the gradient and the benchmark vector;

determining if the angle is less than a threshold;

if the angle is less than a threshold, projecting the gradient in the direction of the benchmark vector to determine a value of a scalar gradient at said each pixel.

28. The method of claim 24, wherein said generating a second scalar map comprises:

initializing the second scalar map;

for each pixel in the cheek search region;

computing a gradient in Cr−

Cb space;

computing a benchmark vector from a nose tip to said each pixel;

determining an angle between the gradient and the benchmark vector;

determining if the angle is less than a threshold;

if the angle is less than a threshold, projecting the gradient in the direction of the benchmark vector to determine a value of a scalar gradient at said each pixel.

29. The method of claim 1, in said locating a cheek location, said removing falsely detected cheek locations comprises:

(a) for a cheek location candidate, fitting a parabola of an order using the chin location and the cheek location candidate as points in the parabola;

(b) for each point in the parabola, computing a fitting score comprising an absolute value of projection of a gradient vector of the point along a normal vector of the parabola at said each point;

(c) summing fitting scores for points in the parabola;

(d) calculating an averaged fitting score by averaging the fitting scores by a number of the points in the parabola;

(e) repeating steps (a), (b), (c), (d) for at least another order;

(f) repeating steps (a), (b), (c), (d), and (e) for at least another cheek location candidate;

(g) selecting one of the cheek location candidates with a maximum averaged fitting score as the cheek location.