Face detection under varying rotation

• US 20020150280A1
• Filed: 02/21/2001
• Published: 10/17/2002
• Est. Priority Date: 12/04/2000
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.), Sony Electronics Inc. (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.), Sony Electronics Inc. (Sony Group Corp.)
• Inventors: Li, Pingshan
• Granted Patent: US 7,155,036 B2
• Time in Patent Office: Days
• US Class Current: 382/117
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation

Abstract

A feature based face recognition method and apparatus. Global amplitude and direction is calculated for regions defined by a sliding window over an input gray scale image. These values are low pass filtered and a search is carried out for candidates for an eye. Once an eye candidate is identified by comparing the amplitude with a threshold, the other eye is presumed to be located a fixed distance away at the angle of the eye. The mouth is then determined to be in a predetermined location perpendicular to the angle of the eyes. A face contour is estimated by defining two ellipses between which face contours having angles similar to the ellipse are found. Multiple detections are possible and those detections are merged by averaging. The image is resized and the process repeats until the image has been reduced to its minimum size.

Claims

1. A method of detecting a face in a matrix of pixels representing an image, the method comprising:

computing a vector having amplitude and direction for each of a plurality of sub-matrices, the sub-matrices having size estimated to be a facial feature size in the image;

inspecting each sub-matrix to determine if the amplitude exceeds a threshold, wherein sub-matrices having amplitude exceeding the threshold are deemed facial feature candidate sub-matrices,

establishing distance and angle criteria for relationships among candidate sub-matrices that match facial features; and

identifying sets of candidate sub-matrices matching the distance and angle criteria to detect the presence of a face in the image.

2. The method according to claim 1, further comprising low pass filtering the image prior to the inspecting.

3. The method according to claim 1, wherein a facial feature size is assumed to be M×

M pixels.

4. The method according to claim 1, wherein a facial feature candidate includes a first eye, and wherein a candidate for a second eye is assumed to be located approximately two feature sizes away along a direction associated with the candidate first eye.

5. The method according to claim 3, wherein a facial feature candidate includes a first eye, and wherein a candidate for a second eye is assumed to be located approximately 2M away along a direction associated with the candidate first eye.

6. The method according to claim 4, wherein a facial feature candidate includes a mouth, and wherein the mouth is assumed to be located approximately 3 feature sizes away from a center point between the two candidate eyes in a direction perpendicular to a direction defined by a line between the two candidate eyes.

7. The method according to claim 5, further comprising computing an eyeline distance L between the two candidate eyes.

8. The method according to claim 7, wherein a facial feature candidate includes a mouth, and wherein the mouth is assumed to be located approximately 1.5L away from a center point between the two candidate eyes in a direction perpendicular to a direction defined by a line between the two candidate eyes.

9. The method according to claim 6, wherein a facial feature candidate includes a face contour, and wherein:

the facial contour is assumed to be located between an inner and an outer ellipse, the inner and outer ellipses being centered at approximately the center point between the two candidate eyes; and

the inner ellipse having a vertex at approximately the location of the mouth candidate and covertices at approximately the candidate eyes.

10. The method according to claim 9, wherein the outer ellipse has a vertex at approximately 1.5 times the distance from the candidate mouth to the center point between the two candidate eyes and covertices at approximately equaling the distance between the two candidate eyes from the center point between the two candidate eyes, the vertex and covertices of the two ellipses being aligned.

11. The method according to claim 8, wherein a facial feature candidate includes a face contour, and wherein:

the facial contour is assumed to be located between an inner and an outer ellipse, the inner and outer ellipses being centered at approximately the center point between the two candidate eyes; and

the inner ellipse having a vertex at approximately the location of the mouth candidate and covertices at approximately the candidate eyes.

12. The method according to claim 11, wherein the outer ellipse has a vertex at approximately 1.5 times the distance from the center point between the two candidate eyes and the candidate mouth and covertices at approximately two times the feature size, the vertex and covertices of the two ellipses being aligned.

13. The method according to claim 1, further comprising determining that a plurality of sets of overlapping faces are detected, and merging the overlapping detections by finding an average location of the overlapped faces and defining a detected face as residing at the average location.

14. The method according to claim 1, further comprising:

resizing the matrix of pixels defining the image; and

repeating the computing, inspecting, establishing and identifying on the resized matrix of pixels.

15. The method according to claim 1, wherein the amplitude comprises a feature amplitude value.

17. The electronic storage medium according to claim 16, further comprising low pass filtering the image prior to the inspecting.

18. The electronic storage medium according to claim 17, wherein a facial feature size is assumed to be M×

M pixels, and wherein a facial feature candidate includes a first eye, and wherein a candidate for a second eye is assumed to be located approximately two feature sizes away along a direction associated with the first eye candidate.

19. The electronic storage medium according to claim 18, wherein a facial feature candidate includes a mouth, and wherein the mouth is assumed to be located a predetermined distance away from a center point between the two candidate eyes in a direction perpendicular to a direction defined by a line between the two candidate eyes.

20. The electronic storage medium according to claim 19, wherein a facial feature candidate includes a face contour, and wherein:

the facial contour is assumed to be located between an inner and an outer ellipse, the inner and outer ellipses being centered at approximately the center point between the two candidate eyes; and

the inner ellipse having a vertex at approximately the location of the mouth candidate and covertices at approximately the candidate eyes.

21. The electronic storage medium according to claim 20, further comprising determining that a plurality of sets of overlapping faces are detected, and merging the overlapping detections by finding an average location of the overlapped faces and defining a detected face as residing at the average location.

22. The electronic storage medium according to claim 16, wherein the amplitude comprises a feature amplitude value.

24. The apparatus according to claim 23 further comprising a low pass filter that filters the input image prior to the inspecting.

25. The apparatus according to claim 23, wherein a facial feature candidate includes a first eye, and wherein a candidate for a second eye is assumed to be located approximately two feature sizes away along a direction associated with the first candidate eye.

26. The apparatus according to claim 25, wherein a facial feature candidate includes a mouth, and wherein the candidate mouth is assumed to be located approximately 3 feature sizes away from a center point between the two candidate eyes in a direction perpendicular to a direction defined by a line between the two candidate eyes.

27. The apparatus according to claim 25, wherein a facial feature candidate includes a mouth, and wherein the mouth is assumed to be located approximately 1.5 times the distance between the two candidate eyes away from a center point between the two candidate eyes in a direction perpendicular to a direction defined by a line between the two candidate eyes.

28. The apparatus according to claim 27, wherein a facial feature candidate includes a face contour, and wherein:

the facial contour is assumed to be located between an inner and an outer ellipse, the inner and outer ellipses being centered at approximately the center point between the two candidate eyes; and

the inner ellipse having a vertex at approximately the location of the mouth candidate and covertices at approximately the candidate eyes.

29. The apparatus according to claim 27, wherein the outer ellipse has a vertex at approximately 1.5 times the distance from the candidate mouth to the center point between the two candidate eyes and covertices at approximately equaling the distance between the two candidate eyes from the center point between the two candidate eyes, the vertex and covertices of the two ellipses being aligned.

30. The apparatus according to claim 27, wherein a facial feature candidate includes a face contour, and wherein:

the facial contour is assumed to be located between an inner and an outer ellipse, the inner and outer ellipses being centered at approximately the center point between the two candidate eyes; and

the inner ellipse having a vertex at approximately the location of the mouth candidate and covertices at approximately the candidate eyes.

31. The apparatus according to claim 30, wherein the outer ellipse has a vertex at approximately 1.5 times the distance from the center point between the two candidate eyes and the candidate mouth and covertices at approximately two times the feature size, the vertex and covertices of the two ellipses being aligned.

32. The apparatus according to claim 23, wherein the processor further determines that a plurality of sets of overlapping faces are detected, and merging the overlapping detections by finding an average location of the overlapped faces and defining a detected face as residing at the average location.

33. The apparatus according to claim 23, wherein the processor further carries out a process of:

resizing the matrix of pixels defining the image; and

repeating the computing, inspecting, establishing and identifying on the resized matrix of pixels.

34. The apparatus according to claim 23, wherein the amplitude comprises a feature amplitude value.

36. The method according to claim 35, further comprising:

finding the second candidate facial feature;

computing a geometric relationship between the first facial feature and the second feature, the geometric relationship including a second direction; and

searching for a third candidate facial feature at a third location determined by the geometric relationship including the second direction.

37. The method according to claim 36, further comprising:

finding the third candidate facial feature;

creating a bounded geometric construct about the first, second and third features; and

searching for a facial contour within the bounds of the geometric construct.

38. The method according to claim 35, wherein the first and second candidate facial features comprise first and second eyes.

39. The method according to claim 35, wherein the first candidate facial feature comprises an eye and wherein the first direction comprises an eye direction.

40. The method according to claim 36, wherein the first and second candidate facial features comprise first and second candidate eyes, and wherein the geometric relationship includes an eyeline having length L equaling a distance between the first and second candidate eyes, and wherein the second direction comprises a direction of the eyeline.

41. The method according to claim 40, wherein the third candidate facial feature comprises a candidate mouth, and wherein the third location is situated a predetermined distance from the eyeline along a line perpendicular to the eyeline and passing through a center of the eyeline.

42. The method according to claim 37, wherein the bounded geometric construct comprises an inner and an outer ellipse, and wherein searching comprises searching for the facial contour between the inner and outer ellipse.

43. The method according to claim 42, wherein the searching further comprises searching for points between the inner and outer ellipse that have direction within a specified angle of tangent angles of the inner and outer ellipses.

44. The method according to claim 35, wherein the identifying comprises finding dark regions of the image.

45. The method according to claim 36, wherein the first and second candidate eyes are tested against the following criteria to determine that they are eye candidates:

D(θ

_lefteye, θ

_righteye)<

15°

;

A_lefteye<

α

·

A_middle;

A_righteye<

α

·

A_middle;

G_lefteye<

β

·

G_middle;

G_righteye<

β

·

G_middle;

where D is the distance function, θ

is an angle defining the direction, A is an amplitude of the candidate feature, G is a low pass filtered amplitude, and α and

β

are constants, and wherein the subscript middle denotes quantities associated with a midpoint between the two eyes.

46. The method according to claim 45, wherein α

is approximately equal to 2 and β

is approximately equal to ⅞

.

47. The method according to claim 40, wherein the first and second candidate eyes and the candidate mouth are tested against the following criteria to determine that they are eye candidates and mouth candidate:

D[θ

eyeline, θ

mouth]<

10°

A_mouth>

λ

·

(A_lefteye+A_righteye);

(G_mouth<

γ

·

G_rightface and G_mouth<

μ

·

G_leftface) or

(G_mouth<

γ

·

G_leftface and G_mouth<

μ

·

G_rightface);

G_lefteye<

η

·

G_leftface;

G_righteye<

η

·

G_rightface;

where G_leftface and G_rightface are face location points located L/2 directly below the eyes along a line perpendicular to the eyeline and where λ

, γ

, μ

, and η

are constants.

48. The method according to claim 47, wherein λ

is approximately ⅛

, λ

is approximately {fraction (8/7)}, μ

is approximately ⅞ and

η

is approximately ⅞

.

49. The method according to claim 43, a center point of the eyeline is (x_c, y_c) and the distance between the two eyes is 2r_eyes and wherein the inner ellipses are given by:

50. The method according to claim 49, wherein candidate points on the candidate face contour are expected to meet the criterion of:

D(θ

_face, θ

_ellipse)<

30°

where D is the distance function, the angle the face contour where the line crosses is denoted by θ

_face and the angle of the tangent line of the ellipse at the point where the line crosses θ

_ellipse.

51. The method according to claim 50, further comprising counting a number of points meeting the criterion and comparing the number to a threshold to determine existence of a face contour.

16. An electronic storage medium storing instructions which, when executed on a programmed processor, carry out a method of detecting a face in a matrix of pixels representing an image, the method comprising:

computing a vector having amplitude and direction for each of a plurality of sub-matrices, the sub-matrices having size estimated to be a facial feature size in the image;

inspecting each sub-matrix to determine if the amplitude exceeds a threshold, wherein sub-matrices having amplitude exceeding the threshold are deemed facial feature candidate sub-matrices;

establishing distance and angle criteria for relationships among candidate sub-matrices that match facial features; and

identifying sets of candidate sub-matrices matching the distance and angle criteria to detect the presence of a face in the image.

23. A face detection apparatus for detecting a face in a matrix of pixels representing an image, comprising:

input means for receiving an input image;

a programmed processor carrying out a process of;

computing a vector having amplitude and direction for each of a plurality of sub-matrices, the sub-matrices having size estimated to be a facial feature size in the image;

inspecting each sub-matrix to determine if the amplitude exceeds a threshold, wherein sub-matrices having amplitude exceeding the threshold are deemed facial feature candidate sub-matrices;

establishing distance and angle criteria for relationships among candidate sub-matrices that match facial features; and

identifying sets of candidate sub-matrices matching the distance and angle criteria to detect the presence of a face in the image.

35. A method of face detection in an image, comprising:

identifying a first candidate facial feature with a feature size;

computing a first direction associated with the first candidate facial; and

searching for a second candidate facial feature at a first location determined in part by the first direction.