Method of and apparatus for detecting a human face and observer tracking display

• US 6,633,655 B1
• Filed: 08/30/1999
• Issued: 10/14/2003
• Est. Priority Date: 09/05/1998
• Status: Expired due to Fees

• Current Assignee: Sharp Kabushiki Kaisha (Hon Hai Precision Industry Co., Ltd.)
• Original Assignee: Sharp Kabushiki Kaisha (Hon Hai Precision Industry Co., Ltd.)
• Inventors: Holliman, Nicolas Steven, Hong, Qi He, Ezra, David
• Primary Examiner(s): CHANG, JON CARLTON
• Application Number:
  US09/386,527
• Time in Patent Office: 1,506 Days
• Field of Search: 382/118, 382/174, 382/164, 382/165, 382/206
• US Class Current: 382/118
• CPC Class Codes: G02B 30/33 involving directional light or back-light sources
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/67 by interactive indications to the user
• CPC Class Codes: H04N 13/286 having separate monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/302 for viewing without the aid of special glasses, i.e. using autostereoscopic displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof

Abstract

A method is provided for detecting a human face in an image, such as a sequence of images supplied by a video camera. The method comprises locating in each image a candidate face region and analyzing the candidate face region for a first characteristic indicative of a facial feature. The locating step may comprise detecting uniformity saturated regions of predetermined shape in a reduced resolution version of the image. The analyzing step may comprise selecting a signal color component, forming a vertical integral projection profile and detecting an omega shape in the profile characteristic of an eye region of a face.

Claims

1. A method of detecting a human face in an image, comprising locating in the image a candidate face region and analyzing the candidate face region for a first characteristic indicative of a facial feature, wherein the first characteristic comprises a substantially symmetrical horizontal brightness profile comprising a maximum disposed between first and second minima and in that the analyzing step comprises forming a vertical integral projection of a portion of the candidate face region and determining whether the vertical integral projection has first and second minima disposed substantially symmetrically about a maximum.

2. A method as claimed in claim 1, wherein the locating and analyzing steps are repeated for each image of a sequence of images.

3. A method as claimed in claim 1, wherein the image is a color image and the analyzing step is performed on a color component of the color image.

4. A method as claimed in claim 1, wherein the image is a color image and the analyzing step is performed on a contrast image derived from the color image.

5. A method as claimed in claim 1, wherein the analyzing step determines whether the vertical integral projection has first and second minima whose horizontal separation is within a predetermined range.

6. A method as claimed in claim 1, wherein the analyzing step determines whether the vertical integral projection has a maximum and first and second minima such that the ratio of the difference between the maximum and the smaller of the first and second minima to the maximum is greater than a first threshold.

7. A method as claimed in claim 6, wherein vertical integral projections are formed for a plurality of portions of the candidate face region and the portion having the highest ratio is selected as a potential target image.

8. A method as claimed in claim 1, wherein the analyzing step comprises forming a measure of the symmetry of the portion.

9. A method as claimed in claim 8, wherein the symmetry measure is formed as:

$\underset{x=0}{\overset{x_0}{\sum <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br><br><br>V<br><br>\left(x_0+x\right)-V<br><br>\left(x_0-x\right)<br><br>$

where V(x) is the value of the vertical integral projection at horizontal position x and x₀ is the horizontal position of the middle of the vertical integral projection.

10. A method as claimed in claim 8, wherein the vertical integral projection is formed for a plurality of portions of the face candidate and the portion having the highest symmetry measure is selected as a potential target image.

11. A method as claimed in claim 1, wherein the analyzing step comprises dividing a portion of the candidate face region into left and right halves, forming a horizontal integral projection of each of the halves, and comparing a measure of horizontal symmetry of the left and right horizontal integral projections with a second threshold.

12. A method as claimed in claim 1, wherein the analyzing step determines whether the candidate face region has first and second brightness minima disposed at substantially the same height with a horizontal separation within a predetermined range.

13. A method as claimed in claim 12, wherein the analyzing step determines whether the candidate face region has a vertically extending region of higher brightness than and disposed between first and second brightness minima.

14. A method as claimed in claim 13, wherein the analyzing step determines whether the candidate face region has a horizontally extending region disposed below and of lower brightness than the vertically extending region.

15. A method as claimed in claim 1, wherein the analyzing step comprises locating, in the candidate face region, candidate eye pupil regions where a green image component is greater than a red image component or where a blue image component is greater than a green image component.

16. A method as claimed in claim 15, wherein locating the candidate eye pupil regions is restricted to candidate eye regions of the candidate face region.

17. A method as claimed in claim 16, wherein the analyzing step forms a function E(x,y) for picture elements (x,y) in the candidate eye regions such that:

$E<br><br>\left(x,y\right)=\{\begin{array}{cc}0& \mathrm{for}<br><br>\text{\hspace{1em}<br><br>}<br><br>R-G><br><br>C_1<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{and}<br><br>\text{\hspace{1em}<br><br>}<br><br>G-B><br><br>C_2\\ 1& \mathrm{otherwise}\end{array}$

where R, G and B are red, green and blue image components, C₁ and C₂ are constants'"'"' E(x,y)=1 represents a picture element inside the candidate eye pupil regions and E(x,y)=0 represents a picture element outside the candidate eye pupil regions.

18. A method as claimed in claim 17, wherein the analyzing step detects the centers of the eye pupils as the centroids of the candidate eye pupil regions.

19. A method as claimed in claim 15, wherein the analyzing step comprises locating a candidate mouth region in a sub-region of the candidate face region which is horizontally between the candidate eye pupil regions and vertically below the level of the candidate eye pupil regions by between substantially half and substantially one and a half times the distance between the candidate eye pupil regions.

20. A method as claimed in claim 19, wherein the analyzing step forms a function M(x,y) for picture elements (x,y) within the sub-region such that:

$M<br><br>\left(x,y\right)=\{\begin{array}{cc}1& \mathrm{for}<br><br>\text{\hspace{1em}<br><br>}<br><br>R><br><br>G><br><br>B<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{and}<br><br>\text{\hspace{1em}<br><br>}<br><br>R<<br><br>\mathrm{\eta <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>G\\ 0& \mathrm{otherwise}\end{array}$

where R, G and B are red, green and blue image components, η

is a constant, M(x,y)=1 represents a picture element inside the candidate mouth region and M(x,y)=0 represents a picture element outside the candidate mouth region.

21. A method as claimed in claim 20, wherein vertical and horizontal projection profiles of the function M(x,y) are formed and a candidate lip region is defined in a rectangular sub-region where the vertical and horizontal projection profiles exceed first and second predetermined thresholds, respectively.

22. A method as claimed in claim 21, wherein the first and second predetermined thresholds are proportional to maxima of the vertical and horizontal projection profiles, respectively.

23. A method as claimed in claim 21, wherein the analyzing step checks whether the aspect ratio of the candidate lip region is between first and second predefined thresholds.

24. A method as claimed in claim 21, wherein the analyzing step checks whether the ratio of the vertical distance from the candidate eye pupil regions to the top of the candidate lip region to the spacing between the candidate eye pupil regions is between first and second preset thresholds.

25. A method as claimed in claim 1, wherein the analyzing step comprises dividing a portion of the candidate face region into left and right halves and comparing the angles of the brightness gradients of horizontally symmetrically disposed pairs of points for symmetry.

26. A method as claimed in claim 2, wherein the locating and analyzing steps are stopped when the first characteristic is found r times in R consecutive images of the sequence.

27. A method as claimed in claim 1, wherein the locating step comprises searching the image for a candidate face region having a second characteristic indicative of a human face.

28. A method as claimed in claim 27, wherein the second characteristic is uniform saturation.

29. A method as claimed in claim 28, wherein the searching step comprises reducing the resolution of the image by averaging the saturation to form a reduced resolution image and searching for a region of the reduced resolution image having, in a predetermined shape, a substantially uniform saturation which is substantially different from the saturation of the portion of the reduced resolution image surrounding the predetermined shape.

30. A method as claimed in claim 29, wherein the image comprises a plurality of picture elements and the resolution is reduced such that the predetermined shape is from two to three reduced resolution picture elements across.

31. A method as claimed in claim 30, wherein the image comprises a rectangular array of M by N picture elements, the reduced resolution image comprises (M/m) by (N/n) picture elements, each of which corresponds to m by n picture elements of the image, and the saturation P of each picture element of the reduced resolution image is given by:

$P=\left(1/n<br><br>\text{\hspace{1em}<br><br>}<br><br>m\right)<br><br>\underset{i=0}{\overset{m-1}{\sum <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>\underset{j=0}{\overset{n-1}{\sum <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>f<br><br>\left(i,j\right)$

where f(i,j) is the saturation of the picture element of the ith column and the jth row of the m by n picture elements.

32. A method of claimed in claim 31, further comprising storing the saturations in a store.

33. A method as claimed in claim 31, wherein a uniformity value is ascribed to each of the reduced resolution picture elements by comparing the saturation of each of the reduced resolution picture elements with the saturation of at least one adjacent reduced resolution picture element.

34. A method as claimed in claim 33, wherein each uniformity value is ascribed a first value if

35. A method as claimed in claim 34, wherein T is substantially equal to 0.15.

36. A method as claimed in claim 33, further comprising storing the saturations in a store, wherein the or each adjacent reduced resolution picture element has not been ascribed a uniformity value and each uniformity value is stored in the store in place of the corresponding saturation.

37. A method as claimed in claim 34, wherein the resolution is reduced such that the predetermined shape is two or three reduced resolution picture elements across and wherein the method further comprises indicating detection of a candidate face region when a uniformity value of the first value is ascribed to any of one reduced resolution picture element, two vertically or horizontally adjacent reduced resolution picture elements and a rectangular two-by-two array of picture elements and when a uniformity value of the second value is ascribed to each surrounding reduced resolution picture element.

38. A method as claimed in claim 37, further comprising storing the saturations in a store, wherein detection is indicated by storing a third value different from the first and second values in the store in place of the corresponding uniformity value.

39. A method as claimed in claim 30, further comprising repeating the resolution reduction and searching at least once with the reduced resolution picture elements shifted with respect to the image picture elements.

40. A method as claimed in claim 29, wherein the saturation is derived from red, green and blue components as

41. A method as claimed in claim 1, wherein a first image is captured while illuminating an expected range of positions of a face, a second image is captured using ambient light, and the second image is subtracted from the first image to form the image.