Method for face orientation determination in digital color images

• US 7,120,279 B2
• Filed: 01/30/2003
• Issued: 10/10/2006
• Est. Priority Date: 01/30/2003
• Status: Expired due to Fees

• Current Assignee: Monument Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Eastman Kodak Company
• Inventors: Chen, Shoupu, Ray, Lawrence A.
• Primary Examiner(s): Mancuso, Joseph
• Assistant Examiner(s): Kuhn, Jordan
• Application Number:
  US10/354,837
• Publication Number:
  US 20040151371A1
• Time in Patent Office: 1,349 Days
• Field of Search: 382/118, 382/164
• US Class Current: 382/118
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships

Abstract

A digital image processing method for determining an orientation of a face in a digital color image generates a mean grid pattern element image from a plurality of sample face images and an integral image from the digital color image. A face is located in the color digital image by using the integral image to perform a correlation test between the mean grid pattern element image and the digital color image at a plurality of effective resolutions by reducing the digital color image to a plurality of grid pattern element images at different effective resolutions and correlating the mean grid pattern element image with the plurality of grid pattern element images, whereby either the mean grid pattern element image or the grid pattern element images are provided at a plurality of different orientations. Accordingly, an orientation of the face in the color digital image is determined by using the images with different orientations in the correlation test.

Claims

1. A digital image processing method for locating faces in a digital image, comprising the steps of:

generating an integral image from the digital image;

generating a grid pattern image from said integral image, said grid pattern image having a grid pattern having a plurality of cells each of said cells having a plurality of pixels;

reducing said grid pattern image to a corresponding base grid pattern element image, said base grid pattern element image having a number of pixels equal to the number of cells in said grid pattern image;

providing a set of different orientation images, said set including a selected one of said base grid pattern element image and a predetermined mean grid pattern element image of sample face images and excluding the other of said base grid pattern element image and said mean grid pattern element image, said set also including a plurality of additional grid pattern element images, each said additional grid pattern element image being a different rearrangement of elements of the selected one of said base grid pattern element image and said mean grid pattern element image; and

performing a correlation test between said images of said set of different orientation images and said excluded one of said base grid pattern element image and said mean grid pattern element image.

2. The method of claim 1 wherein said digital image is a digital color image.

3. The method claimed in claim 1 wherein said generating an integral image step further comprises:

cropping a sub-image from said digital color image; and

computing said integral image from said sub-image.

4. The method claimed in claim 3 wherein said grid pattern has a first cell size and the method further comprises iterating said generating of said grid pattern image, said reducing, and said performing steps with each of a plurality of additional grid patterns, said grid patterns all having different cell sizes.

5. The method claimed in claim 4 wherein said digital image has a plurality of different areas, said sub-image is of one of said areas;

and the method further comprises repeating said generating, generating, reducing, providing, performing, and iterating steps on a plurality of different additional sub-images of said digital image.

6. The method claimed in claim 1 further comprising prior to said generating steps:

generating a mean face image from a collection of sample face images;

generating a mean integral image from the mean face image;

generating a mean grid pattern image from said mean integral image, said mean grid pattern image having a plurality of cells in said grid pattern; and

reducing said mean grid pattern image to a corresponding mean grid pattern element image, said mean grid pattern element image having a number of pixels equal to the number of cells in said mean grid pattern image.

7. The method claimed in claim 6 wherein said base grid pattern element image and said mean grid pattern element image are one-dimensional.

8. The method as claimed in claim 6, wherein said method further comprises the step of selecting an irregular grid pattern, wherein said irregular grid pattern has a plurality of different size grid cells that cover major features including at least one of eyes, nose, mouth, forehead, and cheek of the mean face image.

9. The method claimed in claim 1, wherein the grid pattern is regular.

10. The method claimed in claim 1, wherein the grid pattern is irregular.

11. A computer program product stored on a computer readable medium for performing the method of claim 1.

12. A digital image processing method for locating faces in a digital image, comprising the steps of:

generating an integral image from the digital image;

generating a grid pattern image from said integral image, said grid pattern image having a grid pattern having a plurality of cells each of said cells having a plurality of pixels;

reducing said grid pattern image to a corresponding base grid pattern element image, said base grid pattern element image having a number of pixels equal to the number of cells in said grid pattern image;

providing a set of different orientation images, said set including said base grid pattern element image and a plurality of additional grid pattern element images, each said additional grid pattern element image being a different rearrangement of elements of said base grid pattern element image; and

performing a correlation test between said images of said set of different orientation images and said mean grid pattern element image.

13. The method of claim 12 wherein said providing further comprises generating said additional grid pattern element images from said base grid pattern element image.

14. The method claimed in claim 12 wherein said generating an integral image step further comprises:

cropping a sub-image from said digital color image; and

computing said integral image from said sub-image.

15. The method claimed in claim 14 wherein said grid pattern has a first cell size and the method further comprises iterating said generating of said grid pattern image, said reducing, and said performing steps with each of a plurality of additional grid patterns, said grid patterns all having different cell sizes.

16. The method claimed in claim 15 wherein said digital color image has a plurality of different areas, said sub-image is of one of said areas;

and the method further comprises repeating said generating, generating, reducing, providing, performing, and iterating steps on a plurality of different additional sub-images of said digital color image.

17. The method claimed in claim 12 further comprising prior to said generating steps:

generating a mean face image from a collection of sample face images;

generating a mean integral image from the mean face image;

generating a mean grid pattern image from said mean integral image, said mean grid pattern image having a plurality of cells in said grid pattern; and

reducing said mean grid pattern image to a corresponding mean grid pattern element image, said mean grid pattern element image having a number of pixels equal to the number of cells in said mean grid pattern image.

18. A computer program product stored on a computer readable medium for performing the method of claim 12.

19. A digital image processing method for locating faces in a digital image, comprising the steps of:

generating an integral image from the digital image;

generating a grid pattern image from said integral image, said grid pattern image having a grid pattern having a plurality of cells each of said cells having a plurality of pixels;

reducing said grid pattern image to a corresponding base grid pattern element image, said base grid pattern element image having a number of pixels equal to the number of cells in said grid pattern image;

providing a set of different orientation images, said set including a predetermined mean grid pattern element image of sample face images and a plurality of additional grid pattern element images, each said additional grid pattern element image being a different rearrangement of elements of said mean grid pattern element image; and

performing a correlation test between said images of said set of different orientation images and said base grid pattern element image.

20. The method claimed in claim 19 wherein said generating an integral image step further comprises:

cropping a sub-image from said digital color image; and

computing said integral image from said sub-image.

21. The method claimed in claim 20 wherein said grid pattern has a first cell size and the method further comprises iterating said generating of said grid pattern image, said reducing, and said performing steps with each of a plurality of additional grid patterns, said grid patterns all having different cell sizes.

22. The method claimed in claim 21 wherein said digital image has a plurality of different areas, said sub-image is of one of said areas;

and the method further comprises repeating said generating, generating, reducing, providing, performing, and iterating steps on a plurality of different additional sub-images of said digital image.

23. The method claimed in claim 19 further comprising prior to said generating steps:

generating a mean face image from a collection of sample face images;

generating a mean integral image from the mean face image;

generating a mean grid pattern image from said mean integral image, said mean grid pattern image having a plurality of cells in said grid pattern; and

reducing said mean grid pattern image to a corresponding mean grid pattern element image, said mean grid pattern element image having a number of pixels equal to the number of cells in said mean grid pattern image.

24. A computer program product for performing the method of claim 19.

25. A digital image processing method for determining orientation of faces located in a digital color image, comprising the steps of:

a) generating a mean grid pattern element image from a plurality of sample face images;

b) generating an integral image from the digital color image;

c) locating a face in the color digital image by using the integral image to perform a correlation test between the mean grid pattern element image and the digital color image at a plurality of effective resolutions by reducing the digital color image to a plurality of grid pattern element images at different effective resolutions and correlating the mean grid pattern element image with the plurality of grid pattern element images, whereby either the mean grid pattern element image or the grid pattern element images are provided at a plurality of different orientations; and

d) determining an orientation of the face in the color digital image by using the images with different orientations in the correlation test of step c);

wherein the step a) of generating a mean grid pattern element image comprises the steps of;

a1) collecting sample face images;

a2) generating a mean face image from the sample face images;

a3) selecting a regular grid pattern;

a4) reducing the resolution of the mean face image to the resolution of the selected grid pattern by averaging; and

a5) generating mean grid pattern element images with different orientations by rearranging the order of the elements; and

wherein the step of selecting a regular grid pattern comprises computing a distance between two eye centers of the mean face image;

computing a center position between the two eye centers; and

using the distance and position to determine M and N dimensions and a position of a region wherein said region contains M by N grid cells with each cell having m by n pixels.

26. A digital image processing method for determining orientation of faces located in a digital color image, comprising the steps of:

a) generating a mean grid pattern element image from a plurality of sample face images;

b) generating an integral image from the digital color image;

c) locating a face in the color digital image by using the integral image to perform a correlation test between the mean grid pattern element image and the digital color image at a plurality of effective resolutions by reducing the digital color image to a plurality of grid pattern element images at different effective resolutions and correlating the mean grid pattern element image with the plurality of grid pattern element images, whereby either the mean grid pattern element image or the grid pattern element images are provided at a plurality of different orientations; and

d) determining an orientation of the face in the color digital image by using the images with different orientations in the correlation test of step c);

wherein the step b) of generating an integral image further comprises the steps of;

b1) replacing non-skin color pixels in the digital color image with black to produce an image having skin color pixels;

b2) replacing non-face shaped clusters of pixels with black to produce an image having skin colored and face shaped clusters;

b3) labeling skin colored and face shaped clusters as face clusters; and

b4) generating the integral image from each labeled face cluster of the image.

27. The method claimed in claim 26, further comprising the steps of:

eliminating face clusters that contain more than a predetermined percentage of black pixels; and

merging face clusters that substantially overlap.

28. The method claimed in claim 26, wherein the step of replacing non face shaped clusters comprises the steps of:

clustering skin-color pixels in the image into clusters;

applying morphological opening and closing processes to the skin-colored pixel clusters; and

replacing the pixels of a cluster with black if they do not meet a geometrical criterion for a face, thereby resulting in a processed image; and

wherein the step of labeling skin colored clusters comprises the step of generating a linked list of sets of parameters including a starting position, width, and height, that defines regions containing a cluster of skin-colored pixels.

29. A digital image processing method for determining orientation of faces located in a digital color image, comprising the steps of:

generating a mean grid pattern element image from a plurality of sample face images;

generating an integral image from the digital color image;

locating a face in the color digital image by using the integral image to perform a correlation test between the mean grid pattern element image and the digital color image at a plurality of effective resolutions by reducing the digital color image to a plurality of grid pattern element images at different effective resolutions and correlating the mean grid pattern element image with the plurality of grid pattern element images, whereby either the mean grid pattern element image or the grid pattern element images are provided at a plurality of different orientations; and

determining an orientation index of the face in the color digital image by using the images with different orientations in the correlation test of step c);

repeating said generating, generating, locating, and determining steps to provide a plurality of faces;

wherein said determining step further comprises the steps of;

rank ordering the orientation indicies associated with the faces;

determining the image orientation by applying an order statistics analysis to the rank ordered orientation indices associated with the faces; and

determining the image orientation by voting using a top few of the orientation indices.