System and method for automatically detecting and correcting red eye

• US 7,155,058 B2
• Filed: 04/24/2002
• Issued: 12/26/2006
• Est. Priority Date: 04/24/2002
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Gaubatz, Matthew D., Ulichney, Robert A.
• Primary Examiner(s): Couso, Jose L.
• Assistant Examiner(s): SHERALI, ISHRAT I
• Application Number:
  US10/131,173
• Publication Number:
  US 20030202105A1
• Time in Patent Office: 1,707 Days
• Field of Search: 382162-165, 382/167, 382/115, 382117-118, 382274-275, 382282-283, 382224-225, 358517-520, 358/527, 358/1.9, 348 77- 78, 348576-577, 348370-371, 348/560, 359/487, 359483-485
• US Class Current: 382/167
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: H04N 1/62 Retouching, i.e. modification of isolated colours only or in isolated picture areas only
• CPC Class Codes: H04N 1/624 Red-eye correction control of camera based on recognised facial parts H04N23/611

Abstract

A system and method automatically detects and corrects the occurrence of red-eye in digital photographs and images without user intervention. The system includes a face recognition and locator engine for locating human faces within the image and for generating a bounding box around each face. A red-eye locator engine analyzes the pixel data for each bounding box and computes one or more predefined metrics. The preferred metrics include color variation, redness, redness variation and glint. The red-eye locator engine also generates one or more detection masks based upon the computed metrics, and searches the detection mask for an occurrence of red-eye. A red-eye correction engine receives the detection mask including the detected occurrences of red-eye, and generates a correction mask. Pixels identified as being occurrences of red-eye are then de-saturated in accordance with the correction mask.

Claims

1. A system for detecting and correcting red-eye in an image without user intervention, the system comprising:

a face recognition and location engine configured to identify one or more faces within the image, wherein at least a portion of the image is in the chrominance-luminance color space;

a red-eye locator engine configured to detect an occurrence of red-eye within the one or more identified faces, wherein the red-eye locator engine computes a color variation metric value for a given pixel of an identified face by computing a mean luminance-chrominance value for a patch of pixels centered on the given pixel, a squared distance between the luminance-chrominance value of the given pixel and the computed mean luminance-chrominance value for the patch, and an average of the squared distances within a region centered on the given pixel; and

a red-eye correction engine configured to correct the detected occurrences of red-eye within the image, wherein the red-eye locator engine detects an occurrence of red-eye based upon one or more metrics computed for one or more identified faces, the one or more metrics including a color variation metric tat corresponds to a change in color within a given area, the color variation metric corresponds to a change in both luminance and chrominance within the given area.

2. The system of claim 1 wherein the size of the patch corresponds to an expected size of a pupil of the respective face, and the size of the region corresponds to an expected size of an eye of the respective face.

3. The system of claim 2 wherein the face recognition and location engine is further configured to place a bounding box around each identified face, and the expected sizes of the pupil and the eye are a function of the size of the bounding box.

4. The system of claim 3 wherein the red-eye locator engine is further configured to generate a mask for an identified face, the mask having 1-bit mask values that are set by comparing the computed color variation metric values to a predetermined threshold, and the red eye-locator engine searches the mask to locate one or more occurrences of red-eye.

5. The system of claim 4 wherein each mask value corresponds to a pixel of the identified face, a mask value is asserted if the color variation metric for the corresponding pixel exceeds the threshold, otherwise the mask value is de-asserted, and the predetermined threshold is in the range of 0.5 to 1.0 for color variation metric values normalized between 0.0 and 1.0

6. The system of claim 5 wherein the red-eye locator engine searches the mask for occurrences of red-eye by determining whether the mask has two generally eye-sized areas of assated mask values, and the eye-sized areas of asserted mask values are at positions expected for eyes of the respective identified face.

7. The system of claim 3 wherein the red-eye correction engine places a candidate red-eye box around each detected occurrence of red-eye, computes a correction value for each pixel within the candidate red-eye box, and de-saturates the chrominance values of at least some of the pixels within the candidate red-eye box based on the computed correction values.

8. The system of claim 7 wherein de-saturation includes attenuating the respective chrominance value while preserving the respective luminance value.

9. The system of claim 7 wherein the correction values are computed as a function of the luminance and redness metric values of the pixels within the candidate red-eye boxes.

10. The system of claim 1 wherein the one or metrics computed by the red-eye locator engine farther include a redness metric, a redness variation metric and a glint metric.

11. The system of claim 10 wherein the red-eye locator engine initially relies solely on the mask generated from the computed color variation metric values to detect an occurrence of red-eye.

12. The system of claim 11 wherein the red-eye locator engine computes a redness metric value for at least some of the in a bounding box, provided that the mask generated from the computed color variation metric values failed to reveal any occurrences of red-eye, and generates a redness mask based on a product of the color variation mask and the computed redness metric values.

13. The system of claim 12 wherein each redness mask value corresponds to a pixel of the identified face, a redness mask value is asserted if the product of the color variation mask value and the computed redness metric value for the corresponding pixel exceeds a redness threshold, otherwise the redness mask value is de-asserted, and the redness threshold is in the rage of 0.5 to 1.0 for redness metric values normalized between 0.0 and 1.0.

14. The system of claim 10 wherein the image is in Red (R), Green (G) and Blue (B) color space and the redness metric for the given pixel is given by the following equation:

15. The system of claim 14 wherein K is in the range of approximately 13–

15.

16. The system of claim 1 wherein the system is disposed within a printer and is configured to detect and correct occurrences of red-eye within images transmitted to the printer.

17. A method for detecting and correcting red-eye in an image without user intervention, the method comprising the steps of:

identifying one or more faces within the image, wherein at least a portion of the one or more faces are represented by pixels in luminance-chrominance color space;

detecting occurrences of red-eye for an identified face based on one or more metrics computed for at least some of the pixels of the identified face, wherein the detecting comprises the steps of;

computing a mean luminance-chrominance value for a patch pixels centered on a given pixel, computing a squared distance between the luminance-chrominance value of the given pixel and the computed mean luminance-chrominance value for the patch, computing an average of the squared distance within a region centered on the given pixel, and assigning the computed average as the color variation metric value for the given pixel; and

correcting the detected occurrences of red-eye, wherein the one or more metrics includes a color variation metric that corresponds to a change in both luminance and chrominance within a given area of the identified face.

18. The method of claim 17 wherein the step of identifying further includes the step of placing a bounding box around each identified face, and the size of the patch corresponds to an expected size of a pupil of the respective face, the size of the region corresponds to an expected size of an eye of the respective face, and the expected sizes of the pupil and the eye are a function of the size of the bounding box.

19. The method of claim 18 wherein the one or more metrics further include a redness metric that is given by the following equation

20. The method of claim 17 further comprising the step of calculating a plurality of summed area table values for use in optimizing the computation of one or more of the metrics.