Detection correction of red-eye features in digital images

• US 20040184670A1
• Filed: 05/04/2004
• Published: 09/23/2004
• Est. Priority Date: 02/22/2002
• Status: Abandoned Application

• Current Assignee: Pixology Software Limited
• Original Assignee: Pixology Ltd. (Oracle Corporation)
• Inventors: Normington, Daniel, Archibald, Marion, Lafferty, Richard, Stroud, Mike, Jarman, Nick, Biggs, Nigel
• Application Number:
  US10/475,536
• Publication Number:
  US 20040184670A1
• Time in Patent Office: Days
• US Class Current: 382/274
• CPC Class Codes: H04N 1/624 Red-eye correction control of camera based on recognised facial parts H04N23/611

Abstract

A method of correcting red-eye features in a digital image includes generating a list of possible features by scanning through each pixel in the image searching for saturation and/or lightness profiles characteristic of red-eye features. For each feature in the list, an attempt is made to find an isolated area of correctable pixels which could correspond to a red-eye feature. Each successful attempt is recorded in a list of areas. Each area is then analysed to calculate statistics and record properties of that area, and validated using the calculated statistics and properties to determine whether or not that area is caused by red-eye. Areas not caused by red-eye and overlapping areas are removed from the list. Each area remaining is corrected to reduce the effect of red-eye. More than one type of feature may be identified in the initial search for features.

Claims

1. A method of detecting red-eye features in a digital image, comprising:

identifying pupil regions in the image by searching for a row of pixels having a predetermined saturation and/or lightness profile;

identifying further pupil regions in the image by searching for a row of pixels having a different predetermined saturation and/or lightness profile; and

determining whether each pupil region corresponds to part of a red-eye feature on the basis of further selection criteria.

2. A method as claimed in claim 1, comprising identifying two or more types of pupil regions, a pupil region in each type being identified by a row of pixels having a saturation and/or lightness profile characteristic of that type.

3. A method as claimed in claim 2, wherein a first type of pupil region has a saturation profile including a region of pixels having higher saturation than the pixels therearound.

4. A method as claimed in claim 2 or 3, wherein a second type of pupil region has a saturation profile including a saturation trough bounded by two saturation peaks, the pixels in the saturation peaks having higher saturation than the pixels in the area outside the saturation peaks.

5. A method as claimed in claim 2, 3 or 4, wherein a third type of pupil region has a lightness profile including a region of pixels whose lightness values form a “

W”

shape.

6. A method as claimed in any of claims 2 to 5, wherein a fourth type of pupil region has a saturation and lightness profile including a region of pixels bounded by two local saturation minima, wherein:

at least one pixel in the pupil region has a saturation higher than a predetermined saturation threshold;

the saturation and lightness curves of pixels in the pupil region cross twice; and

two local lightness minima are located in the pupil region.

7. A method as claimed in claim 6, wherein the predetermined saturation threshold is about 100.

8. A method as claimed in claim 7, wherein:

the saturation of at least one pixel in the pupil region is at least 50 greater than the lightness of that pixel;

the saturation of the pixel at each local lightness minimum is greater than the lightness of that pixel;

one of the local lightness minima includes the pixel having the lowest lightness in the region between the two lightness minima; and

the lightness of at least one pixel in the pupil region is greater than a predetermined lightness threshold.

9. A method as claimed in claim 6, 7 or 8, wherein the hue of the at least one pixel having a saturation higher than a predetermined threshold is greater than about 210 or less than about 20.

10. A method as claimed in any preceding claim, wherein a fifth type of pupil region has a saturation and lightness profile including a high saturation region of pixels having a saturation above a predetermined threshold and bounded by two local saturation minima, wherein:

the saturation and lightness curves of pixels in the pupil region cross twice at crossing pixels;

the saturation is greater than the lightness for all pixels between the crossing pixels; and

two local lightness minima are located in the pupil region.

11. A method as claimed in claim 10, wherein:

the saturation of pixels in the high saturation region is above about 100;

the hue of pixels at the edge of the high saturation region is greater than about 210 or less than about 20; and

no pixel up to four outside each local lightness minimum has a lightness lower than the pixel at the corresponding local lightness minimum.

13. A method as claimed in claim 12, wherein the step of generating a list of possible features is performed using a method as claimed in any of claims 1 to 11.

49. Apparatus arranged to carry out the method of any preceding claim.

50. Apparatus as claimed in claim 49, which apparatus is a personal computer, printer, digital printing mini-lab, camera, portable viewing device, PDA, scanner, mobile phone, electronic book, public display system, video camera, television, digital film editing equipment, digital projector, head-up-display system, or photo booth.

51. A computer storage medium having stored thereon a program arranged when executed to carry out the method of any of claims 1 to 48.

52. A digital image to which has been applied the method of any of claims 1 to 48.

12. A method of correcting red-eye features in a digital image, comprising:

generating a list of possible features by scanning through each pixel in the image searching for saturation and/or lightness profiles characteristic of red-eye features;

for each feature in the list of possible features, attempting to find an isolated area of correctable pixels which could correspond to a red-eye feature;

recording each successful attempt to find an isolated area in a list of areas;

analysing each area in the list of areas to calculate statistics and record properties of that area;

validating each area using the calculated statistics and properties to determine whether or not that area is caused by red-eye;

removing from the list of areas those which are not caused by red-eye;

removing some or all overlapping areas from the list of areas; and

correcting some or all pixels in each area remaining in the list of areas to reduce the effect of red-eye.

31. A method as claimed in claim 12 or 13, wherein the step of correcting some or all pixels in each area remaining in the list of areas to reduce the effect of red-eye is performed using a method as claimed in any of claims 14 to 30.

41. A method as claimed in any of claims 12 to 31, wherein the step of attempting to find an isolated area which could correspond to a red-eye feature is performed using a method as claimed in any of claims 37 to 40.

42. A method as claimed in any of claims 12 to 41, wherein the step of analysing each area in the list of areas includes determining some or all of:

the mean of the hue, luminance and/or saturation of the pixels in the area;

the standard deviation of the hue, luminance and/or saturation of the pixels in the area;

the mean and standard deviation of the value of hue×

saturation, hue×

lightness and/or lightness×

saturation of the pixels in the area;

the sum of the squares of differences in hue, luminance and/or saturation between adjacent pixels for all of the pixels in the area;

the sum of the absolute values of differences in hue, luminance and/or saturation between adjacent pixels for all of the pixels in the area;

a measure of the number of differences in lightness and/or saturation above a predetermined threshold between adjacent pixels;

a histogram of the number of correctable pixels having from 0 to 8 immediately adjacent correctable pixels;

a histogram of the number of uncorrectable pixels having from 0 to 8 immediately adjacent correctable pixels;

a measure of the probability of the area being caused by red-eye based on the probability of the hue, saturation and lightness of individual pixels being found in a red-eye feature; and

a measure of the probability of the area being a false detection of a red-eye feature based on the probability of the hue, saturation and lightness of individual pixels being found in a detected feature not caused by red-eye.

43. A method as claimed in claim 42, wherein the measure of the probability of the area being caused by red-eye is determined by evaluating the arithmetic mean, over all pixels in the area, of the product of the independent probabilities of the hue, lightness and saturation values of each pixel being found in a red-eye feature.

44. A method as claimed in claim 42 or 43, wherein the measure of the probability of the area being a false detection is determined by evaluating the arithmetic mean, over all pixels in the area, of the product of the independent probabilities of the hue, lightness and saturation values of each pixel being found in detected feature not caused by red-eye.

45. A method as claimed in any of claims 12 to 44, wherein the step of analysing each area in the list of areas includes analysing an annulus outside the area, and categorising the area according to the hue, luminance and saturation of pixels in said annulus.

46. A method as claimed in any of claims 42 to 45, wherein the step of validating the area includes comparing the statistics and properties of the area with predetermined thresholds and tests.

47. A method as claimed in claim 46, wherein the thresholds and tests used to validate the area depend on the type of feature and area detected.

48. A method as claimed in any of claims 12 to 47, wherein the step of removing some or all overlapping areas from the list of areas includes:

comparing all areas in the list of areas with all other areas in the list;

if two areas overlap because they are duplicate detections, determining which area is the best to keep, and removing the other area from the list of areas;

if two areas overlap or nearly overlap because they are not caused by red-eye, removing both areas from the list of areas.

14. A method of correcting an area of correctable pixels corresponding to a red-eye feature in a digital image, comprising:

constructing a rectangle enclosing the area of correctable pixels;

determining a saturation multiplier for each pixel in the rectangle, the saturation multiplier calculated on the basis of the hue, lightness and saturation of that pixel;

determining a lightness multiplier for each pixel in the rectangle by averaging the saturation multipliers in a grid of pixels surrounding that pixel;

modifying the saturation of each pixel in the rectangle by an amount determined by the saturation multiplier of that pixel; and

modifying the lightness of each pixel in the rectangle by an amount determined by the lightness multiplier of that pixel.

15. A method as claimed in claim 14, wherein the step of determining the saturation multiplier for each pixel includes:

on a 2D grid of saturation against lightness, calculating the distance of the pixel from a calibration point having predetermined lightness and saturation values;

if the distance is greater than a predetermined threshold, setting the saturation multiplier to be 0 so that the saturation of that pixel will not be modified; and

if the distance is less than or equal to the predetermined threshold, calculating the saturation multiplier based on the distance from the calibration point so that it approaches 1 when the distance is small, and 0 when the distance approaches the threshold, so that the multiplier is 0 at the threshold and 1 at the calibration point.

16. A method as claimed in claim 15, wherein the calibration point has lightness 128 and saturation 255.

17. A method as claimed in claim 15 or 16, wherein the predetermined threshold is about 180.

18. A method as claimed in claim 15, 16 or 17, wherein the saturation multiplier for a pixel is set to 0 if the hue of that pixel is between about 20 and about 220.

19. A method as claimed in any of claims 14 to 18, further comprising applying a radial adjustment to the saturation multipliers of pixels in the rectangle, the radial adjustment comprising:

leaving the saturation multipliers of pixels inside a predetermined circle within the rectangle unchanged; and

smoothly graduating the saturation multipliers of pixels outside the predetermined circle from their previous values, for pixels at the predetermined circle, to 0 for pixels at the corners of the rectangle.

20. A method as claimed in any of claims 14 to 19, further comprising:

for each pixel immediately outside the area of correctable pixels, calculating a new saturation multiplier by averaging the value of the saturation multipliers of pixels in a 3×

3 grid around that pixel.

21. A method as claimed in any of claims 14 to 20, further comprising:

scaling the lightness multiplier of each pixel according to the mean of the saturation multipliers for all of the pixels in the rectangle.

22. A method as claimed in any of claims 14 to 21, further comprising:

for each pixel immediately outside the area of correctable pixels, calculating a new lightness multiplier by averaging the value of the lightness multipliers of pixels in a 3×

3 grid around that pixel.

23. A method as claimed in any of claims 14 to 22, further comprising:

for each pixel in the rectangle, calculating a new lightness multiplier by averaging the value of the lightness multipliers of pixels in a 3×

3 grid around that pixel.

24. A method as claimed in any of claims 14 to 23, further comprising applying a radial adjustment to the lightness multipliers of pixels in the rectangle, the radial adjustment comprising:

leaving the lightness multipliers of pixels inside an inner predetermined circle within the rectangle unchanged; and

smoothly graduating the lightness multipliers of pixels outside the inner predetermined circle from their previous values, for pixels at the inner predetermined circle, to 0 for pixels at or outside an outer predetermined circle having a diameter greater than the dimensions of the rectangle.

25. A method as claimed in any of claims 14 to 24, wherein the step of modifying the saturation of each pixel includes:

if the saturation of the pixel is greater than or equal to 200, setting the saturation of the pixel to 0; and

if the saturation of the pixel is less than 200, modifying the saturation of the pixel such that the modified saturation=(saturation×

(1−

saturation multiplier))+(saturation multiplier×

64).

26. A method as claimed in any of claims 14 to 25, wherein the step of modifying the lightness of each pixel includes:

if the saturation of the pixel is not zero and the lightness of the pixel is less than 220, modifying the lightness such that the modified lightness=lightness×

(1−

lightness multiplier).

27. A method as claimed in any of claims 14 to 26, comprising applying a further reduction to the saturation of each pixel if, after modification of the saturation and lightness of the pixel, the red value of the pixel is higher than both the green and blue values.

28. A method as claimed in any of claims 14 to 27, further comprising:

if the area, after correction, does not include a bright highlight region and dark pupil region therearound, modifying the saturation and lightness of the pixels in the area to give the effect of a bright highlight region and dark pupil region therearound.

29. A method as claimed in claim 28, further comprising:

determining if the area, after correction, substantially comprises pixels having high lightness and low saturation;

simulating a highlight region comprising a small number of pixels within the area;

modifying the lightness values of the pixels in the simulated highlight region so that the simulated highlight region comprises pixels with high lightness; and

reducing the lightness values of the pixels in the area outside the simulated highlight region so as to give the effect of a dark pupil.

30. A method as claimed in claim 29, further comprising increasing the saturation of the pixels in the simulated highlight region.

32. A method of correcting a red-eye feature in a digital image, comprising adding a simulated highlight region of pixels having a high lightness to the red-eye feature.

33. A method as claimed in claim 32, further comprising increasing the saturation of pixels in the simulated highlight region.

34. A method as claimed in claim 32 or 33, further comprising darkening the pixels in a pupil region around the simulated highlight region.

35. A method as claimed in claim 32, 33 or 34, further comprising:

identifying a flare region of pixels having high lightness and low saturation;

eroding the edges of the flare region to determine the simulated highlight region;

decreasing the lightness of the pixels in the flare region; and

increasing the lightness of the pixels in the simulated highlight region.

36. A method as claimed in any of claims 32 to 35, wherein the correction is not performed if a highlight region of light pixels is already present in the red-eye feature.

37. A method of detecting red-eye features in a digital image, comprising:

determining whether a red-eye feature could be present around a reference pixel in the image by attempting to identify an isolated, substantially circular area of correctable pixels around the reference pixel, a pixel being classed as correctable if it satisfies at least one set of predetermined conditions from a plurality of such sets.

38. A method as claimed in claim 37, wherein one set of predetermined conditions includes the requirements that:

the hue of the pixel is greater than or equal to about 220 or less than or equal to about 10;

the saturation of the pixel is greater than or equal to about 80; and

the lightness of the pixel is less than about 200.

39. A method as claimed in claim 37 or 38, wherein one set of predetermined conditions includes the requirements either that:

the saturation of the pixel is equal to 255; and

the lightness of the pixel is greater than about 150;

or that;

the hue of the pixel is greater than or equal to about 245 or less than or equal to about 20;

the saturation of the pixel is greater than about 50;

the saturation of the pixel is less than (1.8×

lightness−

92);

the saturation of the pixel is greater than (1.1×

lightness−

90); and

the lightness of the pixel is greater than about 100.

40. A method as claimed in claim 37, 38 or 39, wherein one set of predetermined conditions includes the requirements that:

the hue of the pixel is greater than or equal to about 220 or less than or equal to about 10; and

the saturation of the pixel is greater than or equal to about 128.