Detecting and correcting red-eye in a digital image

• US 20040213476A1
• Filed: 04/28/2003
• Published: 10/28/2004
• Est. Priority Date: 04/28/2003
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Yen, Jonathan, Luo, Huitao, Tretter, Daniel R.
• Granted Patent: US 7,116,820 B2
• Time in Patent Office: Days
• US Class Current: 382/254
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: H04N 1/62 Retouching, i.e. modification of isolated colours only or in isolated picture areas only
• CPC Class Codes: H04N 1/624 Red-eye correction control of camera based on recognised facial parts H04N23/611

Abstract

Systems and methods of detecting and correcting red-eye in a digital image are described. In one aspect, measures of pixel redness in the digital image are computed. A preliminary set of candidate red-eye pixel areas is identified based on the computed pixel redness measures. Each candidate red-eye pixel area having a computed redness contrast relative to at least one respective neighboring pixel area less than a prescribed redness contrast threshold is filtered from the preliminary set. In addition, each candidate red-eye pixel area located in an area of the digital image having a computed grayscale contrast relative to at least one respective neighboring pixel area less than a prescribed grayscale contrast threshold is filtered from the preliminary set. In another aspect, at least one candidate red-eye pixel area in the digital image is detected. Pixels in each candidate red-eye pixel area are classified based on redness and skin tone coloration. For each candidate red-eye pixel area, an initial pixel mask identifying all pixels in the candidate red-eye pixel area classified as red and non-skin tone is generated. A final pixel mask is generated based on each initial pixel mask. Red-eye in the digital image is corrected by darkening and desaturating original color values identified as red in the final pixel mask, wherein original color values of pixels identified as red in the final pixel mask are desaturated by respective amounts that vary with pixel location in the final pixel mask.

Claims

1. A method of processing a digital image containing pixels with original color values, comprising:

computing measures of pixel redness in the digital image;

identifying a preliminary set of candidate red-eye pixel areas based on the computed pixel redness measures;

filtering from the preliminary set each candidate red-eye pixel area having a computed redness contrast relative to at least one respective neighboring pixel area less than a prescribed redness contrast threshold; and

filtering from the preliminary set each candidate red-eye pixel area located in an area of the digital image having a computed grayscale contrast relative to at least one respective neighboring pixel area less than a prescribed grayscale contrast threshold.

2. The method of claim 1, wherein pixel redness measures are computed based on a ratio of a measure of a red component of pixel energy to a measure of total pixel energy.

3. The method of claim 2, wherein pixel redness measures (R) are computed based on the following ratio:

4. The method of claim 1, wherein identifying the preliminary set of candidate red-eye pixel areas comprises applying a two-dimensional redness filter to the computed pixel redness measures, wherein the redness filter is operable to compute a redness score based on a central kernel pixel area and a pixel area surrounding the kernel pixel area.

5. The method of claim 4, wherein the kernel pixel area and the surrounding pixel area are concentric.

6. The method of claim 4, wherein the kernel pixel area corresponds to a square pixel area of side length dl and the surrounding pixel area corresponds to a rectangular path defined between square pixel area of side length d2 and the central kernel pixel area, wherein d1<

d2.

7. The method of claim 4, wherein the computed redness score includes a component measuring average redness of the kernel pixel area and a component measuring average contrast of the kernel pixel area relative to the surrounding pixel area.

8. The method of claim 4, wherein redness scores are computed for multiple different kernel pixel area sizes.

9. The method of claim 4, further comprising applying a prescribed threshold to the computed redness scores to identify candidate red-eye pixels.

10. The method of claim 9, wherein computing the preliminary set of candidate red-eye pixel areas comprises segmenting candidate red-eye pixels into groups.

11. The method of claim 10., further comprising computing a measure of average pixel redness for each segmented pixel group, and maximizing each segmented pixel group to a candidate red-eye pixel area having an average pixel redness measure at least as large as the average pixel redness measure computed for the corresponding segmented pixel group.

12. The method of claim 1, further comprising filtering candidate red-eye pixel areas from the preliminary set based on candidate red-eye pixel area shape.

13. The method of claim 1, wherein each candidate red-eye pixel area having a computed redness contrast relative to each of a set of corresponding surrounding pixel areas less than the prescribed redness contrast threshold is filtered from the preliminary set.

14. The method of claim 13, wherein each surrounding pixel area is equal in size to a corresponding candidate red-eye pixel area.

15. The method of claim 1, further comprising computing measures of pixel grayscale in the digital image and computing for a given candidate red-eye pixel area a candidate eyeball area centered at the given candidate red-eye pixel area and having a size maximizing grayscale contrast between the candidate eyeball area and areas surrounding the candidate eyeball area.

16. The method of claim 15, further comprising computing a measure of grayscale contrast between the candidate eyeball area and at least a portion of the areas surrounding the candidate eyeball area and applying a threshold to the computed grayscale contrast measure to filter candidate red-eye pixel areas from the preliminary set.

17. The method of claim 1, further comprising pairing candidate red-eye pixel areas, and filtering unpaired candidate red-eye pixel areas from the preliminary set.

18. The method of claim 1, further comprising filtering from the preliminary set candidate red-eye pixel areas merged with adjacent skin tone regions in the digital image.

19. The method of claim 1, further comprising filtering candidate red-eye pixel areas from the preliminary set based on proportions of non-skin tone pixels in the candidate red-eye pixel areas and proportions of skin tone pixels in areas surrounding the candidate red-eye pixel areas.

20. The method of claim 1, further comprising mapping digital image pixels in candidate red-eye pixel areas into YCbCr color space, and applying thresholds to mapped YCbCr color space pixel values to filter candidate red-eye pixel areas.

21. The method of claim 1, further comprising computing a ratio of YCbCr color space chrominance components for each pixel in image areas respectively encompassing candidate red-eye pixels areas and applying a threshold to the computed ratios to identify candidate red-eye regions.

22. The method of claim 21, further comprising identifying in the candidate red-eye regions final red-eye pixel regions corresponding to isolated pixel islands.

23. The method of claim 1, further comprising correcting red-eye in the digital image.

24. The method of claim 23, wherein correcting red-eye comprises classifying pixels in each candidate red-eye pixel area based on redness and skin tone coloration and, for each candidate red-eye pixel area, generating an initial pixel mask identifying all pixels in the candidate red-eye pixel area classified as red and non-skin tone.

25. The method of claim 24, further comprising modifying an initial pixel mask to achieve a mask size and location matching a corresponding red-eye pixel area.

26. The method of claim 25, further comprising generating a final pixel mask by increasing the modified pixel mask in size to accommodate pink areas surrounding the corresponding red-eye pixel area.

27. The method of claim 26, further comprising correcting original color values of pixels corresponding to the final pixel mask by desaturating original color values.

28. The method of claim 27, wherein original color values identified as red in the final pixel mask are desaturated by respective amounts that vary with pixel location in the final pixel mask.

29. The method of claim 27, wherein original color values of a given pixel are desaturated based on a luminance level of the given pixel.

30. The method of claim 27, wherein original color values of a given pixel are desaturated based on a number of pixels in the final pixel mask neighboring the given pixel.

31. The method of claim 27, further comprising correcting original color values of pixels corresponding to the final pixel mask by darkening the original color values.

32. The method of claim 31, wherein original color values of a given pixel are darkened by an amount decreasing with a luminance level of the given pixel.

33. A method of processing a digital image containing pixels with original color values, comprising:

detecting at least one candidate red-eye pixel area in the digital image;

classifying pixels in each candidate red-eye pixel area based on redness and skin tone coloration;

for each candidate red-eye pixel area, generating an initial pixel mask identifying all pixels in the candidate red-eye pixel area classified as red and non-skin tone;

generating a final pixel mask based on each initial pixel mask; and

correcting red-eye in the digital image by darkening and desaturating original color values identified as red in the final pixel mask, wherein original color values of pixels identified as red in the final pixel mask are desaturated by respective amounts that vary with pixel location in the final pixel mask.

34. The method of claim 33, further comprising modifying an initial pixel mask to achieve a mask size and location matching a corresponding red-eye pixel area.

35. The method of claim 34, further comprising generating the final pixel mask by increasing the modified pixel mask in size to accommodate pink areas surrounding the corresponding red-eye pixel area.

36. The method of claim 33, wherein original color values of a given pixel are desaturated based on a luminance level of the given pixel.

37. The method of claim 36, wherein original color values of a given pixel are desaturated based on a number of pixels in the final pixel mask neighboring the given pixel.

38. The method of claim 33, wherein original color values of a given pixel are darkened by an amount increasing with a darkness level of the given pixel.

39. A system for processing a digital image containing pixels with original color values, comprising a global screening module operable to:

compute measures of pixel redness in the digital image;

identify a preliminary set of candidate red-eye pixel areas based on the computed pixel redness measures;

filter from the preliminary set each candidate red-eye pixel area having a computed redness contrast relative to at least one respective neighboring pixel area less than a prescribed redness contrast threshold; and

filter from the preliminary set each candidate red-eye pixel area located in an area of the digital image having a computed grayscale contrast relative to at least one respective neighboring pixel area less than a prescribed grayscale contrast threshold.

40. The system of claim 39, wherein pixel redness measures are computed based on a ratio of a measure of a red component of pixel energy to a measure of total pixel energy.

41. The system of claim 39, wherein identifying the preliminary set of candidate red-eye pixel areas comprises applying a two-dimensional redness filter to the computed pixel redness measures, wherein the redness filter is operable to compute a redness score based on a central kernel pixel area and a pixel area surrounding the kernel pixel area.

42. The system of claim 39, wherein the global screening module is operable to filter candidate red-eye pixel areas from the preliminary set based on candidate red-eye pixel area shape.

43. The system of claim 39, wherein each candidate red-eye pixel area having a computed redness contrast relative to each of a set of corresponding surrounding pixel areas less than the prescribed redness contrast threshold is filtered from the preliminary set.

44. The system of claim 39, wherein the global screening module is operable to compute measures of pixel grayscale in the digital image and computing for a given candidate red-eye pixel area a candidate eyeball area centered at the given candidate red-eye pixel area and having a size maximizing grayscale contrast between the candidate eyeball area and areas surrounding the candidate eyeball area.

45. The system of claim 39, further comprising a local verification module operable to pair candidate red-eye pixel areas and filter unpaired candidate red-eye pixel areas from the preliminary set.

46. The system of claim 39, further comprising a local verification module operable to filter from the preliminary set candidate red-eye pixel areas merged with adjacent skin tone regions in the digital image.

47. The system of claim 39, further comprising a local verification module operable to filter candidate red-eye pixel areas from the preliminary set based on proportions of non-skin tone pixels in the candidate red-eye pixel areas and proportions of skin tone pixels in areas surrounding the candidate red-eye pixel areas.

48. The system of claim 39, further comprising a local verification module operable to map digital image pixels in candidate red-eye pixel areas into YCbCr color space, and applying thresholds to mapped YCbCr color space pixel values to filter candidate red-eye pixel areas.

49. The system of claim 39, further comprising a local verification module operable to compute a ratio of YCbCr color space chrominance components for each pixel in image areas respectively encompassing candidate red-eye pixels areas and applying a threshold to the computed ratios to identify candidate red-eye regions.

50. The system of claim 49, wherein the local verification module is operable to identify in the candidate red-eye regions final red-eye pixel regions corresponding to isolated pixel islands.

51. The system of claim 39, further comprising a red-eye segmentation module operable to classify pixels in each candidate red-eye pixel area based on redness and skin tone coloration and, for each candidate red-eye pixel area, generate an initial pixel mask identifying all pixels in the candidate red-eye pixel area classified as red and non-skin tone.

52. The method of claim 51, wherein the red-eye segmentation module operable to modify an initial pixel mask to achieve a mask size and location matching a corresponding red-eye pixel area.

53. The method of claim 52, wherein the red-eye segmentation module operable to generate a final pixel mask by increasing the modified pixel mask in size to accommodate pink areas surrounding the corresponding red-eye pixel area.

54. The system of claim 39, further comprising a red-eye correction module operable to correct red-eye in the digital image.

55. The system of claim 54, wherein the red-eye correction module operable to correct original color values of pixels corresponding to the final pixel mask by desaturating original color values.

56. The system of claim 55, wherein original color values of a given pixel are desaturated based on a luminance level of the given pixel.

57. The system of claim 55, wherein original color values of pixels identified as red in the final pixel mask are desaturated by respective amounts that vary with pixel location in the final pixel mask.

58. The system of claim 55, wherein original color values of a given pixel are desaturated based on a number of pixels in the final pixel mask neighboring the given pixel.

59. The system of claim 55, wherein the red-eye correction module operable to correct original color values of pixels corresponding to the final pixel mask by darkening the original color values.

60. The system of claim 59, wherein original color values of a given pixel are darkened by an amount decreasing with a luminance level of the given pixel.

61. A system for processing a digital image containing pixels with original color values, comprising:

a red-eye detection module operable to detect at least one candidate red-eye pixel area in the digital image;

a red-eye segmentation module operable to

classify pixels in each candidate red-eye pixel area based on redness and skin tone coloration,

for each candidate red-eye pixel area, generate an initial pixel mask identifying all pixels in the candidate red-eye pixel area classified as red and non-skin tone, and

generate a final pixel mask based on each initial pixel mask; and

a red-eye correction module operable to correct red-eye in the digital image by darkening and desaturating original color values identified as red in the final pixel mask, wherein original color values of pixels identified as red in the final pixel mask are desaturated by respective amounts that vary with location in the final pixel mask.

62. A method of processing a digital image containing pixels with original color values, comprising:

classifying an initial window region of the digital image as a red-eye region or a non-red-eye region;

adjusting size and location parameters of the window region to encompass a red-eye area of the digital image;

identifying in the adjusted window region red-eye pixels corresponding to an isolated red pixel island; and

correcting original color values of at least some of the red-eye pixels identified in the adjusted window region.

63. The method of claim 62, further comprising locating the initial window region in the digital image based on user input.

64. The method of claim 62, wherein the window region is classified based on a skin tone classification test.

65. The method of claim 62, wherein the window region is classified based on a red pixel classification test.

66. The method of claim 62, wherein the window region is classified based on a red region classification test.

67. The method of claim 62, wherein adjusting the size and location parameters of the initial window region comprises computing a ratio of YCbCr color space chrominance components for each pixel in an image area encompassing the window region, and iteratively applying a threshold to the computed ratios for each adjusted window.