Systems and methods of detecting and correcting redeye in an image suitable for embedded applications

• US 7,454,040 B2
• Filed: 08/29/2003
• Issued: 11/18/2008
• Est. Priority Date: 08/29/2003
• Status: Expired due to Fees

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Tretter, Daniel, Yen, Jonathan, Luo, Huitao, Pavel, Niaz R.
• Primary Examiner(s): Chawan; Sheela C
• Application Number:
  US10/653,021
• Publication Number:
  US 20050047656A1
• Time in Patent Office: 1,908 Days
• Field of Search: 382/117, 382/167, 382/165, 382/275, 382/164, 382/103, 382/274, 382/118, 382/199, 382/173, 382/162, 382/190, 382/115, 348/239, 348/241, 396/191, 396/198, 426/72, 426/311, 426/620, 426/640, 426/641, 426/658
• US Class Current: 382/117
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: H04N 1/62 Retouching, i.e. modification of isolated colours only or in isolated picture areas only
• CPC Class Codes: H04N 1/624 Red-eye correction control of camera based on recognised facial parts H04N23/611

Abstract

Systems and methods of detecting and correcting redeye in an image are described. In one aspect, the input image is sub-sampled to generate a thumbnail image, redeye pixel areas are detected in the thumbnail image. In another aspect, an input image having lines of pixels with original color values is processed. One or more redeye pixel areas corresponding to respective areas in the input image are detected. Each pixel in the input image corresponding to the detected redeye pixel areas is classified as a redeye pixel or a non-redeye pixel on a line-by-line basis without reference to pixels in adjacent lines. The original color values of pixels in the input image classified as redeye pixels are corrected.

Claims

1. A method of processing an input image, comprising:

sub-sampling the input image to generate a thumbnail image comprising a reduced-size version of the input image in its entirety;

detecting redeye pixel areas in the thumbnail image, wherein detecting redeye pixel areas comprises determining measures of pixel redness in the thumbnail image, and identifying a first set of candidate redeye pixel areas based on the determined pixel redness measures;

correcting redeye in a version of the input image based on the redeye pixel areas detected in the thumbnail image to produce a corrected version of the input image; and

producing an output image from the corrected version of the input image.

2. The method of claim 1, wherein pixel redness measures are determined based on a ratio of a measure of a red component of pixel energy to a measure of total pixel energy.

3. The method of claim 1, wherein identifying the first set of candidate redeye pixel areas comprises applying a two-dimensional redness filter to the determined pixel redness measures, wherein the redness filter is operable to determine for each region of the thumbnail image a respective redness score based on a comparison of an average of ones of the pixel redness measures in a central kernel pixel area of the thumbnail image and an average of the pixel redness measures in a pixel area of the thumbnail image surrounding the kernel pixel area.

4. The method of claim 3, further comprising applying a prescribed threshold to the determined redness scores to identify candidate redeye pixels.

5. The method of claim 4, wherein detecting redeye pixel areas further

comprises segmenting redeye pixels by scanning a redness map of the redness measures in stripes of one or more pixel lines and tracking objects containing candidate redeye pixels connected across stripes.

6. The method of claim 3, wherein detecting redeye pixel areas further comprises filtering from the first set each candidate redeye pixel area having a determined redness contrast relative to at least one respective neighboring pixel area less than a prescribed redness contrast threshold.

7. The method of claim 6, wherein each candidate redeye pixel area having a determined redness contrast relative to each of a set of corresponding surrounding pixel areas less than the prescribed redness contrast threshold is filtered from the first set.

10. The method of claim 1, wherein detecting redeye pixel areas further comprises filtering from the first set each candidate redeye pixel area located in an area of the digital image having a determined grayscale contrast relative to at least one respective neighboring pixel area less than a prescribed grayscale contrast threshold.

11. The method of claim 10, further comprising:

determining measures of pixel grayscale in the digital image;

determining, for a given candidate redeye pixel area, a candidate iris area centered at the given candidate redeye pixel area and having a size maximizing grayscale contrast between the candidate iris area and areas surrounding the candidate iris area;

determining a measure of grayscale contrast between the candidate iris area and at least a portion of the areas surrounding the candidate iris area;

and applying a threshold to the determined grayscale contrast measure to filter candidate redeye pixel areas from the first set.

12. The method of claim 1, further comprising:

identifying a pixel boundary of a pixel region surrounding a given candidate redeye pixel area;

classifying pixels within the pixel boundary as red pixels and non-red pixels by applying a threshold to the determined pixel redness measures; and

filtering the given candidate redeye pixel area from the first set when a set of contiguous red pixels extends from the given candidate redeye pixel area to the pixel boundary.

13. The method of claim 12, further comprising identifying the set of

contiguous pixels by scanning a redness map of the redness measures in stripes of one or more pixel lines and tracking objects containing red pixels connected across stripes.

14. The method of claim 1, further comprising filtering candidate redeye pixel areas from the first set based on proportions of detected skin tone pixels in regions respectively surrounding the candidate redeye pixels areas.

15. The method of claim 1, further comprising pairing candidate redeye pixel areas in the first set, and filtering unpaired candidate redeye pixels areas from the first set.

16. The method of claim 15, wherein pairing candidate redeye pixel areas comprises comparing a candidate texture pattern determined for a candidate pair of candidate redeye pixel areas in the first set with a reference texture pattern.

17. The method of claim 1, further comprising detecting redeye pixel areas in the input image, and generating a set of detected redeye pixel areas by merging redeye pixel areas detected in the input image with redeye pixel areas detected in the thumbnail image.

18. The method of claim 1, wherein the correcting redeye comprises mapping the detected redeye pixel areas to the version of the input image.

19. The method of claim 18, wherein the correcting redeye comprises enlarging redeye pixel areas mapped to the version of the input image.

20. The method of claim 19, wherein the mapped redeye pixel areas are enlarged by amounts decreasing inversely with respect to original sizes of the mapped redeye pixel areas.

21. The method of claim 19, further comprising cropping corners from each of the enlarged redeye pixel areas.

22. The method of claim 19, wherein the correcting comprises classifying pixels in each of the mapped redeye pixel areas based on a redness threshold.

23. The method of claim 19, wherein the pixels in each of the mapped redeye pixel areas are classified on a pixel-by-pixel basis.

24. The method of claim 19, wherein each pixel in the mapped redeye pixel areas is classified with reference to an adjacent, previously-classified pixel.

25. The method of claim 19, wherein the correcting comprises correcting original color values of pixels in the mapped redeye pixel areas based on integer arithmetic computations.

26. The method of claim 18, further comprising classifying pixels as redeye pixels for correction before mapping detected redeye pixel areas to the version of the input image.

27. The method of claim 1, further comprising correcting redeye in the thumbnail image based on redeye pixel areas detected in the thumbnail image.

28. The method of claim 27, further comprising displaying the thumbnail image it corrected redeye, and correcting redeye in the input image based on redeye pixel areas detected in the thumbnail image in respond to a user command.

8. A method of processing an input image, comprising:

sub-sampling the input image to generate a thumbnail image;

detecting redeye pixel areas in the thumbnail image, wherein detecting redeye pixel areas comprises determining measures of pixel redness in the thumbnail image, and identifying a first set of candidate redeye pixel areas based on the determined pixel redness measures, wherein identifying the first set of candidate redeye pixel areas comprises enlarging a given candidate redeye pixel area having a dimension below a threshold size to generate an enlarged pixel area;

correcting redeye in a version of the input image based on the redeye pixel areas detected in the thumbnail image to produce a corrected version of the input image; and

producing an output image from the corrected version of the input image.

9. The method of claim 8, wherein identifying the first set of candidate redeye pixel areas comprises comparing the enlarged pixel area to multiple pixel areas surrounding the enlarged pixel area, and selecting a pixel area to replace the given candidate redeye pixel area from among the enlarged pixel area and the surrounding pixel areas based on measures of redness determined for each of the enlarged pixel area and the surrounding pixel areas.

29. A method of processing an input image, comprising:

sub-sampling the input image to generate a thumbnail image;

detecting redeye pixel areas in the thumbnail image, wherein detecting redeye pixel areas comprises

determining measures of pixel redness in the thumbnail image,

identifying a first set of candidate redeye pixel areas based on the determined pixel redness measures,

pairing candidate redeye pixel areas in the first set, wherein pairing

candidate redeye pixel areas comprises comparing a candidate texture pattern determined for a candidate pair of candidate redeye pixel areas in the first set with a reference texture pattern,

wherein comparing the candidate texture pattern with the reference texture pattern comprises generating a feature vector representative of the candidate texture pattern and comparing the generated feature vector with a statistical model of the reference texture pattern, and

filtering unpaired candidate redeye pixels areas from the first set;

correcting redeye in a version of the input image based on the redeye pixel areas detected in the thumbnail image to produce a corrected version of the input image; and

producing an output image from the connected version of the input image.

30. The method of claim 29, wherein generating the feature vector representative of the candidate texture pattern comprises mapping a candidate redeye pair region encompassing the candidate redeye pail to a standardized candidate redeye pair template.

31. The method of claim 30, wherein mapping the candidate redeye pair region comprises cropping a pixel region from a grayscale map of the thumbnail image, rotating the cropped pixel region, and scaling the rotated pixel region.

32. The method of claim 31, wherein mapping the candidate redeye pair region comprises normalizing and equalizing the scaled pixel region.

33. The method of claim 30, wherein generating the feature vector representative of the candidate texture pattern comprises converting the mapped candidate redeye pair region to the feature vector.

34. A method of processing an input image, comprising:

sub-sampling the input image to generate a thumbnail image;

detecting redeye pixel areas in the thumbnail image;

correcting redeye in a version of the input image based on redeye pixel areas detected in the thumbnail image to produce a corrected version of the input image, wherein correcting redeye comprises identifying discrete redeye pixel areas separated from eyelid regions; and

producing an output image from the corrected version of the input image.

35. The method of claim 34, wherein identifying discrete redeye pixel areas comprises comparing at least one redeye pixel area size dimension to a threshold.

36. The method of claim 35, wherein a discrete redeye pixel area is identified based at least in part on a prescribed fraction of a respective grayscale iris area centered at a corresponding pixel area and having a size maximizing grayscale contrast between the grayscale iris area and areas surrounding the grayscale iris area.

37. The method of claim 34, wherein correcting redeye comprises classifying pixels in each non-discrete redeye pixel area based on skin tone coloration.

38. A method of processing an input image, comprising:

sub-sampling the input image to generate a thumbnail image;

detecting redeye pixel areas in the thumbnail image;

correcting redeye in a version of the input image based on redeye pixel areas detected in the thumbnail image to produce a corrected version of the input image, wherein correcting redeye comprises classifying pixels between concentric inner and outer bounding regions based on a grayscale threshold; and

producing an output image from the corrected version of the input image.

39. The method of claim 38, further comprising correcting original color values of pixels in a redeye pixel correction region encompassing pixels classified as redeye pixels.

40. The method of claim 39, wherein original color values of pixels in the redeye pixel correction region are corrected by desaturating original color values.

41. The method of claim 40, wherein the desaturating comprises desaturating original color values by respective amounts varying with pixel location in the final pixel mask.

42. The method of claim 40, wherein the correcting comprises darkening original color values of pixels in the redeye pixel correction region.

43. The method of claim 39, further comprising correcting original color values of pixels in a smoothing region surrounding the redeye pixel correction region.

44. The method of claim 43, wherein the correcting comprises correcting original color values of pixels in the smoothing region by an amount decreasing with distance from the given redeye pixel correction region.

45. The method of claim 44, wherein the correcting comprises correcting original color values of pixels in the redeye pixel correction region without reference to position within the redeye pixel correction region.

46. The method of claim 38, further comprising determining a size of the inner bounding region between a given redeye pixel area size and a corresponding grayscale iris area size, and determining a size of the outer bounding region larger than the determined size of the inner bounding region by a predetermined relative amount.

47. A method of processing an input image having lines of pixels with original color values, comprising:

detecting one or more redeye pixel areas corresponding to respective areas in the input image;

in a version of the input image, classifying each pixel corresponding to the detected redeye pixel areas as a redeye pixel or a non-redeye pixel on a line-by-line basis without reference to pixels in adjacent lines;

correcting the original color values of pixels classified as redeye pixels in the version of the input image to produce a corrected version of the input image; and

producing an output image from tire corrected version of the input image.

48. The method of claim 47, wherein a pixel in a given line is classified with reference to an adjacent, previously-classified pixel in the given line.

49. The method of claim 47, wherein correcting redeye comprises identifying discrete redeye pixel areas separated from eyelid regions.

50. The method of claim 49, wherein identifying discrete redeye pixel areas comprises comparing at least one redeye pixel area size dimension to a threshold.

51. The method of claim 50, wherein a discrete redeye pixel area is identified based at least in part on a prescribed fraction of a respective grayscale iris area centered at a corresponding pixel area and having a size maximizing grayscale contrast between the grayscale iris area and areas surrounding the grayscale iris area.

52. The method of claim 49, wherein correcting redeye comprises classifying pixels in each non-discrete redeye pixel area based on skin tone coloration.

53. The method of claim 47, wherein correcting redeye comprises classifying pixels in each redeye pixel area based on a redness threshold.

54. The method of claim 47, further comprising correcting original color values of pixels classified as redeye pixels by desaturating original color values.

55. The method of claim 47, further comprising correcting original color values of pixels classified as redeye pixels by darkening the original color values.

56. A system for processing an input image, comprising computing hardware operable to perform operations comprising:

sub-sampling the input image to generate a thumbnail image comprising reduced-size versions of all regions of the input image;

detecting redeye pixel areas in the thumbnail image, wherein in the detecting the computing hardware is operable to perform operations comprising determining measures of pixel redness in the thumbnail image, and identifying a first set of candidate redeye pixel areas based on the determined pixel redness measures;

correcting redeye in a version of the input image based on the redeye pixel areas detected in the thumbnail image to produce a corrected version of the input image.

57. The system of claim 56, wherein the computing hardware is operable to perform operations comprising applying a two-dimensional redness filter to the determined pixel redness measures, wherein the redness filter is operable to determine a redness score based on a central kernel pixel area and a pixel area surrounding the kernel pixel area.

58. The system of claim 56, wherein the computing hardware operable to perform operations comprising filtering candidate redeye pixel areas from the first set based on proportions of detected skin tone pixels in regions respectively surrounding the candidate redeye pixels areas.

59. The system of claim 56, wherein the computing hardware operable to perform operations comprising pairing candidate redeye pixel areas in the first set and filters unpaired candidate redeye pixels areas from the first set.

60. The system of claim 56, wherein the computing hardware operable to perform operations comprising detecting redeye pixel areas in the input image and generates a set of detected redeye pixel areas by merging redeye pixel areas detected in the input image with redeye pixel areas detected in the thumbnail image.

61. A system for processing an input image, comprising computing hardware operable to perform operations comprising

sub-sampling the input image to generate a thumbnail image;

detecting redeye pixel areas in the thumbnail image, wherein in the detecting the computing hardware is operable to perform operations comprising determining measures of pixel redness in the thumbnail image, identifying a first set of candidate redeye pixel areas based on the determined pixel redness measures, and enlarging a given candidate redeye pixel area having a dimension below a threshold size to generate an enlarged pixel area; and

correcting redeye in a version of the input image based on the redeye pixel areas detected in the thumbnail image to produce a corrected version of the input image.

62. A system for processing an input image, comprising computing hardware operable to perform operations comprising:

sub-sampling the input image to generate a thumbnail image;

detecting redeye pixel areas in the thumbnail image, wherein in the detecting the computing hardware is operable to perform operations comprising determining measures of pixel redness in the thumbnail image, identifying a first set of candidate redeye pixel areas based on the determined pixel redness measures, and filtering from the first set each candidate redeye pixel area located in an area of the digital image having a determined grayscale contrast relative to at least one respective neighboring pixel area less than a prescribed grayscale contrast threshold;

correcting redeye in a version of the input image based on the redeye pixel areas detected in the thumbnail image to produce a corrected version of the input image.

63. A system for processing an input image, comprising computing hardware operable to perform operations comprising:

sub-sampling the input image to generate a thumbnail image;

detecting redeye pixel areas in the thumbnail image, wherein in the detecting the computing hardware is operable to perform operations comprising

determining measures of pixel redness in the thumbnail image,

identifying a first set of candidate redeye pixel areas based on the determined pixel redness measures,

identifying a pixel boundary of a pixel region surrounding a given candidate redeye pixel area;

classifying pixels within the pixel boundary as red pixels and non-red pixels by applying a threshold to the determined pixel redness measures, and

filtering the given candidate redeye pixel area from the first set when a set of contiguous red pixels extends from the given candidate redeye pixel area to the pixel boundary; and

correcting redeye in a version of the input image based on the redeye pixel areas detected in the thumbnail image to produce a corrected version of the input image.

64. A system for processing an input image having lines of pixels with original color values, comprising computing hardware operable to perform operations comprising:

detecting one or more redeye pixel areas corresponding to respective areas in the input image;

classifying each pixel in the input image corresponding to the detected redeye pixel areas as a redeye pixel or a non-redeye pixel on a line-by-line basis without reference to pixels in adjacent lines; and

correcting the original color values of pixels in the input image classified as redeye pixels to produce a corrected version of the input image.