Correction of "red-eye" effects in images

• US 20030007687A1
• Filed: 07/05/2001
• Published: 01/09/2003
• Est. Priority Date: 07/05/2001
• Status: Active Grant

• Current Assignee: Tahoe Research Limited (f/k/a Learndale Limited) (Vector Capital Corporation)
• Original Assignee: JASC Software (Corel Holdings LP)
• Inventors: Khvatov, Vladimir Alexandrovich, Nesterov, Victor Anatol?apos;evich, Lalyko, Leonid Borisovich, Zaklika, Krzysztof Antoni
• Granted Patent: US 6,980,691 B2
• Time in Patent Office: Days
• US Class Current: 382/167
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30216 Redeye defect
• CPC Class Codes: G06T 5/005 Retouching; Inpainting; Scratch removal detecting, correction, reducing or removing defects, e.g. non-responsive pixels of solid state image sensors H04N25/68, scratch removal for cinematographic films scanned by electronic image sensor H04N5/253
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: H04N 1/624 Red-eye correction control of camera based on recognised facial parts H04N23/611
• CPC Class Codes: H04N 9/3114 by using a sequential colour filter producing one colour at a time

Abstract

After identification of an image or image area where correction is to be effected for red-eye defects, and identification of eye regions within the image or the image area, color data from the area is provided, collected or analyzed according to color channel information. The color channel information may be defined for specific or collections of wavelength values or color content values, usually with at least two channels, up to four or more color channels used to carry all of the information (and with digital images, pixel information) for each color. By using color channel information, local color content information can be readily accessed and evaluated, and after the exact nature or degree of a red-eye effect is determined from evaluation of the color channel information, the data may be readily corrected by a number of different techniques. Templates of colors and/or templates for features (such as glint, pigmentation, or artistic features) may be added to identified regions to be corrected for red-eye

Claims

1. a method of automatically identifying a red-eye defect in a region of an image comprising classifying pixels within the region according to values of a ratio of color channels, red-eye defects being identified when values of the ratio exceed a predetermined value:

2. The method of claim 1 wherein a region of an image is first selected for analysis of values of color channel ratios to search for red-eye defects.

3. The method of claim 1 wherein the ratio of values of color channels comprises a ratio of the darkest color channel to the lightest color channel.

4. The method of claim 1 wherein the ratio of values of color channels comprises a ratio of the lightest color channel to the darkest color channel.

11. The process of claim 1 wherein the classifying of pixels is performed without reference to a specific color in the red-eye defect.

13. The method of claim 1 wherein the classification is effected by a procedure selected from the group consisting of segmentation, iterative segmentation, and iterative segmentation based on a threshold value of area ratios.

18. The method of claim 17 wherein lightness distribution within the outer portion of the virtual geometric shape that represents the iris is determined, and the correction of image data in the outer portion alters at least one property selected from hue and saturation without substantially changing the relative spatial distribution of lightness variation determined to have been in the outer portion of the virtual geometric shape that represents the iris.

19. The method of claim 17 wherein lightness distribution within the outer portion of the virtual geometric shape that represents the iris is determined, and the correction of image data in the outer portion alters hue without substantially changing the relative spatial distribution of lightness variation determined to have been in the outer portion of the virtual geometric shape that represents the iris.

20. A method of correcting a defect in the image of an eye comprising

defining a defect area by the method of claim 1;

drawing a virtual geometric shape whose dimensions approximate a dimension of the defect area to inscribe the defect area.;

assuming or defining a central portion of the virtual geometric shape to represent the pupil and an outer portion of the virtual circle to represent the iris;

providing different corrective image data to the central portion of the virtual geometric shape that represents the pupil as compared to corrective data provided to outer portion of the virtual circle representing the iris.

21. The method of claim 20 wherein lightness distribution within the outer portion of the virtual geometric shape that represents the iris is determined, and the correction of image data in the outer portion alters at least one property selected from hue and saturation without substantially changing the relative spatial distribution of lightness variation determined to have been in the outer portion of the virtual geometric shape that represents the iris.

26. A computer having a program therein that can effect a method for detecting identifying a red-eye defect, wherein the method comprises the method of claim 1.

29. A computer having a program therein that can effect a method for correcting red-eye defect, wherein the method comprises the method of claim 17.

5. A method of identifying a red-eye defect in a region of an image comprising classifying pixels within the region according to values of a ratio of color channels, red-eye defects being identified when values of the ratio exceed a predetermined value.

6. The method of claim 5 wherein a region of an image is first selected for analysis of values of color channel ratios to search for red-eye defects.

7. The method of claim 5 wherein the ratio of values of color channels comprises a ratio of the darkest color channel to the lightest color channel.

8. The method of claim 5 wherein the ratio of values of color channels comprises a ratio of the lightest color channel to the darkest color channel.

9. The method of claim 5 wherein the boundaries containing at least all pixels of a red-eye defect are identified by classifying pixels within the region according to values of a ratio of color channels, red-eye defects being identified when values of the ratio exceed a predetermined value.

10. The method of claim 5 wherein the boundaries containing at least all pixels of a red-eye defect are identified by classifying pixels within the region according to values of a ratio of color channels, red-eye defects being identified when values of the ratio do not exceed a predetermined value.

22. The method of claim 9 wherein pixels within the bounded region are classified into eye and non-eye pixels according to one of a plurality of pixel information channels, wherein the one channel is selected as the one of a plurality of channels that contains the largest number of pixels in said region that exceeds an area threshold at a predetermined measure of central tendency computed for each respective channel.

23. The method of claim 10 wherein pixels within the bounded region are classified into eye and non-eye pixels according to one of a plurality of pixel information channels, wherein the one channel is selected as the one of a plurality of channels that contains the largest number of pixels in said region that exceeds an area threshold at a predetermined measure of central tendency computed for each respective channel.

24. The method of claim 22 wherein after classification the eye pixels are corrected by

drawing a virtual geometric shape whose dimensions approximate a dimension of the defect area to inscribe the defect area.;

assuming or defining a central portion of the virtual geometric shape to represent the pupil and an outer portion of the virtual circle to represent the iris;

providing different corrective image data to the central portion of the virtual geometric shape that represents the pupil as compared to corrective data provided to outer portion of the virtual circle representing the iris.

25. The method of claim 23 wherein after classification the eye pixels are corrected by

drawing a virtual geometric shape whose dimensions approximate a dimension of the defect area to inscribe the defect area.;

assuming or defining a central portion of the virtual geometric shape to represent the pupil and an outer portion of the virtual circle to represent the iris;

providing different corrective image data to the central portion of the virtual geometric shape that represents the pupil as compared to corrective data provided to outer portion of the virtual circle representing the iris.

27. A computer having a program therein that can effect a method for correcting red-eye defect, wherein the method comprises the method of claim 9.

28. A computer having a program therein that can effect a method for correcting red-eye defect, wherein the method comprises the method of claim 10.

30. A computer having a program therein that can effect a method for correcting red-eye defect, wherein the method comprises the method of claim 22.

31. A computer having a program therein that can effect a method for correcting red-eye defect, wherein the method comprises the method of claim 23.

32. A computer having a program therein that can effect a method for correcting red-eye defect, wherein the method comprises the method of claim 24.

33. A computer having a program therein that can effect a method for correcting red-eye defect, wherein the method comprises the method of claim 25.

35. The method of claim 34 wherein selecting an image comprises selecting an area within an image that is less than the whole image.

36. The method of claim 34 wherein the comparison of the color content data comprises determining a ratio of at least two different-color content data values.

37. The method of claim 34 wherein the color channel data comprises data for at least red and one other color.

38. The method of claim 37 wherein the color channel data comprises data for at least red and at least one other color selected from the group consisting of green and blue.

39. The method of claim 34 wherein altering image data comprises replacing original image data with a template of color data.

40. The method of claim 34 wherein altering image data comprises replacing original image data in a region of the image identified as eyes with a template of color data.

41. The method of claim 34 wherein the image comprises an animal other than a human that has eyes.

42. The method of claim 41 wherein the red-eye effect on the eyes of the animal other than a human displays color defects other than a shift to excessive red content.

43. The method of claim 34 wherein an area having a defined geometric shape within the image is selected for comparison of data.

44. The method of claim 43 wherein a central area within said defined geometric shape has its image data altered to and chromatic color black.

45. The method of claim 44 wherein the area within said defined geometric shape corresponds to the pupil of an eye.

47. The method of claim 34 wherein the altering image data to reduce red-eye effects in the image comprises applying a template of color content to at least a portion of the region of the image wherein a correction for red-eye effects will be made.

48. The method of claim 47 wherein the correction for red-eye effects is made by choosing data points in at least two distinct regions of the region of the image wherein a correction for red-eye effects will be made, identifying a first sub-region wherein red-eye correction will be made and a region wherein red-eye correction will not be made, and applying the template of color content to those regions of the region of the image wherein a correction for red-eye effects will be made.

49. The method of claim 48 wherein the template of color content contains image data of a reflection of light that simulates a glint.

50. The method of claim 48 wherein in addition to the template of color content being added to the image, a separate template of image data of a reflection of light that simulates a glint is also added to the first sub-region wherein red-eye correction will be made.

51. The method of claim 34 wherein the identifying of a region of the image wherein a correction for red-eye effects will be made comprises delineating a red-eye defect area in an image that is independent of the specific color of the red-eye defect.

52. The method of claim 51 wherein delineating a red-eye defect area is performed by repeated segmentation of an image using a ratio formed from the darkest channel and the brightest color channel in the search area.

53. A method of claim 51 wherein improving the delineation of red-eye defect area by segmentation uses automatic selection of one of several image information representations characterized either by either the representations all tending to give a higher response value for the red-eye region than for the surroundings or by the representations all tending to give a lower response value for the red-eye region than for the surroundings.

54. The method of claim 34 comprising reconstructing a feature selected from the group consisting of a pupil, iris and glint of an eye subject to the red-eye effect when one or more of these features are missing.

55. The method of claim 34 wherein the color of the iris of an undamaged eye is changed, maintaining an original brightness variation in the iris.

57. The method of claim 56 wherein the first region of an image is selected as representing an eye.

58. The method of claim 57 wherein the first region selected as representing an eye comprises a pupil and an iris.

59. The method of claim 58 wherein a second region representing an iris is separated from a third region representing a pupil from within the first region.

60. The method of claim 59 wherein different colors are provided for the second region and the first region when color is modified in the first region.

61. The method of claim 59 wherein data from the image in the second region is provided at least in terms of brightness distribution within the second region.

62. The method of claim 61 wherein modification of color within the first region comprises altering at least the hue in the second region without altering the brightness distribution in the second region.

63. The method of claim 62 wherein saturation of the color within the second region is altered by operator selection.

64. A computer having a program therein that can effect a method for correcting red-eye defects in images, the method comprising the method of claim 34.

65. A computer having a program therein that can effect a method for correcting red-eye defects in images, the method comprising the method of claim 56.

66. A computer having a program therein that can effect a method for correcting red-eye defects in images, the method comprising the method of claim 59.

12. The method of claim I wherein the classification is effected by a procedure selected from the group consisting of segmentation, iterative segmentation, and iterative segmentation based on a threshold value of color channel ratios.

14. A method of automatically identifying an object within in a region of an image comprising classifying pixels within the region according to values of a ratio of color channels, an object being identified when values of the ratio exceed a predetermined value.

15. A method of selecting pixels from a digital image that has an eye defect, comprising circumscribing a region including all pixels of a portion of an image comprising an eye, and classifying pixels in this region into eye and non-eye pixels according to one of a plurality of pixel information channels, wherein the one channel is selected as the one of a plurality of channels that contains the largest number of pixels in said region that exceeds an area threshold at a predetermined measure of central tendency computed for each respective channel.

16. A method of selecting pixels from a digital image that has an eye defect, comprising circumscribing a region including all pixels of a portion of an image comprising an eye by classification of pixels to one of a plurality of pixel information channels, wherein the one channel is selected as the one of a plurality of channels that contains the largest number of pixels in said region that does not exceed an area threshold at a predetermined measure of central tendency computed for each respective channel.

17. A method of correcting a defect in the image of an eye comprising

defining a defect area;

drawing a virtual geometric shape whose dimensions approximates a dimension of the defect area to inscribe the defect area.;

assuming or defining a central portion of the virtual geometric shape to represent the pupil and an outer portion of the virtual circle to represent the iris;

providing different corrective image data to the central portion of the virtual geometric shape that represents the pupil as compared to corrective data provided to outer portion of the virtual circle representing the iris.

34. A method for correcting red-eye defects in images comprising:

selecting an image to have corrections for red-eye effects applied thereto;

identifying a region of the image wherein a correction for red-eye effects will be made;

providing data for the region of the image wherein a correction for red-eye effects will be made, the data comprising color channel data for at least two colors;

comparing the color content data for the at least two colors;

when the comparison indicates that a predetermined relationship has been exceeded or has not been met between the compared color content data for the at least two colors, altering image data to reduce red-eye effects in the image.

46. A method for correcting red-eye defects in an image comprising:

selecting an image to have corrections for red-eye effects applied thereto;

selecting a first area within the image that has image data of eyes that exhibit a red-eye defect;

providing data from within the first area within the image comprising color channel data for at least two colors;

comparing the color channel data within said first area for the at least two colors;

when the comparison of the color channel data for the at least two colors from within said first area indicates that a predetermined relationship has been exceeded or has not been met between the compared color channel data for the at least two colors, altering at least some image data within said first area to reduce red-eye effects in the image.

56. A method for modifying color within a region of an image comprising:

selecting an image to have modification of color applied thereto;

defining a first area within the image that has image data of a first region where color is to be modified;

providing data from within the first area within the image comprising color channel data for at least two colors;

comparing the color channel data within said first area for the at least two colors;

altering at least some image data within said first area to modify the color within the first region of the image.