Method and system for enhancing portrait images

• US 7,082,211 B2
• Filed: 05/31/2002
• Issued: 07/25/2006
• Est. Priority Date: 05/31/2002
• Status: Active Grant

• Current Assignee: Monument Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Eastman Kodak Company
• Inventors: Simon, Richard A., Nicponski, Henry, Bolin, Mark R., Matraszek, Tomasz
• Primary Examiner(s): Wu, Jingge
• Assistant Examiner(s): CARTER, AARON W
• Application Number:
  US10/160,421
• Publication Number:
  US 20030223622A1
• Time in Patent Office: 1,516 Days
• Field of Search: 382/118, 382254-275, 358/531
• US Class Current: 382/118
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2207/20012 Locally adaptive
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06V 10/30 Noise filtering
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: H04N 1/62 Retouching, i.e. modification of isolated colours only or in isolated picture areas only
• CPC Class Codes: H04N 1/628 Memory colours, e.g. skin or sky

Abstract

A retouching method for enhancing an appearance of a face located in a digital image involves acquiring a digital image containing one or more faces and detecting a location of facial feature points in the faces, where the facial feature points include points identifying salient features such as skin, eyes, eyebrows, nose, mouth, and hair. The location of the facial feature points are used to segment the face into different regions such as skin, eyes, eyebrows, nose, mouth, neck and hair regions. Facially relevant characteristics of the different regions are determined and, based on these facially relevant characteristics, an ensemble of enhancement filters are selected, each customized especially for a particular region, and the default parameters for the enhancement filters are selected. The enhancement filters are then executed on the particular regions, thereby producing an enhanced digital image from the digital image.

Claims

1. A method for enhancing the skin texture of a face appearing in a digital image, said method comprising the steps of:

(a) generating a luminance image from the digital image;

(b) using a valley/peak detector to detect skin features in the luminance image;

(c) classifying the skin features according to their feature-based characteristics;

(d) selecting relevant skin features for modification;

(e) modifying the relevant skin features using an adaptive interpolation procedure, thereby producing a modified image; and

(f) blending the digital image and the modified image to produce an enhanced image.

2. The method as claimed in claim 1 wherein the valley/peak detector used in step (b) has a spatial size that is dependent upon a size of the face.

3. The method as claimed in claim 2 wherein the skin features are classified in step (c) according to at least one of their size, shape, color and location in the skin region.

4. The method as claimed in claim 3 wherein relevant skin features are selected in step (d) based on their size being between a minimum and maximum size, in order to preserve skin texture.

5. The method as claimed in claim 4 wherein the minimum and maximum sizes are scaled to the size of the face.

6. The method as claimed in claim 3 wherein relevant skin features are selected in step (d) based on their size being larger than a specified minimum size, in order to preserve skin texture.

7. The method as claimed in claim 3 wherein the selected skin features from step (d) are adaptively dilated using morphological operators based upon the size of the feature.

8. The method as claimed in claim 1 wherein the step of modifying the relevant skin features in step (e) uses a directional interpolation filter having a pinwheel configuration;

representing line segment lengths.

9. The method as claimed in claim 8 wherein a minimum and maximum number of points on a line segment depends upon a size of the face.

10. The method as claimed in claim 8 wherein a minimum and maximum number of points on the line segment depends upon a size of the feature.

11. The method as claimed in claim 8 wherein the line segments do not cross a boundary of a facial region or a boundary of any skin feature not selected for modification in step (d).

12. The method as claimed in claim 8 wherein the line segments do not cross a boundary of any skin feature.

13. The method as claimed in claim 8 wherein processed pixel values for the relevant skin features are obtained by adaptively blending new pixel values obtained from the pinwheel filter with original pixel values from the digital image as a function of a size of the relevant skin feature.

14. The method as claimed in claim 3 wherein the step (f) of blending comprises adaptively blending the digital image and the modified image as a function of the size of a skin feature to produce an enhanced image.

15. The method as claimed in claim 3 wherein the step (f) of blending comprises adaptively blending the digital image and the modified image as a function of the location of a skin feature to produce an enhanced image.

16. The method as claimed in claim 1 wherein step (b) detects valley skin features and peak skin features and step (e) preferentially modifies valley skin features before modifying peak skin features.

17. A computer storage medium having instructions stored therein for causing a computer to perform the method of claim 1.

18. A method for enhancing the skin tone of a face appearing in a digital image, said method comprising the steps of:

(a) generating a blurred version of the digital image;

(b) generating a blending coefficient; and

(c) applying a blending operation to the digital image and its blurred version to generate an enhanced image with improved skin tone, wherein the blending operation uses the blending coefficient to combine the digital image and its blurred version;

wherein the blending coefficient is generated from a shadow/highlight strength image obtained from the digital image, said shadow/highlight strength image having a shadow/highlight strength value for each pixel of said digital image, and wherein the blending operation uses the blending coefficient to adaptively combine the digital image and its blurred version.

19. A method for enhancing the skin tone of a face appearing in a digital image, said method comprising the steps of:

(a) generating a blurred version of the digital image;

(b) generating a blending coefficient; and

(c) applying a blending operation to the digital image and its blurred version to generate an enhanced image with improved skin tone, wherein the blending operation uses the blending coefficient to combine the digital image and its blurred version;

wherein the blending coefficient is generated from a shadow/highlight strength image obtained from the digital image, and wherein the blending operation uses the blending coefficient to adaptively combine the digital image and its blurred version; and

wherein the step (b) of generating a blending coefficient comprises the steps of generating a luminance image from the digital image, generating an average luminance image from the luminance image, and subtracting the average luminance image from the luminance, image to generate a shadow/highlight image.

20. The method as claimed in claim 19 wherein the digital image is a color image having a plurality of color channels and the blending operation is applied to each channel of the color image.

21. A computer storage medium having instructions stored therein for causing a computer to perform the method of claim 19.

22. A method for enhancing the skin texture of a face appearing in a digital image, said method comprising the steps of:

generating a luminance image from the digital image;

using a valley/peak detector to detect a plurality of skin features in said luminance image;

classifying said skin features according to feature-based characteristics;

modifying one or more of said skin features using an adaptive interpolation procedure to produce a modified image; and

blending the digital image and said modified image to produce an enhanced image.

23. The method of claim 22 wherein said valley/peak detector has a spatial size that is dependent upon a size of the face.

24. The method of claim 23 wherein said feature-based characteristics include at least one of size, shape, color and location.

25. The method of claim 24 wherein said one or more skin features each have a size between predetermined minimum and maximum sizes.

26. The method of claim 25 wherein said predetermined minimum and maximum sizes are scaled to the size of the face.

27. The method of claim 24 wherein said one or more skin features are larger than a specified minimum size.

28. The method of claim 24 wherein said modifying further comprises adaptively dilating using morphological operators based upon respective size of said one or more skin features.

29. The method of claim 22 wherein said modifying further comprises using a directional interpolation filter.

30. The method of claim 29 wherein said filter has a pinwheel configuration representing line segment lengths.

31. The method of claim 30 further comprising determining a plurality of facial regions and wherein said pinwheel configuration has line segments that do not cross a boundary between two or more of said facial regions.

32. The method of claim 30 wherein said pinwheel configuration has line segments that do not cross into ones of said plurality of skin features excluded from said modifying.

33. The method of claim 30 wherein said pinwheel configuration has line segments that do not cross into said skin features.

34. The method of claim 33 wherein said blending is adaptive as a function of a size of the respective said modified skin feature.

35. The method of claim 33 wherein said blending is adaptive as a function of a size of the respective said modified skin feature.

36. The method as claimed in claim 22 wherein said using step detects valley skin features and peak skin features and said modifying further comprises modifying said valley skin features then modifying said peak skin features.

37. A computer storage medium having instructions stored therein for causing a computer to perform the method of claim 22.

38. A system for enhancing the skin texture of a face appearing in a digital image, said system comprising:

means for generating a luminance image from the digital image;

means for using a valley/peak detector to detect a plurality of skin features in said luminance image;

means for classifying said skin features according to feature-based characteristics;

means for modifying one or more of said skin features using an adaptive interpolation procedure to produce a modified image; and

means for blending the digital image and said modified image to produce an enhanced image.

39. The system of claim 38 wherein said valley/peak detector has a spatial size that is dependent upon a size of the face.

40. The system of claim 39 wherein said feature-based characteristics include at least one of size, shape, color and location.

41. The system of claim 40 wherein said one or more skin features each have a size between predetermined minimum and maximum sizes.

42. The system of claim 41 wherein said predetermined minimum and maximum sizes are scaled to the size of the face.

43. The system of claim 40 wherein said one or more skin features are larger than a specified minimum size.

44. The system of claim 40 wherein said means for modifying further comprises means for adaptively dilating using morphological operators based upon respective size of said one or more skin features.

45. The system of claim 38 wherein said means for modifying further comprises means for using a directional interpolation filter.

46. The system of claim 45 wherein said filter has a pinwheel configuration representing line segment lengths.

47. The system of claim 46 further comprising means for determining a plurality of facial regions and wherein said pinwheel configuration has line segments that do not cross a boundary between two or more of said facial regions.

48. The system of claim 46 wherein said pinwheel configuration has line segments that do not cross into ones of said plurality of skin features excluded from said modifying.

49. The system of claim 46 wherein said pinwheel configuration has line segments that do not cross into said skin features.

50. The system of claim 49 wherein said blending is adaptive as a function of a size of the respective said modified skin feature.

51. The system of claim 49 wherein said blending is adaptive as a function of a size of the respective said modified skin feature.

52. The system of claim 38 wherein said mean for using detects valley skin features and peak skin features and said means for modifying modifies in the order said valley skin features, and then, said peak skin features.

53. A method for enhancing the skin tone of a face in a digital image, said method comprising the steps of:

generating a blurred version of the digital image;

generating a blending coefficient; and

blending the digital image and said blurred version using said blending coefficient;

said method further comprising;

obtaining a shadow/highlight strength image from the digital image, said shadow/highlight strength image having a shadow/highlight strength value for each pixel of said digital image; and

generating said blending coefficient from said shadow/highlight strength image.

54. A method for enhancing the skin tone of a face in a digital image, said method comprising the steps of:

generating a blurred version of the digital image;

generating a blending coefficient; and

blending the digital image and said blurred version using said blending coefficient;

said method further comprising;

obtaining a shadow/highlight strength image from the digital image; and

generating said blending coefficient from said shadow/highlight strength image;

wherein said obtaining further comprises;

generating a luminance image from the digital image;

generating an average luminance image from said luminance image; and

subtracting said average luminance image from said luminance image to generate said shadow/highlight strength image.

55. The method of claim 54 wherein said blending is adaptive.

56. The method of claim 54 wherein the digital image is a color image having a plurality of color channels and said blending is applied to each channel of the color image.

57. A computer storage medium having instructions stored therein for causing a computer to perform the method of claim 54.

58. A system for enhancing the skin tone of a face in a digital image, said system comprising:

means for generating a blurred version of the digital image;

means for generating a blending coefficient; and

means for blending the digital image and said blurred version using said blending coefficient;

said system further comprising;

means for obtaining a shadow/highlight strength image from the digital image, said shadow/highlight strength image having a shadow/highlight strength value for each pixel of said digital image; and

means for generating said blending coefficient from said shadow/highlight strength image.

60. The system of claim 58 wherein said blending is adaptive.

61. The system of claim 58 wherein the digital image is a color image having a plurality of color channels and said blending is applied to each channel of the color image.

59. A system for enhancing the skin tone of a face in a digital image, said system comprising:

means for generating a blurred version of the digital image;

means for generating a blending coefficient; and

means for blending the digital image and said blurred version using said blending coefficient,

further comprising;

means for obtaining a shadow/highlight strength image from the digital image; and

means for generating said blending coefficient from said shadow/highlight strength image;

wherein said means for obtaining further comprises;

means for generating a luminance image from the digital image;

means for generating an average luminance image from said luminance image; and

means for subtracting said average luminance image from said luminance image to generate said shadow/highlight strength image.