Method and system for enhancing portrait images

• US 20030223622A1
• Filed: 05/31/2002
• Published: 12/04/2003
• Est. Priority Date: 05/31/2002
• Status: Active Grant

• Current Assignee: Monument Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Eastman Kodak Company
• Inventors: Nicponski, Henry, Simon, Richard A., Bolin, Mark R., Matraszek, Tomasz
• Granted Patent: US 7,082,211 B2
• Time in Patent Office: Days
• US Class Current: 382/118
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2207/20012 Locally adaptive
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 5/005 Retouching; Inpainting; Scratch removal detecting, correction, reducing or removing defects, e.g. non-responsive pixels of solid state image sensors H04N25/68, scratch removal for cinematographic films scanned by electronic image sensor H04N5/253
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06V 10/30 Noise filtering
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: H04N 1/62 Retouching, i.e. modification of isolated colours only or in isolated picture areas only
• CPC Class Codes: H04N 1/628 Memory colours, e.g. skin or sky

Abstract

A retouching method for enhancing an appearance of a face located in a digital image involves acquiring a digital image containing one or more faces and detecting a location of facial feature points in the faces, where the facial feature points include points identifying salient features such as skin, eyes, eyebrows, nose, mouth, and hair. The location of the facial feature points are used to segment the face into different regions such as skin, eyes, eyebrows, nose, mouth, neck and hair regions. Facially relevant characteristics of the different regions are determined and, based on these facially relevant characteristics, an ensemble of enhancement filters are selected, each customized especially for a particular region, and the default parameters for the enhancement filters are selected. The enhancement filters are then executed on the particular regions, thereby producing an enhanced digital image from the digital image.

Claims

1. A method for enhancing an appearance of a face located in a digital image, said method comprising the steps of:

(a) acquiring a digital image containing one or more faces;

(b) detecting a location of facial feature points in the one or more faces, said facial feature points including points identifying salient features including one or more of skin, eyes, eyebrows, nose, mouth, and hair;

(c) using the location of the facial feature points to segment the face into different regions, said different regions including one or more of skin, eyes, eyebrows, nose, mouth, neck and hair regions;

(d) determining one or more facially relevant characteristics of the different regions;

(e) based on the facially relevant characteristics of the different regions, selecting (1) two or more enhancement filters each customized especially for a particular region and (2) the default parameters for the enhancement filters; and

(f) executing the enhancement filters on the particular regions, thereby producing an enhanced digital image from the digital image.

2. The method as claimed in claim 1 further comprising the step of (g) applying an image utilization process to the enhanced digital image.

3. The method as claimed in claim 1 wherein the enhancement filters include at least two of texture, skin tone, eye, teeth and shape filters.

4. The method as claimed in claim 1 wherein a gender classification algorithm is applied to determine the gender of the faces prior to the selection of the enhancement filters in step (e).

5. The method as claimed in claim 1 wherein an age classification algorithm is applied to determine the age of the faces prior to the selection of the enhancement filters in step (e).

6. The method as claimed in claim 1 wherein a distance between the eyes of the face is used to determine the default parameters for one or more of the enhancement filters selected in step (e).

7. The method as claimed in claim 2 wherein the utilization process applied in step (g) includes printing the image on a local device.

8. The method as claimed in claim 2 wherein the utilization process applied in step (g) includes archiving the image on a local device.

9. The method as claimed in claim 2 wherein the utilization process applied in step (g) includes printing the image on a remote device.

10. The method as claimed in claim 2 wherein the utilization process applied in step (g) includes archiving the image on a remote device.

11. The method as claimed in claim 1 wherein the facial characteristics determined in step (d) include at least one characteristic selected from the group including a size of the face, a distance between facial features in a particular region, a gender of a subject in the image, and an age of the subject.

12. A computer storage medium having instructions stored therein for causing a computer to perform the method of claim 1.

13. A user-interactive retouching method for enhancing an appearance of a face located in a digital image, said method comprising the steps of:

(a) acquiring and displaying a digital image containing one or more faces;

(b) detecting facial feature points in the one or more faces, said facial feature points including points identifying salient features including one or more of skin, eyes, nose, mouth, and hair;

(c) displaying the location of the facial feature points by overlaying the facial feature points on the displayed digital image;

(d) providing an optional user-initiated adjustment of the location of the facial feature points;

(e) using the location of the facial feature points to segment the face into different regions, said different regions including one or more of skin, eyes, nose, mouth and hair regions;

(f) determining one or more facially relevant characteristics of the different regions;

(g) based on the facially relevant characteristics of the different regions, selecting (1) two or more enhancement filters customized especially for each region and (2) the default parameters for the enhancement;

(h) executing the enhancement filters on the regions, thereby producing an enhanced digital image from the digital image;

(i) displaying the enhanced digital image;

(j) providing an optional user-initiated approval of the displayed enhanced digital image, whereby if the image is not approved at least one of steps (d), (e), (f) and (g) are repeated with adjusted regions or adjustments to the default parameters of the filters and step (h) is re-executed on the segmented regions; and

(k) applying an image utilization process to the enhanced digital image.

14. The method as claimed in claim 13 wherein the enhancement filters include at least two of texture, skin tone, eye, teeth and shape filters.

15. The method as claimed in claim 13 wherein a gender classification algorithm is applied to determine the gender of the faces prior to the selection of the enhancement filters in step (g).

16. The method as claimed in claim 13 wherein an age classification algorithm is applied to determine the age of the faces prior to the selection of the enhancement filters in step (g).

17. The method as claimed in claim 13 wherein a distance between the eyes of the face is used to determine the default parameters for one or more of the enhancement filters selected in step (g).

18. The method as claimed in claim 13 wherein the utilization process applied in step (k) includes printing the image on a local device.

19. The method as claimed in claim 13 wherein the utilization process applied in step (k) includes archiving the image on a local device.

20. The method as claimed in claim 13 wherein the utilization process applied in step (k) includes printing the image on a remote device.

21. The method as claimed in claim 13 wherein the utilization process applied in step (k) includes archiving the image on a remote device.

22. The method as claimed in claim 13 wherein the facially relevant characteristics determined in step (f) include at least one characteristic selected from the group including a size of the face, a distance between facial features in a particular region, a gender of a subject in the digital image, and an age of the subject.

23. A retouching method for enhancing an appearance of a face located in a digital image, said method comprising the steps of:

(a) acquiring a digital image containing one or more faces;

(b) detecting facial feature points in the one or more faces, said facial feature points including points identifying salient facial features including eyes, nose, mouth, or hair but not including skin;

(c) using the location of the salient facial feature points to identify a skin region of the face, wherein said skin region excludes the salient facial features;

(d) based on the identified skin region, selecting an enhancement filter and one or more default parameters for the filter for correcting a predefined skin type feature; and

(e) executing the enhancement filter on the skin region, thereby producing an enhanced digital image from the source digital image.

24. The method as claimed in claim 23 further comprising the steps of:

selecting two or more enhancement filters and default parameters for the filters for correcting a predefined plurality of particular skin type features;

analyzing each pixel in the skin region and detecting feature pixels each having one of the predefined plurality of particular skin type features and labelling each feature pixel in the skin region with its corresponding particular skin type feature; and

enhancing each feature pixel with the enhancement filter appropriate for the particular skin type feature.

25. The method as claimed in claim 23 wherein the skin region identified in step (c) includes a neck region.

26. A computer storage medium having instructions stored therein for causing a computer to perform the method of claim 23.

27. A system for enhancing an appearance of one or more faces appearing in a digital image, said system comprising:

means for detecting a location of facial feature points in the one or more faces, said facial feature points including points identifying salient features including one or more of skin, eyes, eyebrows, nose, mouth, and hair;

means for using the location of the facial feature points to segment the face into different regions, said different regions including one or more of skin, eyes, eyebrows, nose, mouth, neck and hair regions;

means for determining one or more facially relevant characteristics of the different regions;

means for, based on the facially relevant characteristics of the different regions, selecting (1) two or more enhancement filters each customized especially for a particular region and (2) the default parameters for the enhancement filters; and

means for executing the enhancement filters on the particular regions, thereby producing an enhanced digital image from the digital image.

28. The system as claimed in claim 27 further comprising means for utilizing the enhanced digital image, wherein the utilizing means includes a printer.

29. The system as claimed in claim 27 further comprising means for utilizing the enhanced digital image, wherein the utilizing means includes a storage device.

30. The system as claimed in claim 27 wherein the enhancement filters include at least two of texture, skin tone, eye, teeth and shape filters.

31. The method as claimed in claim 27 wherein said means for determining the facial characteristics identifies at least one characteristic selected from the group including a size of the face, a distance between facial features in a particular region, a gender of a subject in the image, and an age of the subject.

32. A user-interactive retouching system for enhancing an appearance of one or more faces appearing in a digital image, said system comprising:

means for detecting facial feature points in the one or more faces, said facial feature points including points identifying salient features including one or more of skin, eyes, nose, mouth, and hair;

a display for displaying the location of the facial feature points by overlaying the facial feature points on the displayed digital image;

a user interface for providing an optional user-initiated adjustment of the location of the facial feature points;

means for using the location of the facial feature points to segment the face into different regions, said different regions including one or more of skin, eyes, nose, mouth and hair regions;

means for determining one or more facially relevant characteristics of the different regions;

means, based on the facially relevant characteristics of the different regions, for selecting (1) two or more enhancement filters customized especially for each region and (2) the default parameters for the enhancement;

means for executing the enhancement filters on the regions, thereby producing an enhanced digital image from the source digital image, and displaying the enhanced digital image, whereby the user interface provides an optional user-initiated approval of the displayed enhanced digital image and, if the image is not approved, consequent adjustment of the location of feature points or default parameters.

33. An automatic retouching method for enhancing an appearance of a face located in a digital image, said method comprising the steps of:

(a) acquiring a digital image containing one or more faces;

(b) detecting a location of facial feature points in the one or more faces, said facial feature points including points identifying salient features including the skin;

(c) providing a generic probability map for a neck region;

(d) scaling the generic probability map for the neck region to fit the feature points generated in step (b), thereby providing a scaled probability map for the neck region; and

(e) based on the scaled probability map for the neck region, using an enhancement filter to enhance at least the texture or the skin tone of the neck region, thereby producing an enhanced digital image from the digital image.

34. The method as claimed in claim 33 wherein the facial feature points detected in step (b) identify at least one of a chin line and a face outline.

35. The method as claimed in claim 33 wherein the step (b) of detecting a location of facial feature points in the one or more faces includes feature points identifying a chin line, said method further comprising the steps of:

segmenting the digital image into pixels within a skin region, thereby producing a skin probability distribution;

using the skin probability distribution and pixels below the chin line to generate a skin color based probability map;

generating a final probability map for the neck region by combining the scaled probability map for the neck region and the skin color based probability map; and

based on the final probability map for the neck region, using an enhancement filter to enhance at least the texture or the skin tone of the neck region, thereby producing an enhanced digital image from the digital image.

36. A computer storage medium having instructions stored therein for causing a computer to perform the method of claim 33.

37. An automatic retouching method for enhancing an appearance of a face located in a digital image, said method comprising the steps of:

(a) acquiring a digital image containing one or more faces;

(b) detecting a location of facial feature points in the one or more faces, including feature points identifying a chin line;

(c) segmenting the digital image into pixels within a skin region, thereby producing a skin probability distribution;

(d) using the skin probability distribution and pixels below the chin line to generate a skin color based probability map for the neck region; and

(e) based on the skin color based probability map for the neck region, using an enhancement filter to enhance at least the texture or the skin tone of the neck region, thereby producing an enhanced digital image from the digital image.

38. The method as claimed in claim 37 further comprising the steps of:

providing a generic probability map for a neck region;

scaling the generic probability map for the neck region to fit the feature points generated in step (b), thereby providing a scaled probability map for the neck region;

generating a final probability map for the neck region by combining the scaled probability map for the neck region and the skin color based probability map; and

based on the final probability map for the neck region, using an enhancement filter to enhance at least the texture or the skin tone of the neck region, thereby producing an enhanced digital image from the digital image.

39. A computer storage medium having instructions stored therein for causing a computer to perform the method of claim 37.

40. A method for enhancing the skin texture of a face appearing in a digital image, said method comprising the steps of:

(a) generating a luminance image from the digital image;

(b) using a valley/peak detector to detect skin features in the luminance image;

(c) classifying the skin features according to their feature-based characteristics;

(d) selecting relevant skin features for modification;

(e) modifying the relevant skin features using an adaptive interpolation procedure, thereby producing a modified image; and

(f) blending the digital image and the modified image to produce an enhanced image.

41. The method as claimed in claim 40 wherein the valley/peak detector used in step (b) has a spatial size that is dependent upon a size of the face.

42. The method as claimed in claim 41 wherein the skin features are classified in step (c) according to at least one of their size, shape, color and location in the skin region.

43. The method as claimed in claim 42 wherein relevant skin features are selected in step (d) based on their size being between a minimum and maximum size, in order to preserve skin texture.

44. The method as claimed in claim 43 wherein the minimum and maximum sizes are scaled to the size of the face.

45. The method as claimed in claim 42 wherein relevant skin features are selected in step (d) based on their size being larger than a specified minimum size, in order to preserve skin texture.

46. The method as claimed in claim 42 wherein the selected skin features from step (d) are adaptively dilated using morphological operators based upon the size of the feature.

47. The method as claimed in claim 40 wherein the step of modifying the relevant skin features in step (e) uses a directional interpolation filter having a pinwheel configuration representing line segment lengths.

48. The method as claimed in claim 47 wherein a minimum and maximum number of points on a line segment depends upon a size of the face.

49. The method as claimed in claim 47 wherein a minimum and maximum number of points on the line segment depends upon a size of the feature.

50. The method as claimed in claim 47 wherein the line segments do not cross a boundary of a facial region or a boundary of any skin feature not selected for modification in step (d).

51. The method as claimed in claim 47 wherein the line segments do not cross a boundary of any skin feature.

52. The method as claimed in claim 47 wherein processed pixel values for the relevant skin features are obtained by adaptively blending new pixel values obtained from the pinwheel filter with original pixel values from the digital image as a function of a size of the relevant skin feature.

53. The method as claimed in claim 42 wherein the step (f) of blending comprises adaptively blending the digital image and the modified image as a function of the size of a skin feature to produce an enhanced image.

54. The method as claimed in claim 42 wherein the step (f) of blending comprises adaptively blending the digital image and the modified image as a function of the location of a skin feature to produce an enhanced image.

55. The method as claimed in claim 40 wherein step (b) detects valley skin features and peak skin features and step (e) preferentially modifies valley skin features before modifying peak skin features.

56. A computer storage medium having instructions stored therein for causing a computer to perform the method of claim 40.

57. A method for enhancing the skin tone of a face appearing in a digital image, said method comprising the steps of:

(a) generating a blurred version of the digital image;

(b) generating a blending coefficient; and

(c) applying a blending operation to the digital image and its blurred version to generate an enhanced image with improved skin tone, wherein the blending operation uses the blending coefficient to combine the digital image and its blurred version.

58. The method as claimed in claim 57 wherein the blending coefficient is a constant value.

59. The method as claimed in claim 57 wherein the blending coefficient is generated from a shadow/highlight strength image obtained from the digital image, and wherein the blending operation uses the blending coefficient to adaptively combine the digital image and its blurred version.

60. The method as claimed in claim 59 wherein the step (b) of generating a blending coefficient comprises the steps of generating a luminance image from the digital image, generating an average luminance image from the luminance image, and subtracting the average luminance image from the luminance image to generate a shadow/highlight image.

61. The method as claimed in claim 57 wherein the digital image is a color image having a plurality of color channels and the blending operation is applied to each channel of the color image.

62. A computer storage medium having instructions stored therein for causing a computer to perform the method of claim 57.

63. A method for enhancing the luminance and chrominance of facial regions in a face located in a digital image, said method comprising the steps of:

(a) acquiring a digital color image containing one or more faces;

(b) detecting a location of facial feature points in the one or more faces, said facial feature points including points identifying salient features including at least one of eye and teeth regions;

(c) using the facial feature points to generate one or more masks identifying pixels corresponding to at least one of the eye and teeth regions;

(d) using a valley/peak detector to generate a valley/peak map from the digital image;

(e) using the valley/peak map from the digital image to calculate one or more scaling factors for luminance and chrominance values of the pixels within the masks; and

(f) applying the scaling factors to pixels within the masks to modify the luminance and chrominance values of at least one of the eye and teeth regions of the face.

64. The method as claimed in claim 63 wherein the step (d) of using a valley/peak detector to generate a valley/peak map from the digital image uses a radius of blur determined by an eye separation.

65. A computer storage medium having instructions stored therein for causing a computer to perform the method of claim 63.

66. An automatic retouching method for enhancing an appearance of a face located in a digital image, said method comprising the steps of:

(a) acquiring a digital image containing one or more faces;

(b) detecting a location of facial feature points in the one or more faces, said facial feature points including points identifying salient features including one or more of skin, eyes, eyebrows, nose, mouth, and hair;

(c) using the location of the facial feature points to segment the face into different regions, said different regions including one or more of skin, eyes, eyebrows, nose, mouth, neck and hair regions;

(d) determining one or more facially relevant characteristics of the different regions;

(e) based on the determined facially relevant characteristics of the different regions of the face, selecting a plurality of enhancement filters each customized especially for a particular region, said filters including texture and skin tone filters, and at least one of eye, teeth, and shape filters; and

(f) executing the enhancement filters on the particular regions in a predetermined order wherein the texture enhancement filter is always executed before the skin tone filter, thereby producing an enhanced digital image from the digital image.

67. A computer storage medium having instructions stored therein for causing a computer to perform the method of claim 66.

68. A graphical user interface method for use in an automatic retouching method for enhancing the appearance of a face, said method comprising the steps of:

(a) acquiring and displaying a digital image containing one or more faces which show skin features in an original state;

(b) using one or more enhancement filters to perform an enhancement of a skin feature, thereby producing a displayed digital image showing the skin feature in an enhanced state;

(c) providing a tool for undoing and redoing the enhancement performed on the skin feature; and

(d) using the tool to toggle the skin feature between its enhanced and original state.

69. A computer storage medium having instructions stored therein for causing a computer to perform the method of claim 68.

70. A graphical user interface method for use in a retouching method for enhancing the appearance of a face, said method comprising the steps of:

(a) acquiring and displaying a digital image containing one or more faces;

(b) generating a skin feature map representative of the skin features on the one or more faces;

(c) using one or more enhancement filters to perform an enhancement of a skin feature;

(d) providing a pointing tool for pointing at a skin feature; and

(e) in response to using the pointing tool, referencing the skin feature map and initiating the enhancement of the skin feature being pointed at.

71. A computer storage medium having instructions stored therein for causing a computer to perform the method of claim 71.

72. A graphical user interface method for use in a retouching method for enhancing the appearance of a face, said method comprising the steps of:

(a) acquiring and displaying a digital image containing one or more faces in an original state;

(b) using a plurality of enhancement filters incorporating respective parameters for performing a plurality of enhancements of the digital image, thereby producing a displayed digital image as an enhanced image;

(c) providing a plurality of slider tools for varying the respective parameters of the enhancement filters; and

(d) providing a combination tool for varying the effect of all of the respective parameters at once.

73. The method as claimed in claim 72 wherein the plurality of slider tools have initial slider positions corresponding to default values for the respective parameters.

74. The method as claimed in claim 72 wherein the enhancement filters used in step (b) are applied in a predetermined order.

75. A computer storage medium having instructions stored therein for causing a computer to perform the method of claim 72.

76. A method for enhancing the shape of features appearing in a face located in a digital image, said method comprising the steps of:

(a) acquiring a digital image containing one or more faces;

(b) detecting a location of facial feature points in the one or more faces, said facial feature points including points identifying one or more salient features amenable to shape adjustment;

(c) using the facial feature points to determine two or more source points that are positioned relative to the salient feature;

(d) using the facial feature points to determine two or more destination points which determine the desired change of the salient feature; and

(e) warping the shape of the salient feature by using the source points and the destination points.

77. The method as claimed in claim 76 further including a user interface having a slider adjustment and the method further comprises the step of adjusting the slider to vary the amount of shape enhancement.

78. A computer storage medium having instructions stored therein for causing a computer to perform the method of claim 76.