Automatic face and skin beautification using face detection

• US 8,384,793 B2
• Filed: 07/30/2009
• Issued: 02/26/2013
• Est. Priority Date: 07/30/2008
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: DigitalOptics Corporation Europe Limited (Adeia Inc.)
• Inventors: Ciuc, Mihai, Capata, Adrian, Pososin, Alexei, Florea, Corneliu, Corcoran, Peter, Mocanu, Valentin
• Primary Examiner(s): Tran, Sinh
• Assistant Examiner(s): Nazrul, Shahbaz
• Application Number:
  US12/512,819
• Publication Number:
  US 20100026832A1
• Time in Patent Office: 1,307 Days
• Field of Search: 348/222.1, 382/118, 382/254
• US Class Current: 348/222.1
• CPC Class Codes: G06T 11/40 Filling a planar surface by adding surface attributes, e.g. colour or texture
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 23/10 for generating image signals from different wavelengths
• CPC Class Codes: H04N 23/40 Circuit details for pick-up tubes
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability

Abstract

Sub-regions within a face image are identified to be enhanced by applying a localized smoothing kernel to luminance data corresponding to the sub-regions of the face image. An enhanced face image is generated including an enhanced version of the face that includes certain original pixels in combination with pixels corresponding to the one or more enhanced sub-regions of the face.

Claims

1. A method of enhancing an appearance of a face within a digital image, comprising

using a processor;

generating in-camera, capturing or otherwise obtaining in-camera a collection of one or more relatively low resolution images including a face;

identifying the face within the one or more relatively low resolution images;

segmenting skin tone portions of the face including a forehead, one or two cheeks or a chin, or combinations thereof, from face features including one or two eyes or a mouth or combinations thereof;

within the skin tone portions of the face, identifying one or more blemish regions that vary in luminance at least a threshold amount from the skin tone portions;

acquiring a main image of higher resolution than the one or more relatively low resolution images, including capturing the main image using a lens and an image sensor, or receiving said main image following capture by a device that includes a lens and an image sensor, or a combination thereof;

applying one or more localized luminance smoothing kernels each to one of the skin tone portions identified within the face to produce one or more enhanced skin tone portions of the face, wherein the applying comprises applying the one or more localized luminance smoothing kernels only to luminance data of the one or more skin tone portions identified within said face;

smoothing certain original color data of one or more regions of the main image that correspond to the same one or more blemish regions identified in the relatively low resolution images to generate smoothed color data for those one or more blemish regions of the main image;

generating an enhanced version of the main image including an enhanced version of the face that has the luminance data of the one or more skin tone portions replaced with the enhanced skin tone portions of the face, and that also has the certain original color data of the one or more blemish regions replaced with the smoothed color data; and

displaying, transmitting, communicating or digitally storing or otherwise outputting the enhanced image or a further processed version, or combinations thereof.

2. The method of claim 1, further comprising tracking said face within a collection of said relatively low resolution images.

3. The method of claim 1, wherein the localized luminance smoothing comprises blurring or averaging luminance data, or a combination thereof.

4. The method of claim 1, further comprising applying one or more localized color smoothing kernels to one or more additional sub-regions including one or more sub-regions of the one or more skin tone portions or non-skin tone face features, or a combination thereof, to produce one or more additional enhanced sub-regions, and wherein the one or more additional enhanced sub-regions of the corrected image further comprise pixels modified from original pixels of the face at least by localized color smoothing.

5. The method of claim 1, further comprising applying noise reduction or enhancement, or both, to one or more of the skin tone portions or non-skin tone face features or sub-regions thereof, to produce one or more further enhanced sub-regions and wherein the one or more further enhanced sub-regions of the corrected image further comprise pixels modified from original pixels of the face at least by localized noise reduction or enhancement, or both.

6. The method of claim 1, further comprising determining certain non-skin tone pixels within the one or more skin tone portions that do not comprise a threshold skin tone, and removing, replacing, reducing an intensity of, or modifying a color of said certain non-skin tone pixels, or combinations thereof.

7. The method of claim 1, wherein enhanced pixels of the one or more enhanced skin tone portions comprise enhanced intensities which comprise one or more functions of a relationship between original pixel intensities and local average intensities within the one or more original or enhanced skin tone portions, or combinations thereof.

8. The method of claim 1, further comprising detecting one or more mouth or eye regions, or combinations thereof, within the face, and identifying and enhancing a natural color of one or more sub-regions within the one or more mouth or eye regions, including one or more teeth, lips, tongues, eye whites, eye brows, iris'"'"'s, eye lashes, or pupils, or combinations thereof.

9. The method of claim 1, further comprising classifying the face according to its age based on comparing one or more default image attribute values with one or more determined values;

and adjusting one or more camera acquisition or post-processing parameters, or combinations thereof, based on the classifying of the face according to its age.

18. The device of claim 1, wherein the method further comprises classifying the face according to its age based on comparing one or more default image attribute values with one or more determined values;

and adjusting one or more camera acquisition or post-processing parameters, or combinations thereof, based on the classifying of the face according to its age.

10. A digital image acquisition device, comprising a lens, an image sensor and a processor, and a processor-readable memory having embodied therein processor-readable code for programming the processor to perform a method of enhancing an appearance of a face within a digital image, wherein the method comprises:

generating in-camera, capturing or otherwise obtaining in-camera a collection of one or more relatively low resolution images including a face;

identifying the face within the one or more relatively low resolution images;

segmenting skin tone portions of the face including a forehead, one or two cheeks or a chin, or combinations thereof, from face features including one or two eyes or a mouth or combinations thereof;

within the skin tone portions of the face, identifying one or more blemish regions that vary in luminance at least a threshold amount from the skin tone portions;

acquiring a main image of higher resolution than the one or more relatively low resolution images, including capturing the main image using a lens and an image sensor, or receiving said main image following capture by a device that includes a lens and an image sensor, or a combination thereof;

applying one or more localized luminance smoothing kernels each to one of the skin tone portions identified within the face to produce one or more enhanced skin tone portions of the face, wherein the applying comprises applying the one or more localized luminance smoothing kernels only to luminance data of the one or more skin tone portions identified within said face;

smoothing certain original color data of one or more regions of the main image that correspond to the same one or more blemish regions identified in the relatively low resolution images to generate smoothed color data for those one or more blemish regions of the main image;

generating an enhanced version of the main image including an enhanced version of the face that has the luminance data of the one or more skin tone portions replaced with the enhanced skin tone portions of the face, and that also has the certain original color data of the one or more blemish regions replaced with the smoothed color data; and

displaying, transmitting, communicating or digitally storing or otherwise outputting the enhanced image or a further processed version, or combinations thereof.

11. The device of claim 10, wherein the method further comprises tracking said face within a collection of said relatively low resolution images.

12. The device of claim 10, wherein the localized luminance smoothing comprises blurring or averaging luminance data, or a combination thereof.

13. The device of claim 10, wherein the method further comprises applying one or more localized color smoothing kernels to one or more additional sub-regions including one or more sub-regions of the one or more skin tone portions or non-skin tone face features, or a combination thereof, to produce one or more additional enhanced sub-regions, and wherein the one or more additional enhanced sub-regions of the corrected image further comprise pixels modified from original pixels of the face at least by localized color smoothing.

14. The device of claim 10, wherein the method further comprises applying noise reduction or enhancement, or both, to one or more of the skin tone portions or non-skin tone face features or sub-regions thereof, to produce one or more further enhanced sub-regions and wherein the one or more further enhanced sub-regions of the corrected image further comprise pixels modified from original pixels of the face at least by localized noise reduction or enhancement, or both.

15. The device of claim 10, wherein the method further comprises determining certain non-skin tone pixels within the one or more skin tone portions that do not comprise a threshold skin tone, and removing, replacing, reducing an intensity of, or modifying a color of said certain non-skin tone pixels, or combinations thereof.

16. The device of claim 10, wherein enhanced pixels of the one or more enhanced skin tone portions comprise enhanced intensities which comprise one or more functions of a relationship between original pixel intensities and local average intensities within the one or more original or enhanced skin tone portions, or combinations thereof.

17. The device of claim 10, wherein the method further comprises detecting one or more mouth or eye regions, or combinations thereof, within the face, and identifying and enhancing a natural color of one or more sub-regions within the one or more mouth or eye regions, including one or more teeth, lips, tongues, eye whites, eye brows, iris'"'"'s, eye lashes, or pupils, or combinations thereof.

19. One or more non-transitory processor-readable media having embodied therein code for programming one or more processors to perform a method of enhancing an appearance of a face within a digital image, wherein the method comprises:

generating in-camera, capturing or otherwise obtaining in-camera a collection of one or more relatively low resolution images including a face;

identifying the face within the one or more relatively low resolution images;

segmenting skin tone portions of the face including a forehead, one or two cheeks or a chin, or combinations thereof, from face features including one or two eyes or a mouth or combinations thereof;

within the skin tone portions of the face, identifying one or more blemish regions that vary in luminance at least a threshold amount from the skin tone portions;

acquiring a main image of higher resolution than the one or more relatively low resolution images, including capturing the main image using a lens and an image sensor, or receiving said main image following capture by a device that includes a lens and an image sensor, or a combination thereof;

applying one or more localized luminance smoothing kernels each to one of the skin tone portions identified within the face to produce one or more enhanced skin tone portions of the face, wherein the applying comprises applying the one or more localized luminance smoothing kernels only to luminance data of the one or more skin tone portions identified within said face;

smoothing certain original color data of one or more regions of the main image that correspond to the same one or more blemish regions identified in the relatively low resolution images to generate smoothed color data for those one or more blemish regions of the main image;

generating an enhanced version of the main image including an enhanced version of the face that has the luminance data of the one or more skin tone portions replaced with the enhanced skin tone portions of the face, and that also has the certain original color data of the one or more blemish regions replaced with the smoothed color data; and

displaying, transmitting, communicating or digitally storing or otherwise outputting the enhanced image or a further processed version, or combinations thereof.

20. The one or more processor-readable media of claim 19, wherein the method further comprises tracking said face within a collection of said relatively low resolution images.

21. The one or more processor-readable media of claim 19, wherein the localized luminance smoothing comprises blurring or averaging luminance data, or a combination thereof.

22. The one or more processor-readable media of claim 19, wherein the method further comprises applying one or more localized color smoothing kernels to the one or more additional sub-regions including one or more sub-regions of the one or more skin tone portions or non-skin tone face features, or a combination thereof, to produce one or more additional enhanced sub-regions, and wherein the one or more additional enhanced sub-regions of the corrected image further comprise pixels modified from original pixels of the face at least by localized color smoothing.

23. The one or more processor-readable media of claim 19, wherein the method further comprises applying noise reduction or enhancement, or both, to one or more of the skin tone portions or non-skin tone face features or sub-regions thereof, to produce one or more further enhanced sub-regions and wherein the one or more further enhanced sub-regions of the corrected image further comprise pixels modified from original pixels of the face at least by localized noise reduction or enhancement, or both.

24. The one or more processor-readable media of claim 19, wherein the method further comprises determining certain non-skin tone pixels within the one or more skin tone portions that do not comprise a threshold skin tone, and removing, replacing, reducing an intensity of, or modifying a color of said certain non-skin tone pixels, or combinations thereof.

25. The one or more processor-readable media of claim 19, wherein enhanced pixels of the one or more enhanced skin tone portions comprise enhanced intensities which comprise one or more functions of a relationship between original pixel intensities and local average intensities within the one or more original or enhanced skin tone portions, or combinations thereof.

26. The one or more processor-readable media of claim 19, wherein the method further comprises detecting one or more mouth or eye regions, or combinations thereof, within the face, and identifying and enhancing a natural color of one or more sub-regions within the one or more mouth or eye regions, including one or more teeth, lips, tongues, eye whites, eye brows, iris'"'"'s, eye lashes, or pupils, or combinations thereof.

27. The one or more processor-readable media of claim 19, wherein the method further comprises classifying the face according to its age based on comparing one or more default image attribute values with one or more determined values;

and adjusting one or more camera acquisition or post-processing parameters, or combinations thereof, based on the classifying of the face according to its age.