Real-time face beautification features for video images

• US 10,152,778 B2
• Filed: 09/11/2015
• Issued: 12/11/2018
• Est. Priority Date: 09/11/2015
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Chen, Ke, Deng, Zhipin, Cai, Xiaoxia, Wang, Chen, Chiu, Yi-Jen, Peng, Ya-Ti, Xu, Lidong
• Primary Examiner(s): Gray, Ryan M
• Application Number:
  US15/127,787
• Publication Number:
  US 20170262970A1
• Time in Patent Office: 1,187 Days
• Field of Search: None
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/174 Facial expression recognition

Abstract

Systems, apparatus, articles, and methods are described below including operations for real-time face beautification features for video images.

Claims

1. A computer-implemented method for video augmentation on a GPU, comprising:

performing, via a facial feature logic unit of the GPU, facial detection on the input video images;

performing, via the facial feature logic unit, facial landmark point detection and/or tracking on the input video images based at least in part on the facial detection; and

modifying, via a beautification module of the GPU, the input video images by facial beautification into output video images based at least in part on the facial landmark point detection;

the modifying comprising;

deriving an affected area, comprising;

deriving for the affected area a transform radius R, a transform center C, and a translation vector CM in an input image P based at least in part on the landmark points as well as on one or more strength settings;

performing backward warping by translation model for each pixel d inside the affected area to get source coordinates s; and

performing bilinear interpolation based at least in part on the backward warping of the affected area to output a processed image P′

.

2. The method of claim 1, further comprising performing, via a cute nose filter portion of the beautification module, the following operations:

wherein deriving the affected area comprises deriving the affected area transform radius R, transform center C, and translation vector CM for an associated nose area in the input image P based at least in part on the landmark points as well as on a cute or bigger nose mode of the one or more strength settings.

3. The method of claim 1, further comprising performing, via a happy/sad face filter portion of the beautification module, the following operations:

wherein deriving the affected area comprises deriving the affected area transform radius R, transform center C, and translation vector CM for the mouth area in the input image P based at least in part on the landmark points associated with a mouth area as well as a happy/sad face mode of the one or more strength settings.

4. The method of claim 1, further comprising performing, via a eye bag/wrinkle removal filter portion of the beautification module, the following operations:

generating an eye wrinkles/bags map S from luma component of the input image P based at least in part on landmark points associated with an eye and/or nose area;

generating a smooth version B of input image P via a recursive bilateral filtering based at least in part on a smooth strength setting; and

performing a content adaptive blending of input image P and smooth version B based at least in part on the generated eye wrinkles/bags map S to output a processed image P′

.

5. The method of claim 1, further comprising performing, via a dark eye circle removal filter portion of the beautification module, the following operations:

generating an eye circle map S from luma component of the input image P based at least in part on landmark points associated with an eye and/or nose area;

generating colored pixels F of input image P based at least in part on a desired white color setting and a strength setting; and

performing a content adaptive blending of input image P and colored pixels F based at least in part on the generated eye circle map S to output a processed image P′

.

6. The method of claim 1, further comprising:

performing, via a profile detection module, facial recognition profile detection based at least in part on the face detection;

modifying, via the profile detection module, strength settings associated with various facial beatification operations based at least in part on the face detection; and

performing, via a skin tone beautification portion of the beautification module, one or more of the following skin tone beautification operations based at least in part on the strength settings;

face brightening, face whitening, skin tone enhancement, skin foundation, and skin smoothing.

7. The method of claim 1, further comprising:

performing, via a profile detection module, facial recognition profile detection based at least in part on the face detection;

modifying, via the profile detection module, strength settings associated with various facial beatification operations based at least in part on the face detection; and

performing one or more of the following skin tone beautification operations based at least in part on the skin tone likelihood score;

skin tone enhancement, skin foundation, and skin smoothing.

8. The method of claim 1, further comprising:

performing, via a profile detection module, facial recognition profile detection based at least in part on the face detection;

modifying, via the profile detection module, strength settings associated with various facial beatification operations based at least in part on the face detection; and

performing, via a landmark beautification portion of the beautification module, one or more of the following landmark beautification operations based at least in part on the strength settings and the landmark points;

red lip alteration, big eye alteration, slim face alteration, cute/big nose alteration, happy/sad face alteration, eye wrinkle removal, eye bags removal, and dark eye circles removal.

9. The method of claim 1, further comprising:

performing, via a cute nose filter portion of the beautification module, the following operations;

wherein deriving the affected area comprises deriving the affected area radius R, transform center C, and translation vector CM for an associated nose area in the input image P based at least in part on the landmark points as well as on a cute or bigger nose mode of the one or more strength settings;

performing, via a happy/sad face filter portion of the beautification module, the following operations;

wherein deriving the affected area comprises deriving the affected area transform radius R, transform center C, and translation vector CM for the mouth area in the input image P based at least in part on the landmark points associated with a mouth area as well as a happy/sad face mode of the one or more strength settings;

performing, via a eye bag/wrinkle removal filter portion of the beautification module, the following operations;

generating an eye wrinkles/bags map S from luma component of the input image P based at least in part on landmark points associated with an eye and/or nose area;

generating a smooth version B of input image P via a recursive bilateral filtering based at least in part on a smooth strength setting;

performing a content adaptive blending of input image P and smooth version B based at least in part on the generated eye wrinkles/bags map S to output a processed image P′

;

performing, via a dark eye circle removal filter portion of the beautification module, the following operations;

generating an eye circle map S from luma component of the input image P based at least in part on landmark points associated with an eye and/or nose area;

generating colored pixels F of input image P based at least in part on a desired white color setting and a strength setting; and

performing a content adaptive blending of input image P and colored pixels F based at least in part on the generated eye circle map S to output a processed image P′

.

10. The method of claim 1, further comprising:

receiving input video images via a skin tone logic unit, the facial feature logic unit, and the beautification module;

determining, via the skin tone logic unit, a skin tone likelihood score on input video images;

smoothing, via the skin tone logic unit, the skin tone likelihood score;

performing, via a profile detection module, facial recognition profile detection based at least in part on the face detection;

modifying, via the profile detection module, strength settings associated with various facial beatification operations based at least in part on the face detection, wherein the strength settings are based at least in part on one or more of the following detected facial attributes;

gender, age, and race;

performing, via a skin tone beautification portion of the beautification module, one or more of the following skin tone beautification operations based at least in part on the strength settings;

face brightening, face whitening, skin tone enhancement, skin foundation, and skin smoothing, and performing one or more of the following skin tone beautification operations based at least in part on the skin tone likelihood score;

skin tone enhancement, skin foundation, and skin smoothing;

performing, via a landmark beautification portion of the beautification module, one or more of the following landmark beautification operations based at least in part on the strength settings and the landmark points as well as the performed skin tone beautification operations;

red lip alteration, big eye alteration, and slim face alteration;

performing, via a cute nose filter portion of the beautification module, the following operations;

wherein deriving the affected area comprises deriving the affected area radius R, transform center C, and translation vector CM for an associated nose area in the input image P based at least in part on the landmark points as well as on a cute or bigger nose mode of the one or more strength settings;

performing, via a happy/sad face filter portion of the beautification module, the following operations;

wherein deriving the affected area comprises deriving the affected area transform radius R, transform center C, and translation vector CM for the mouth area in the input image P based at least in part on the landmark points associated with a mouth area as well as a happy/sad face mode of the one or more strength settings;

performing, via a eye bag/wrinkle removal filter portion of the beautification module, the following operations;

generating an eye wrinkles/bags map S from luma component of the input image P based at least in part on landmark points associated with an eye and/or nose area;

generating a smooth version B of input image P via a recursive bilateral filtering based at least in part on a smooth strength setting;

performing a content adaptive blending of input image P and smooth version B based at least in part on the generated eye wrinkles/bags map S to output a processed image P′

;

performing, via a dark eye circle removal filter portion of the beautification module, the following operations;

generating an eye circle map S from luma component of the input image P based at least in part on landmark points associated with an eye and/or nose area;

generating colored pixels F of input image P based at least in part on a desired white color setting and a strength setting; and

performing a content adaptive blending of input image P and colored pixels F based at least in part on the generated eye circle map S to output a processed image P′

; and

displaying, via a display, output video images as modified by the beautification module.

11. A mobile computing device for video augmentation, comprising:

one or more graphics processing units, the one or more graphics processing units including a facial feature logic unit and a beautification module;

one or more memory stores communicatively coupled to the one or more graphics processing units;

wherein the one or more graphics processing units are configured to;

perform, via the facial feature logic unit, facial detection on the input video images;

perform, via the facial feature logic unit, facial landmark point detection and/or tracking on the input video images based at least in part on the facial detection; and

modify, via the beautification module, the input video images by facial beautification into output video images based at least in part on the facial landmark point detection;

the modification comprising the following operations;

derive an affected area, comprising;

deriving for the affected area a transform radius R, a transform center C, and a translation vector CM in an input image P based at least in part on the landmark points as well as on one or more strength settings;

perform backward warping by translation model for each pixel d inside the affected area to get source coordinates s; and

perform bilinear interpolation based at least in part on the backward warping of the affected area to output a processed image P′

.

12. The mobile computing device of claim 11, wherein the one or more graphics processing units are further configured to perform, via a cute nose filter portion of the beautification module, the following operations:

wherein the operation to derive the affected area comprises an operation to derive the affected area transform radius R, transform center C, and translation vector CM for an associated nose area in the input image P based at least in part on the landmark points as well as on a cute or bigger nose mode of the one or more strength settings.

13. The mobile computing device of claim 11, wherein the one or more graphics processing units are further configured to perform, via a happy/sad face filter portion of the beautification module, the following operations:

wherein the operation to derive the affected area comprises an operation to derive the affected area transform radius R, transform center C, and translation vector CM for the mouth area in the input image P based at least in part on the landmark points associated with a mouth area as well as a happy/sad face mode of the one or more strength settings.

14. The mobile computing device of claim 11, wherein the one or more graphics processing units are further configured to perform, via a eye bag/wrinkle removal filter portion of the beautification module, the following operations:

generate an eye wrinkles/bags map S from luma component of the input image P based at least in part on landmark points associated with an eye and/or nose area;

generate a smooth version B of input image P via a recursive bilateral filtering based at least in part on a smooth strength setting; and

perform a content adaptive blending of input image P and smooth version B based at least in part on the generated eye wrinkles/bags map S to output a processed image P′

.

15. The mobile computing device of claim 11, wherein the one or more graphics processing units are further configured to perform, via a dark eye circle removal filter portion of the beautification module, the following operations:

generate an eye circle map S from luma component of the input image P based at least in part on landmark points associated with an eye and/or nose area;

generate colored pixels F of input image P based at least in part on a desired white color setting and a strength setting; and

perform a content adaptive blending of input image P and colored pixels F based at least in part on the generated eye circle map S to output a processed image P′

.

16. The mobile computing device of claim 11, wherein the one or more graphics processing units are further configured to:

perform, via a profile detection module, facial recognition profile detection based at least in part on the face detection;

modify, via the profile detection module, strength settings associated with various facial beatification operations based at least in part on the face detection; and

perform, via a skin tone beautification portion of the beautification module, one or more of the following skin tone beautification operations based at least in part on the strength settings;

face brightening, face whitening, skin tone enhancement, skin foundation, and skin smoothing.

17. The mobile computing device of claim 11, wherein the one or more graphics processing units are further configured to:

perform, via a profile detection module, facial recognition profile detection based at least in part on the face detection;

modify, via the profile detection module, strength settings associated with various facial beatification operations based at least in part on the face detection; and

perform one or more of the following skin tone beautification operations based at least in part on the skin tone likelihood score;

skin tone enhancement, skin foundation, and skin smoothing.

18. The mobile computing device of claim 11, wherein the one or more graphics processing units are further configured to:

perform, via a profile detection module, facial recognition profile detection based at least in part on the face detection;

modify, via the profile detection module, strength settings associated with various facial beatification operations based at least in part on the face detection; and

perform, via a landmark beautification portion of the beautification module, one or more of the following landmark beautification operations based at least in part on the strength settings and the landmark points;

red lip alteration, big eye alteration, slim face alteration, cute/big nose alteration, happy/sad face alteration, eye wrinkle removal, eye bags removal, and dark eye circles removal.

19. The mobile computing device of claim 11, wherein the one or more graphics processing units are further configured to:

receive input video images via a skin tone logic unit, the facial feature logic unit, and the beautification module;

determine, via the skin tone logic unit, a skin tone likelihood score on input video images;

smooth, via the skin tone logic unit, the skin tone likelihood score;

perform, via a profile detection module, facial recognition profile detection based at least in part on the face detection;

modify, via the profile detection module, strength settings associated with various facial beatification operations based at least in part on the face detection, wherein the strength settings are based at least in part on one or more of the following detected facial attributes;

gender, age, and race;

perform, via a skin tone beautification portion of the beautification module, one or more of the following skin tone beautification operations based at least in part on the strength settings;

face brightening, face whitening, skin tone enhancement, skin foundation, and skin smoothing, and performing one or more of the following skin tone beautification operations based at least in part on the skin tone likelihood score;

skin tone enhancement, skin foundation, and skin smoothing;

perform, via a landmark beautification portion of the beautification module, one or more of the following landmark beautification operations based at least in part on the strength settings and the landmark points as well as the performed skin tone beautification operations;

red lip alteration, big eye alteration, and slim face alteration;

perform, via a cute nose filter portion of the beautification module, the following operations;

wherein the operation to derive the affected area comprises an operation to derive the affected area transform radius R, transform center C, and translation vector CM for an associated nose area in the input image P based at least in part on the landmark points as well as on a cute or bigger nose mode of the one or more strength settings;

perform, via a happy/sad face filter portion of the beautification module, the following operations;

wherein the operation to derive the affected area comprises an operation to derive the affected area transform radius R, transform center C, and translation vector CM for the mouth area in the input image P based at least in part on the landmark points associated with a mouth area as well as a happy/sad face mode of the one or more strength settings;

perform, via a eye bag/wrinkle removal filter portion of the beautification module, the following operations;

generate an eye wrinkles/bags map S from luma component of the input image P based at least in part on landmark points associated with an eye and/or nose area;

generate a smooth version B of input image P via a recursive bilateral filtering based at least in part on a smooth strength setting;

perform a content adaptive blending of input image P and smooth version B based at least in part on the generated eye wrinkles/bags map S to output a processed image P′

;

perform, via a dark eye circle removal filter portion of the beautification module, the following operations;

generate an eye circle map S from a luma component of the input image P based at least in part on landmark points associated with an eye and/or nose area;

generate colored pixels F of input image P based at least in part on a desired white color setting and a strength setting; and

perform a content adaptive blending of input image P and colored pixels F based at least in part on the generated eye circle map S to output a processed image P′

; and

the mobile computing device further comprising a display, the display configured to display output video images as modified by the beautification module.

20. An apparatus for video augmentation, comprising:

a graphics processing unit (GPU), the graphics processing unit configured to;

perform, via a facial feature logic unit, facial detection on the input video images;

perform, via the facial feature logic unit, facial landmark point detection and/or tracking on the input video images based at least in part on the facial detection; and

modify, via a beautification module, the input video images by facial beautification into output video images based at least in part on the facial landmark point detection;

wherein the modification comprises operations to;

derive an affected area, comprising operations to;

derive for the affected area a transform radius R, a transform center C, and a translation vector CM in an input image P based at least in part on the landmark points as well as on one or more strength settings;

perform backward warping by translation model for each pixel d inside the affected area to get source coordinates s; and

perform bilinear interpolation based at least in part on the backward warping of the affected area to output a processed image P′

.

21. The apparatus of claim 20, wherein the graphics processing unit is further configured to:

receive input video images via a skin tone logic unit, the facial feature logic unit, and the beautification module;

determine, via the skin tone logic unit, a skin tone likelihood score on input video images;

smooth, via the skin tone logic unit, the skin tone likelihood score;

perform, via a profile detection module, facial recognition profile detection based at least in part on the face detection;

modify, via the profile detection module, strength settings associated with various facial beatification operations based at least in part on the face detection, wherein the strength settings are based at least in part on one or more of the following detected facial attributes;

gender, age, and race;

perform, via a skin tone beautification portion of the beautification module, one or more of the following skin tone beautification operations based at least in part on the strength settings;

face brightening, face whitening, skin tone enhancement, skin foundation, and skin smoothing, and performing one or more of the following skin tone beautification operations based at least in part on the skin tone likelihood score;

skin tone enhancement, skin foundation, and skin smoothing;

perform, via a landmark beautification portion of the beautification module, one or more of the following landmark beautification operations based at least in part on the strength settings and the landmark points as well as the performed skin tone beautification operations;

red lip alteration, big eye alteration, and slim face alteration;

perform, via a cute nose filter portion of the beautification module, the following operations;

wherein the operation to derive the affected area comprises an operation to derive the affected area transform radius R, transform center C, and translation vector CM for an associated nose area in the input image P based at least in part on the landmark points as well as on a cute or bigger nose mode of the one or more strength settings;

perform, via a happy/sad face filter portion of the beautification module, the following operations;

wherein the operation to derive the affected area comprises an operation to derive the affected area transform radius R, transform center C, and translation vector CM for the mouth area in the input image P based at least in part on the landmark points associated with a mouth area as well as a happy/sad face mode of the one or more strength settings;

perform, via a eye bag/wrinkle removal filter portion of the beautification module, the following operations;

generate an eye wrinkles/bags map S from luma component of the input image P based at least in part on landmark points associated with an eye and/or nose area;

generate a smooth version B of input image P via a recursive bilateral filtering based at least in part on a smooth strength setting;

perform a content adaptive blending of input image P and smooth version B based at least in part on the generated eye wrinkles/bags map S to output a processed image P′

;

perform, via a dark eye circle removal filter portion of the beautification module, the following operations;

generate an eye circle map S from a luma component of the input image P based at least in part on landmark points associated with an eye and/or nose area;

generate colored pixels F of input image P based at least in part on a desired white color setting and a strength setting; and

perform a content adaptive blending of input image P and colored pixels F based at least in part on the generated eye circle map S to output a processed image P; and

cause to be displayed, via a display, output video images as modified by the beautification module.