Facial image processing apparatus, facial image processing method, and non-transitory computer-readable storage medium

• US 10,621,415 B2
• Filed: 04/24/2017
• Issued: 04/14/2020
• Est. Priority Date: 05/19/2016
• Status: Active Grant

• Current Assignee: BOE Technology Group Co., Ltd.
• Original Assignee: BOE Technology Group Co., Ltd.
• Inventors: Li, Zhenglong, Zhang, Lijie, Zhang, Jingyu
• Primary Examiner(s): Cese, Kenny A
• Application Number:
  US15/565,754
• Publication Number:
  US 20180204051A1
• Time in Patent Office: 1,086 Days
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/60 using machine learning, e.g. neural networks
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

The present application discloses a facial image processing apparatus. The facial image processing apparatus includes a memory; and one or more processor. The memory and the at least one processor are communicatively connected with each other. The memory stores computer-executable instructions for controlling the one or more processors to automatically identify a facial feature from a facial image; automatically extract an image portion defining the facial feature from the facial image; perform a detail enhancement process on the image portion to obtain a detail-enhanced image portion corresponding to the image portion; and perform an image composition process to compose the detail-enhanced image portion and the facial image to obtain an enhanced facial image.

Claims

1. A facial image processing apparatus, comprising:

a memory; and

one or more processor;

wherein the memory and the at least one processor are communicatively connected with each other;

the memory stores computer-executable instructions for controlling the one or more processors to;

automatically identify a facial feature from a facial image;

automatically extract an image portion defining the facial feature from the facial image;

perform a detail enhancement process on the image portion to obtain a detail-enhanced image portion corresponding to the image portion; and

perform an image composition process to compose the detail-enhanced image portion and the facial image to obtain an enhanced facial image;

wherein perform the image composition process comprises replace a portion of the facial image corresponding to the image portion with the detail-enhanced image portion to obtain an integrated facial image comprising the detail-enhanced image portion and a portion outside the detail-enhanced image portion integrated together; and

integrate the detail-enhanced image portion and the portion outside the detail-enhanced image portion, thereby obtaining the enhanced facial image;

wherein perform the detail enhancement process on the image portion to obtain the detail-enhanced image portion corresponding to the image portion comprises;

provide a deep neural network, the deep neural network being trained using facial image training data;

input an input signal comprising the image portion or derived from the image portion into an input layer of the deep neural network;

process the input signal through one or more hidden layers of the deep neural network to obtain a processed signal; and

output the processed signal from an output layer of the deep neural network as the detail-enhanced image portion;

wherein the deep neural network is trained using a plurality of pairs of high-resolution image portion and low-resolution image portion;

wherein the plurality of pairs of high-resolution image portion and low-resolution image portion are generated by;

providing a plurality of high-resolution reference image portions as an input; and

based on the plurality of high-resolution reference image portions, generating a plurality of low-resolution reference image portions respectively corresponding to the plurality of high-resolution reference image portions.

2. The facial image processing apparatus of claim 1, wherein the detail enhancement process is performed using a deep convolutional neural network trained on facial image training data.

3. The facial image processing apparatus of claim 1, wherein automatically extract the image portion defining the facial feature from the facial image comprises:

process the facial image using a convolutional neural network to determine an image region in the facial image containing the facial feature;

segment the image region in the facial image to determine an image segment in the image region containing the facial feature, thereby obtaining an image segment corresponding to the facial feature; and

label a plurality of pixels in the image segment thereby extracting the image portion.

4. The facial image processing apparatus of claim 1, wherein generating a plurality of low-resolution reference image portions comprises generating a noise-perturbed version of a low-resolution reference image portion corresponding to one of the plurality of high-resolution reference image portions;

generating a noise-perturbed version of the low-resolution reference image portion comprises adding noise to a luminance component of the low-resolution reference image.

5. The facial image processing apparatus of claim 1, wherein process the input signal through the one or more hidden layers of the deep neural network to obtain the processed signal comprises process the input signal using a loss function.

6. The facial image processing apparatus of claim 1, wherein integrate the detail-enhanced image portion and the portion outside the detail-enhanced image portion comprises:

smoothing filter a transitional region comprising an interface between the detail-enhanced image portion and the portion outside the detail-enhanced image portion using the facial image as a guiding mask.

7. The facial image processing apparatus of claim 1, further comprises an image capturing device configured to capture the facial image, the image capturing device being communicatively connected with the processor.

8. The facial image processing apparatus of claim 1, wherein the facial image comprises a plurality of facial features;

the memory stores computer-executable instructions for controlling the one or more processors to;

automatically identify a plurality of facial features from a facial image;

automatically extract a plurality of image portions respectively defining the plurality of facial features from the facial image;

perform a detail enhancement process on the plurality of image portions to obtain a plurality of detail-enhanced image portions respectively corresponding to the plurality of image portions; and

perform an image composition process to compose the plurality of detail-enhanced image portions and the facial image to obtain the enhanced facial image.

9. A facial image processing method, comprising:

automatically identifying a facial feature from a facial image;

automatically extracting an image portion defining the facial feature from the facial image;

performing a detail enhancement process on the image portion to obtain a detail-enhanced image portion corresponding to the image portion; and

performing an image composition process to compose the detail-enhanced image portion and the facial image to obtain an enhanced facial image;

wherein performing the detail enhancement process comprises replacing a portion of the facial image corresponding to the image portion with the detail-enhanced image portion to obtain an integrated facial image comprising the detail-enhanced image portion and a portion outside the detail-enhanced image portion integrated together; and

; and

integrating the detail-enhanced image portion and the portion outside the detail-enhanced image portion, thereby obtaining the enhanced facial image;

wherein performing the detail enhancement process on the image portion comprises;

providing a deep neural network, the deep neural network being trained using facial image training data;

inputting an input signal comprising the image portion or derived from the image portion into an input layer of the deep neural network;

processing the input signal through one or more hidden layers of the deep neural network to obtain a processed signal; and

outputting the processed signal from an output layer of the deep neural network as the detail-enhanced image portion;

the method further comprises training the deep neural network using facial image training data;

wherein training the deep neural network comprises;

providing a plurality of high-resolution reference image portions as an input;

based on the plurality of high-resolution reference image portions, generating a plurality of low-resolution reference image portions respectively corresponding to the plurality of high-resolution reference image portions, thereby obtaining a plurality of pairs of high-resolution image portion and low-resolution image portion; and

training the deep neural network using the plurality of pairs of high-resolution image portion and low-resolution image portion.

10. The facial image processing method of claim 9, wherein performing the detail enhancement process comprises performing the detail enhancement process using a deep convolutional neural network trained on facial image training data.

11. The facial image processing method of claim 9, wherein automatically extracting the image portion comprises:

processing the facial image using a convolutional neural network to determine an image region in the facial image containing the facial feature;

segmenting the image region in the facial image to determine an image segment in the image region and containing the facial feature, thereby obtaining an image segment corresponding to the facial feature; and

labeling a plurality of pixels in the image segment thereby extracting the image portion.

12. The facial image processing method of claim 9, wherein generating a plurality of low-resolution reference image portions comprises generating a noise-perturbed version of a low-resolution reference image portion corresponding to one of the plurality of high-resolution reference image portions;

generating a noise-perturbed version of the low-resolution reference image portion comprises adding noise to a luminance component of the low-resolution reference image.

13. The facial image processing method of claim 9, wherein processing the input signal through the one or more hidden layers of the deep neural network comprises processing the input signal using a loss function.

14. The facial image processing method of claim 9, wherein integrating the detail-enhanced image portion and the portion outside the detail-enhanced image portion comprises:

smoothing filtering a transitional region comprising an interface between the detail-enhanced image portion and the portion outside the detail-enhanced image portion using the facial image as a guiding mask.

15. A non-transitory computer-readable storage medium storing computer-readable instructions, the computer-readable instructions being executable by a processor to cause the processor to perform:

automatically identifying a facial feature from a facial image;

automatically extracting an image portion defining the facial feature from the facial image;

performing a detail enhancement process on the image portion to obtain a detail-enhanced image portion corresponding to the image portion; and

performing an image composition process to compose the detail-enhanced image portion and the facial image to obtain an enhanced facial image;

wherein performing the detail enhancement process comprises replacing a portion of the facial image corresponding to the image portion with the detail-enhanced image portion to obtain an integrated facial image comprising the detail-enhanced image portion and a portion outside the detail-enhanced image portion integrated together; and

; and

integrating the detail-enhanced image portion and the portion outside the detail-enhanced image portion, thereby obtaining the enhanced facial image;

wherein performing the detail enhancement process on the image portion comprises;

providing a deep neural network, the deep neural network being trained using facial image training data;

inputting an input signal comprising the image portion or derived from the image portion into an input layer of the deep neural network;

processing the input signal through one or more hidden layers of the deep neural network to obtain a processed signal; and

outputting the processed signal from an output layer of the deep neural network as the detail-enhanced image portion;

wherein the deep neural network is trained by;

providing a plurality of high-resolution reference image portions as an input;

based on the plurality of high-resolution reference image portions, generating a plurality of low-resolution reference image portions respectively corresponding to the plurality of high-resolution reference image portions, thereby obtaining a plurality of pairs of high-resolution image portion and low-resolution image portion; and

training the deep neural network using the plurality of pairs of high-resolution image portion and low-resolution image portion.

16. The non-transitory computer-readable storage medium of claim 15, wherein the detail enhancement process is performed using a deep convolutional neural network trained on facial image training data.