Facial feature adding method, facial feature adding apparatus, and facial feature adding device

• US 10,580,182 B2
• Filed: 11/15/2017
• Issued: 03/03/2020
• Est. Priority Date: 11/16/2016
• Status: Active Grant

• Current Assignee: Beijing Kuangshi Technology Co., Ltd., Megvii (Beijing) Technology Co., Ltd.
• Original Assignee: Beijing Kuangshi Technology Co., Ltd.
• Inventors: Liu, Yu, Zhou, Erjin
• Primary Examiner(s): Wu, Jingge
• Application Number:
  US15/814,076
• Publication Number:
  US 20180137665A1
• Time in Patent Office: 839 Days
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 3/20 Linear translation of whole images or parts thereof, e.g. panning
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/169 Holistic features and representations, i.e. based on the facial image taken as a whole

Abstract

Provided is a facial feature adding method, a facial feature adding apparatus, and a facial feature adding device. The facial feature adding method comprises: generating an image to be superimposed based on a given facial image and a feature to be added on the given facial image; and superimposing the image to be superimposed and the given facial image to generate a synthesized facial image. In addition, the facial feature adding method further comprises: generating a first face satisfaction score and a second face satisfaction score by use of a deep convolutional network for face determination and based on the synthesized facial image and a real image with the feature to be added; calculating an L1 norm of the image to be superimposed; and updating parameters of networks based on the first face satisfaction score, the second face satisfaction score, and the L1 norm.

Claims

1. A facial feature adding method, comprising:

generating an image to be superimposed based on a given facial image and a feature to be added on the given facial image; and

superimposing the image to be superimposed and the given facial image to generate a synthesized facial image,

wherein generating an image to be superimposed comprises;

cutting out a partial image associated with the feature to be added from the given facial image based on the feature to be added on the given facial image;

extracting facial feature images by use of a facial feature image extraction network and based on the partial image that has been cut out; and

generating the image to be superimposed by use of the synthesized feature image generation network and based on the facial feature images and a demanded feature image corresponding to the feature to be added,

wherein a channel number of the partial image is the same as a channel number of the given facial image;

a size of the facial feature images is smaller than a size of the partial image, and a channel number of the facial feature images is larger than a channel number of the partial image; and

a channel number of the image to be superimposed is the same as a channel number of the given facial image.

2. The facial feature adding method according to claim 1, wherein extracting facial feature images based on the partial image that has been cut out comprises:

performing linear transformation on the partial image that has been cut out to obtain an intermediate image with a first predetermined size, a channel number of the intermediate image being the same as a channel number of the partial image;

extracting facial feature images with a second predetermined size by use of the facial feature image extraction network and based on the intermediate image with the first predetermined size, the second predetermined size being smaller than the first predetermined size, and a channel number of the facial feature images being larger than a channel number of the intermediate image.

3. The facial feature adding method according to claim 2, wherein the facial feature image extraction network comprises P layers of convolutional neural network in concatenation, P being an integer larger than or equal to two,

a first layer of convolutional neural network is used to receive the intermediate image with the first predetermined size, a P-th layer of convolutional neural network is used to output the facial feature images with the second predetermined size;

a size of output images of a p-th layer of convolutional neural network is smaller than a size of input images thereof, and a channel number of the output images of the p-th layer of convolutional neural network is larger than a channel number of the input images thereof, p being an integer larger than or equal to one and smaller than or equal to P.

4. The facial feature adding method according to claim 2, wherein generating the image to be superimposed based on the facial feature images and the demanded feature image comprises:

generating a synthesized feature image with a third predetermined size by use of the synthesized feature image generation network and based on the facial feature images and the demanded feature image, the third predetermined size being larger than the second predetermined size, and the third predetermined size being the same as or different than the first predetermined size;

performing inverse linear transformation corresponding to the linear transformation that has been performed after the cutting on the synthesized feature image with the third predetermined size to generate a partial image to be superimposed,

a channel number of the partial image to be superimposed is the same as a channel number of the synthesized feature image, and a size of the partial image to be superimposed is the same as a size of the partial image that has been cut out, and any channel of the partial image to be superimposed uniquely corresponds to one channel of the given facial image.

5. The facial feature adding method according to claim 4, wherein the synthesized feature image generation network comprises at least one layer of fully connected neural network and K layers of integrated convolutional neural network in concatenation, K being an integer larger than or equal to two,

the at least one layer of fully connected neural network is used to receive the facial feature images and the demanded feature image, a first layer of integrated convolutional neural network is used to receive output images of the at least one layer of fully connected neural network, a K-th layer of integrated convolutional neural network outputs the synthesized feature image with the third predetermined size,

a size of output images of a k-th layer of integrated convolutional neural network is larger than a size of input images thereof, and a channel number of the output images of the k-th layer of integrated convolutional neural network is smaller than a channel number of the input images thereof, k being an integer larger than or equal to one and smaller than or equal to K.

6. The facial feature adding method according to claim 4, wherein the partial image to be superimposed serves as the image to be superimposed,

superimposing the image to be superimposed and the given facial image to generate a synthesized facial image comprises:

corresponding to the cutting performed on the given facial image, superimposing, pixel by pixel, corresponding channels of the image to be superimposed and the given facial image at a cutting position, or weighted-superimposing, pixel by pixel, corresponding channels of the image to be superimposed and the given facial image at a cutting position, so as to generate the synthesized facial image, a channel number of the synthesized facial image being the same as a channel number of the given facial image.

7. The facial feature adding method according to claim 4, wherein generating the image to be superimposed based on the facial feature images and the demanded feature image further comprises:

corresponding to the cutting performed on the given facial image, performing image padding on the partial image to be superimposed so as to generate the image to be superimposed,

a size of the image to be superimposed is the same as a size of the given facial image, a channel number of the image to be superimposed is the same as a channel number of the given facial image, and any channel of the image to be superimposed uniquely corresponds to one channel of the given facial image;

superimposing the image to be superimposed and the given facial image to generate a synthesized facial image comprises;

superimposing, pixel by pixel, corresponding channels of the image to be superimposed and the given facial image, or weighted-superimposing, pixel by pixel, corresponding channels of the image to be superimposed and the given facial image, so as to generate the synthesized facial image, a channel number of the synthesized facial image being the same as a channel number of the given facial image.

8. The facial feature adding method according to claim 5, wherein the k-th layer of integrated convolutional neural network comprises:

an amplification network and J layers of convolutional neural network, J being an integer larger than or equal to two;

in the case where k is equal to one, an amplification network of a first layer of integrated convolutional neural network is used to receive output images of the fully connected neural network;

in the case where k is larger than one, an amplification network of a k-th layer of integrated convolutional neural network is used to receive output images of a (k−

1)-th layer of integrated convolutional neural network;

a first layer of convolutional neural network of the k-th layer of integrated convolutional neural network receives output images of the amplification network, and a J-th layer of convolutional neural network of the k-th layer of integrated convolutional neural network is used to output the output images of the k-th layer of integrated convolutional neural network;

a size of the output images of a j-th layer of convolutional neural network is the same as a size of the input images thereof, and a channel number of the output images of the j-th layer of convolutional neural network is smaller than a channel number of the input images thereof, j being an integer larger than or equal to one and smaller than or equal to J.

9. The facial feature adding method according to claim 1, further comprising:

generating a first face satisfaction score by use of a deep convolutional network for face determination and based on the synthesized facial image;

calculating an L1 norm of the image to be superimposed; and

updating parameters of the facial feature image extraction network and the synthesized feature image generation network based on the first face satisfaction score and the L1 norm of the image to be superimposed.

10. The facial feature adding method according to claim 9, further comprising:

generating a second face satisfaction score by use of a deep convolutional network for face determination and based on a real image with the feature to be added; and

updating parameters of the deep convolutional network for face determination based on the first face satisfaction score and the second face satisfaction score.

11. The facial feature adding method according to claim 9, wherein the deep convolutional network for face determination comprises L layers of convolutional neural network and at least one layer of fully connected neural network;

a first layer of convolutional neural network is used to receive an input image, the at least one layer of fully connected neural network is used to receive output images of an L-th layer of convolutional neural network and output the face satisfaction score.

12. A facial feature adding apparatus, comprising:

a to-be-superimposed image generating module configured to generate an image to be superimposed based on a given facial image and a feature to be added on the given facial image; and

a synthesized facial image generating module configured to superimpose the image to be superimposed and the given facial image to generate a synthesized facial image,

wherein the to-be-superimposed image generating module comprises;

an image cutting module configured to cut out a partial image associated with the feature to be added from the given facial image based on the feature to be added on the given facial image;

a feature extracting module configured to extract facial feature images by use of a facial feature image extraction network and based on the partial image that has been cut out; and

a to-be-superimposed image synthesizing module configured to generate the image to be superimposed by use of the synthesized feature image generation network and based on the facial feature images and a demanded feature image corresponding to the feature to be added,

wherein a channel number of the partial image is the same as a channel number of the given facial image;

a size of the facial feature images is smaller than a size of the partial image, and a channel number of the facial feature images is larger than a channel number of the partial image; and

a channel number of the image to be superimposed is the same as a channel number of the given facial image.

13. The facial feature adding apparatus according to claim 12, further comprising:

a face determining module configured to generate a first face satisfaction score by use of a deep convolutional network for face determination and based on the synthesized facial image;

a norm calculating module configured to calculate an L1 norm of the image to be superimposed; and

a first parameter adjusting module configured to update parameters of the facial feature image extraction network and the synthesized feature image generation network based on the first face satisfaction score and the L1 norm of the image to be superimposed.

14. The facial feature adding apparatus according to claim 13, wherein the face determining module is further configured to generate a second face satisfaction score based on a real image with the feature to be added and by use of a deep convolutional network for face determination;

and

the facial feature adding apparatus further comprises;

a second parameter adjusting module configured to update parameters of the deep convolutional network for face determination based on the first face satisfaction score and the second face satisfaction score.

15. A facial feature adding device, comprising:

one or more processors;

one or more memories in which program instructions are stored, the program instructions being executed by the one or more processors to execute the steps of

generating an image to be superimposed based on a given facial image and a feature to be added on the given facial image; and

superimposing the image to be superimposed and the given facial image to generate a synthesized facial image,

wherein generating an image to be superimposed comprises;

cutting out a partial image associated with the feature to be added from the given facial image based on the feature to be added on the given facial image;

extracting facial feature images by use of a facial feature image extraction network and based on the partial image that has been cut out; and

generating the image to be superimposed by use of the synthesized feature image generation network and based on the facial feature images and a demanded feature image corresponding to the feature to be added,

wherein a channel number of the partial image is the same as a channel number of the given facial image;

a size of the facial feature images is smaller than a size of the partial image, and a channel number of the facial feature images is larger than a channel number of the partial image; and

a channel number of the image to be superimposed is the same as a channel number of the given facial image.

16. The facial feature adding device according to claim 15, wherein the one or more processors execute the program instructions further for:

generating a first face satisfaction score by use of a deep convolutional network for face determination and based on the synthesized facial image;

calculating an L1 norm of the image to be superimposed; and

updating parameters of the facial feature image extraction network and the synthesized feature image generation network based on the first face satisfaction score and the L1 norm of the image to be superimposed.

17. The facial feature adding device according to claim 15, wherein the one or more processors execute the program instructions further for:

generating a second face satisfaction score based on a real image with the feature to be added and by use of a deep convolutional network for face determination; and

updating parameters of the deep convolutional network for face determination based on the first face satisfaction score and the second face satisfaction score.