IMAGE STYLE TRANSFORM METHODS AND APPARATUSES, DEVICES AND STORAGE MEDIA

• US 20200134778A1
• Filed: 12/25/2019
• Published: 04/30/2020
• Est. Priority Date: 08/13/2018
• Status: Active Grant

• Current Assignee: Shenzhen SenseTime Technology Co., Ltd. (SenseTime Technology Co., Ltd.)
• Original Assignee: Shenzhen SenseTime Technology Co., Ltd. (SenseTime Technology Co., Ltd.)
• Inventors: HE, Gaoyuan, LIU, Yicun, CHEN, Xiaohao, REN, Sijie
• Granted Patent: US 11,200,638 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 3/04 Context-preserving transformations, e.g. by using an importance map panospheric to cylindrical image transformations G06T3/12
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

An image style transform method includes: acquiring an initial image to be subjected to style transform; inputting a gradient of the initial image to an image style transform model, and obtaining a feature map of the initial image in a gradient domain from the image style transform model, where the image style transform model is obtained by being trained in the gradient domain based on a pixel-wise loss and a perceptual loss; and performing image reconstruction according to the feature map of the initial image in the gradient domain to obtain a style image.

Claims

1. An image style transform method, comprising:

acquiring an initial image to be subjected to style transform;

inputting a gradient of the initial image to an image style transform model, and obtaining a feature map of the initial image in a gradient domain from the image style transform model, wherein the image style transform model is obtained by being trained in the gradient domain based on a pixel-wise loss and a perceptual loss; and

performing image reconstruction according to the feature map of the initial image in the gradient domain to obtain a style image.

2. The method according to claim 1, further comprising:

training the image style transform model, wherein a training target of the image style transform model is that a total loss L_total is minimum, wherein L_total is represented by the following equation;

L_total=α

L_feat+β

L_pixel, where L_feat represents the perceptual loss, L_pixel represents the pixel-wise loss, and values of α and

β

are real numbers.

3. The method according to claim 1, wherein the image style transform model comprises a pixel-wise loss model and a perceptual loss model, wherein the pixel-wise loss model is obtained by taking minimization of the pixel-wise loss as a training target when being trained in the gradient domain, and the perceptual loss model is obtained by taking minimization of the perceptual loss as a training target when being trained in the gradient domain training.

4. The method according to claim 3, wherein the training process of the pixel-wise loss model and the perceptual loss model comprises:

inputting a gradient of a training sample to the pixel-wise loss model, and obtaining a sample output result of the training sample from the pixel-wise loss model;

determining a gradient of a stylized reference image corresponding to the training sample; and

training the perceptual loss model according to a first output feature map of the gradient of the reference image in a j-th convolutional layer of the perceptual loss model and a second output feature map of the sample output result in the j-th convolutional layer of the perceptual loss model.

5. The method according to claim 4, wherein the training the perceptual loss model according to a first output feature map of the gradient of the reference image in a j-th convolutional layer of the perceptual loss model and a second output feature map of the sample output result in the j-th convolutional layer of the perceptual loss model comprises:

training the perceptual loss model by using the following equation;

6. The method according to claim 3, wherein the training process of the pixel-wise loss model comprises:

using the gradient of the training sample as an input of the pixel-wise loss model, and obtaining a sample output result from the pixel-wise loss model;

determining the gradient of the stylized reference image corresponding to the training sample; and

training the pixel-wise loss model according to the gradient of the reference image and the sample output result.

7. The method according to claim 4, wherein the pixel-wise loss model comprises a first convolutional layer set, an up-sampling layer and a second convolutional layer set, and wherein the training the pixel-wise loss model according to the gradient of the reference image and the sample output result comprises:

inputting the gradient of the training sample to the first convolutional layer set to obtain the sample feature map;

inputting the sample feature map to the up-sampling layer, and up-sampling the sample feature map to a pixel size of the initial image; and

inputting the up-sampled sample feature map to the second convolutional layer set to obtain the sample output result.

8. The method according to claim 7, wherein the training the pixel-wise loss model according to the gradient of the reference image and the sample output result comprises:

training the pixel-wise loss model according to an absolute value of a difference between F_w(∂

I_i) and corresponding ∂

(I_i) of each training sample;

where ∂

I_i represents the gradient of the i-th training sample, F_w represents the pixel-wise loss model, F_w(∂

I_i) represents an output result of the gradient of the i-th training sample through the pixel-wise loss model F_w, and ∂

(I_i) represents the gradient of the stylized reference image of the i-th training sample.

9. The method according to claim 8, wherein the training the pixel-wise loss model according to an absolute value of a difference between F_w(∂

I_i) and corresponding ∂

(I_i) of each training sample comprises;

training the pixel-wise loss model by using the following equation

10. The method according to claim 1, wherein the performing image reconstruction according to the feature map of the initial image in the gradient domain to obtain a style image comprises:

using an image that satisfies a structural similarity condition to the feature map of the initial image in the gradient domain as the style image;

wherein satisfying the structural similarity condition to the feature map of the initial image in the gradient domain satisfies comprises;

a degree of structural difference between the style image and the initial image is less than a similarity threshold, or the degree of structural difference between the style image and the initial image is the smallest, wherein the degree of structural difference is a variation trend of the style image in the gradient domain and the feature map of the initial image in the gradient domain in at least one reference direction.

11. The method according to claim 10, wherein the performing image reconstruction according to the feature map of the initial image in the gradient domain to obtain a style image comprises:

performing image reconstruction according to ∥

∂

_xS−

F_w(∂

_xI)∥

²+∥

∂

_yS−

F_w(∂

_yI)∥

² to obtain the style image, where ∂

_xI represents the gradient of the initial image in an x direction, F_w(∂

_xI) represents a feature map of the gradient of the initial image in the x direction in the gradient domain through the image style transform model, ∂

_yI represents the gradient of the initial image in a y direction, F_w(∂

_yI) represents a feature map of the gradient of the initial image in the y direction in the gradient domain through the image style transform model, ∂

_xS represents the gradient of the style image in the x direction, and ∂

_yS represents the gradient of the style image in they direction.

12. The method according to claim 1, wherein the performing image reconstruction according to the feature map of the initial image in the gradient domain to obtain a style image comprises:

performing image reconstruction according to color information of the initial image and the feature map of the initial image in the gradient domain to obtain the style image.

13. The method according to claim 12, wherein the performing image reconstruction according to color information of the initial image and the feature map of the initial image in the gradient domain to obtain a style image comprises:

using an image that satisfies a structural similarity condition to the feature map of the initial image in the gradient domain and an image that satisfies a color similarity condition to the initial image as the style image.

14. The method according to claim 12, further comprising:

performing feature extraction on the initial image to obtain a face region in the initial image; and

correspondingly, the performing image reconstruction according to color information of the initial image and the feature map of the initial image in the gradient domain to obtain a style image comprises;

using an image that satisfies a structural similarity condition to the feature map of the initial image in the gradient domain, and an image that satisfies a color similarity condition to a face region in the initial image as the style image.

15. The method according to claim 13, wherein the using an image that satisfies a structural similarity condition to the feature map of the initial image in the gradient domain and an image that satisfies a color similarity condition to the initial image as the style image comprises:

performing image reconstruction according to ∥

S−

I∥

+λ

{∥

∂

_xS−

F_w(∂

_xI)∥

²+∥

∂

_yS−

F_w(∂

_yI)∥

²} to obtain the style image, where I represents the initial image, S represents the style image, ∂

_xI represents the gradient of the initial image in the x direction, F_w(∂

_xI) represents a feature map of the gradient of the initial image in the x direction in the gradient domain through the image style transform model, ∂

_yI represents the gradient of the initial image in the y direction, F_w(∂

_yI) represents a feature map of the gradient of the initial image in the y direction in the gradient domain through the image style transform model, ∂

_xS represents the gradient of the style image in the x direction, and ∂

_yS represents the gradient of the style image in the y direction.

16. The method according to claim 1, wherein the inputting a gradient of the initial image to an image style transform model, and obtaining a feature map of the initial image in a gradient domain from the image style transform model comprises:

determining a gradient of the initial image in at least one reference direction;

inputting the gradient in the at least one reference direction to the image style transform model, and correspondingly obtaining a feature map of the initial image in the at least one reference direction in the gradient domain from the image style transform model; and

correspondingly performing image reconstruction according to the feature map in the at least one reference direction in the gradient domain to obtain the style image.

17. The method according to claim 16, wherein the at least one reference direction comprises x and y directions in a plane reference coordinate system, and correspondingly determining the gradients of the initial image in the x and y directions, respectively;

respectively inputting the gradients in the x and y directions to the image style transform model, and correspondingly obtaining a feature map of the initial image in the x and y directions in the gradient domain from the image style transform model; and

correspondingly performing image reconstruction according to the feature map in the x and y directions in the gradient domain to obtain the style image.

18. An image style transform apparatus, comprising:

a memory storing processor-executable instructions; and

a processor arranged to execute the stored processor-executable instructions to perform operations of;

acquiring an initial image to be subjected to style transform;

inputting a gradient of the initial image to an image style transform model, and obtaining a feature map of the initial image in a gradient domain from the image style transform model, wherein the image style transform model is obtained by being trained in the gradient domain based on a pixel-wise loss and a perceptual loss; and

performing image reconstruction according to the feature map of the initial image in the gradient domain to obtain a style image.

19. The apparatus according to claim 18, wherein the processor is arranged to execute the stored processor-executable instructions to further perform an operation of:

training the image style transform model, wherein a training target of the image style transform model is that a total loss L_total is minimum, wherein L_total is represented by the following equation;

L_total=α

L_feat+β

L_pixel, where L_feat represents the perceptual loss, L_pixel represents the pixel-wise loss, and values of α and

β

are real numbers.

20. A non-transitory computer-readable storage medium having stored thereon computer executable instructions that, when executed by a processor, cause the processor to implement an image style transform method, the method comprising:

acquiring an initial image to be subjected to style transform;

inputting a gradient of the initial image to an image style transform model, and obtaining a feature map of the initial image in a gradient domain from the image style transform model, wherein the image style transform model is obtained by being trained in the gradient domain based on a pixel-wise loss and a perceptual loss; and

performing image reconstruction according to the feature map of the initial image in the gradient domain to obtain a style image.