Fast image style transfers

• US 10,740,939 B1
• Filed: 12/06/2017
• Issued: 08/11/2020
• Est. Priority Date: 12/09/2016
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Chung, Jaewook, Dakka, Wisam
• Primary Examiner(s): Shen, Qun
• Application Number:
  US15/833,733
• Time in Patent Office: 979 Days
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning

Abstract

Manipulating images using computationally expensive machine learning schemes can be implemented using server-generated models of the machine learning schemes that are transmitted to a client device for application. The schemes can include convolutional neural networks having a kernel comprising a plurality of low-rank matrices.

Claims

1. A method comprising:

generating, using one or more processors of a client device, image data using a fully connected neural network layer that outputs the image data into a convolutional layer;

generating a stylized image from the image data using a convolutional neural network (CNN), the CNN configured with the convolution layer having rank one kernel matrices generated for a sample set of points of an inferred matrix configured to apply a style transfer of a style of one or more training images, the sample set of points comprising at least one element from each row and each column of the inferred matrix, an output of convolution layer generated at least in part by generating dot products of rank one image matrices from the image data and the rank one kernel matrices, the rank one image matrices from the image data being generated for the sample set of points of the inferred matrix; and

storing the stylized image in memory of the client device.

2. The method of claim 1, wherein the image data is from an image generated using an image sensor on the client device;

and wherein the method further comprises;

responsive to generating the image, automatically generating the stylized image as a background process on the client device.

3. The method of claim 1, wherein the rank one kernel matrices and the rank one image matrices are separable rank one matrices of a kernel matrix trained to apply the style transfer.

4. The method of claim 1, wherein the fully connected neural network layer comprises a REctified Linear Unit (RELU) that generates the image data.

5. The method of claim 1, further comprising:

publishing the stylized image as an ephemeral message on a social network.

6. The method of claim 1, wherein the image data is generated from a photograph, and the style of the one or more training images is a paint brush style.

7. The method of claim 1, wherein the inferred matrix is pre-computed before the image data is generated, and wherein the method further comprises:

storing the inferred matrix in memory of the client device.

8. A system comprising:

one or more processors of a machine; and

a memory comprising instructions that, when executed by the one or more processors, cause the machine to perform operations comprising;

generating image data using a fully connected neural network layer that outputs the image data into a convolutional layer;

generating a stylized image from the image data using a convolutional neural network (CNN), the CNN configured with the convolution layer having rank one kernel matrices generated for a sample set of points of an inferred matrix configured to apply a style transfer of a style of one or more training images, the sample set of points comprising at least one element from each row and each column of the inferred matrix, an output of convolution layer generated at least in part by generating dot products of rank one image matrices from the image data and the rank one kernel matrices, the rank one image matrices from the image data being generated for the sample set of points of the inferred matrix; and

storing the stylized image in the memory.

9. The system of claim 8, wherein the image data is from an image generated using an image sensor;

and wherein the operations further comprise;

responsive to generating the image, automatically generating the stylized image as a background process on the system.

10. The system of claim 8, wherein the rank one kernel matrices and the rank one image matrices are separable rank one matrices of a kernel matrix trained to apply the style transfer.

11. The system of claim 8, wherein the fully connected neural network layer comprises a REctified Linear Unit (RELU) that generates the image data.

12. The system of claim 8, the operations further comprising:

publishing the stylized image as an ephemeral message on a social network.

13. The system of claim 8, wherein the image data is generated from a photograph, and the style of the one or more training images is a paint brush style.

14. The system of claim 8, wherein the inferred matrix is pre-computed before the image data is generated, and wherein the operations further comprise:

storing the inferred matrix in the memory.

15. A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors of a device, cause the device to perform operations comprising:

generating image data using a fully connected neural network layer that outputs the image data into a convolutional layer;

generating a stylized image from the image data using a convolutional neural network (CNN), the CNN configured with the convolution layer having rank one kernel matrices generated for a sample set of points of an inferred matrix configured to apply a style transfer of a style of one or more training images, the sample set of points comprising at least one element from each row and each column of the inferred matrix, an output of convolution layer generated at least in part by generating dot products of rank one image matrices from the image data and the rank one kernel matrices, the rank one image matrices from the image data being generated for the sample set of points of the inferred matrix; and

storing the stylized image in memory.

16. The non-transitory computer-readable storage medium of claim 15, wherein the image data is from an image generated using an image sensor;

and wherein the operations further comprise;

responsive to generating the image, automatically generating the stylized image as a background process.

17. The non-transitory computer-readable storage medium of claim 15, wherein the rank one kernel matrices and the rank one image matrices are separable rank one matrices of a kernel matrix trained to apply the style transfer.

18. The non-transitory computer-readable storage medium of claim 15, wherein the fully connected neural network layer comprises a REctified Linear Unit (RELU) that generates the image data.

19. The non-transitory computer-readable storage medium of claim 15, the operations further comprising:

publishing the stylized image as an ephemeral message on a social network.

20. The non-transitory computer-readable storage medium of claim 15, wherein the inferred matrix is pre-computed before the image data is generated, and wherein the operations further comprise:

storing the inferred matrix in memory.