Multistage neural network processing using a graphics processor

• US 10,672,096 B1
• Filed: 10/21/2019
• Issued: 06/02/2020
• Est. Priority Date: 02/12/2018
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Chen, Yanping, Chung, Jaewook, Dakka, Wisam
• Primary Examiner(s): Wilson, Nicholas R
• Application Number:
  US16/659,561
• Time in Patent Office: 225 Days
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 15/005 General purpose rendering architectures

Abstract

A multistage neural network system can store neural networks as shader programs on GPU memory. Neural network weights can be stored as shader objects or textures on the GPU memory. The GPU can receive a number of neural network image processing tasks to perform on images captured by a client device. The GPU can execute the tasks per driver parameters and display results in real time or near real time on the client device.

Claims

1. A method comprising:

transmitting, by a central processing unit (CPU) of a mobile device, first neural network weights and second neural network weights to a graphics processing unit (GPU) for storage in memory of the GPU;

transmitting, by the CPU of the mobile device, a first set of shaders and a second set of shaders to the GPU for storage in the GPU memory, the first set of shaders configured to apply a first neural network using the first neural network weights, the second set of shaders configured to apply a second neural network using the second neural network weights;

generating one or more input images using an image sensor of the mobile device;

transmitting, by the CPU of the mobile device, instructions to the GPU to apply the first set of shaders and the second set of shaders to the one or more input images; and

displaying, on a display device of the mobile device, one or more modified images generated by the GPU by applying the first set of shaders and the second set of shaders to the one or more input images.

2. The method of claim 1, wherein the instructions recite a first instruction to apply the first set of shaders before a second instruction to apply the second set of shaders.

3. The method of claim 1, wherein a GPU driver of the GPU reorders the instructions such that the second set of shaders is applied before the first set of shaders.

4. The method of claim 3, wherein the GPU driver reordering the instructions based on a native configuration of the GPU driver.

5. The method of claim 4, wherein the native configuration of the GPU includes a quantity of shader cores in the GPU, wherein the GPU driver reorders the instructions based on an available quantity of the shaders cores in the GPU.

6. The method of claim 3, wherein the GPU driver reorders the instructions based at least in part on the instructions not including a sync command.

7. The method of claim 1, wherein CPU transmits the instructions to the GPU using an application programming interface (API).

8. The method of claim 1, wherein the CPU is an integrated CPU of the mobile device, the GPU is an integrated GPU of the mobile device, and the display device is an integrated display device of the mobile device.

9. The method of claim 1, wherein the first neural network weights and the second neural network weights are stored in the GPU as one or more of:

texture data, a texture buffer object, or shader storage buffer objects.

10. The method of claim 1, wherein the first neural network weights are generated by training a first neural network on training data, wherein the second neural network weights are generated by training a second neural network on training data, the training occurring not on the mobile device.

11. The method of claim 1, further comprising receiving, from a server, the first neural network weights and the second neural network weights.

12. The method of claim 1, further comprising:

publishing the one or more modified images as an ephemeral message on a social network site.

13. The method of claim 1, wherein the first neural network is a convolutional neural network configured to perform at least one of:

image segmentation, object detection, or style transfer.

14. The method of claim 1, wherein the second neural network is a convolutional neural network configured to perform at least one of:

image segmentation, object detection, or style transfer.

15. A mobile device comprising:

a central processing unit (CPU);

a graphics processing unit (GPU);

a memory storing instructions that, when executed by the CPU cause the mobile device to perform operations comprising;

transmitting, by the central processing unit (CPU) of the mobile device, first neural network weights and second neural network weights to the graphics processing unit (GPU) for storage in memory of the GPU;

transmitting, by the CPU of the mobile device, a first set of shaders and a second set of shaders to the GPU for storage in the GPU memory, the first set of shaders configured to apply a first neural network using the first neural network weights, the second set of shaders configured to apply a second neural network using the second neural network weights;

generating one or more input images using an image sensor of the mobile device;

transmitting, by the CPU of the mobile device, instructions to the GPU to apply the first set of shaders and the second set of shaders to the one or more input images; and

displaying, on a display device of the mobile device, one or more modified images generated by the GPU by applying the first set of shaders and the second set of shaders to the one or more input images.

16. The mobile device of claim 15, wherein a GPU driver of the GPU reorders the instructions such that the second set of shaders is applied before the first set of shaders.

17. The mobile device of claim 16, wherein the GPU driver reordering the instructions based on a native configuration of the GPU driver.

18. The mobile device of claim 17, wherein the native configuration of the GPU includes a quantity of shader cores in the GPU, wherein the GPU driver reorders the instructions based on an available quantity of the shaders cores in the GPU.

19. The mobile device of claim 15, wherein CPU transmits the instructions to the GPU using an application programming interface (API).

20. A machine-readable storage medium embodying instructions that, when executed by a machine, cause a central processing unit (CPU) of a mobile device to perform operations comprising:

transmitting first neural network weights and second neural network weights to a graphics processing unit (GPU) for storage in memory of the GPU;

transmitting a first set of shaders and a second set of shaders to the GPU for storage in the GPU memory, the first set of shaders configured to apply a first neural network using the first neural network weights, the second set of shaders configured to apply a second neural network using the second neural network weights;

generating one or more input images using an image sensor of the mobile device;

transmitting instructions to the GPU to apply the first set of shaders and the second set of shaders to the one or more input images; and

displaying one or more modified images generated by the GPU by applying the first set of shaders and the second set of shaders to the one or more input images.