Training convolutional neural networks on graphics processing units

• US 20070047802A1
• Filed: 08/31/2005
• Published: 03/01/2007
• Est. Priority Date: 08/31/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Puri, Siddhartha
• Granted Patent: US 7,747,070 B2
• Time in Patent Office: Days
• US Class Current: 382/157
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/06 Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/18057 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]

Abstract

A convolutional neural network is implemented on a graphics processing unit. The network is then trained through a series of forward and backward passes, with convolutional kernels and bias matrices modified on each backward pass according to a gradient of an error function. The implementation takes advantage of parallel processing capabilities of pixel shader units on a GPU, and utilizes a set of start-to-finish formulas to program the computations on the pixel shaders. Input and output to the program is done through textures, and a multi-pass summation process is used when sums are needed across pixel shader unit registers.

Claims

1. A computer-implemented method for training a convolutional neural network to identify images using graphics data which can be read by a graphics processing unit (“

GPU”

), and one or more GPU-executable programs, the graphics data representing the state of the convolutional neural network, the method comprising;

executing one or more of the GPU-executable programs on the GPU in order to perform a forward pass in the convolutional neural network;

executing one or more of the GPU-executable programs on the GPU in order to perform a backward pass in the convolutional neural network;

executing one or more of the GPU-executable programs on the GPU in order to modify the graphics data based on the results of the backward pass; and

repeating executing one or more of the GPU-executable programs to perform forward passes, backward passes, and to modify the graphics data until the convolutional neural network is trained.

2. The method of claim 1, wherein the graphics data comprises one or more textures representing one or more neural network variables.

3. The method of claim 2, wherein:

the one or more textures comprises a two-dimensional texture; and

at least one or more of the textures represents a neural network variable with more than two dimensions which has been flattened into two dimensions.

4. The method of claim 2, wherein addresses of values represented in a texture are determined through linear combinations of x and y offset coordinates from the upper-left corner of an output texture.

5. The method of claim 1, wherein:

the GPU-executable programs are written in the form of one or more pixel shader programs.

6. The method of claim 1, wherein the graphics data is adjusted using gradient descent.

7. The method of claim 6, wherein:

the one or more program utilize formulas to compute partial derivatives to determine a gradient; and

the formulas are combined and simplified algebraically in order to reduce pixel shader program invocations.

8. The method of claim 1, wherein:

the neural network comprises one or more fully-connected layers; and

the one or more GPU-executable programs comprise one or more GPU-executable programs specific to the one or more fully-connected layers which utilize separate formulas for fully-connected layers.

9. The method of claim 1, wherein:

the neural network comprises one or more transitional layers; and

the one or more GPU-executable programs comprise one or more GPU-executable programs specific to the one or more transitional layers which utilize separate formulas for transitional layers.

10. The method of claim 1, wherein the graphics data describes a single triangle covering a viewport.

12. The method of claim 1, wherein the convolutional neural network performs handwriting recognition.

13. One or more computer-readable media containing data describing a convolutional network trained by the process of claim 1.

11. The method of claim it, wherein:

the one or more GPU-executable programs comprises one or more summations; and

each of the one or more summations is broken up into multiple passes.

14. One or more computer-readable media comprising instructions which, when executed on a graphics card, cause the graphics card to perform a method for training a neural network, the method comprising:

receiving a plurality of textures representing the neural network;

computing a plurality of forward passes of the neural network on a plurality of input data;

for each of the plurality of forward passes, computing a backward pass of the neural network using a gradient function; and

for each backward pass, based on the results of the gradient function, changing information contained in the plurality of textures to affect a training of the neural network.

15. The computer-readable media of claim 14, wherein:

the neural network is being trained to recognize handwritten characters;

the plurality of textures at least in part represents convolutional kernels; and

the convolutional kernels operate on input data representing handwritten characters.

16. The computer-readable media of claim 14, wherein the plurality of textures at least in part represents a fully-connected neural network level and a transitional level.

17. A graphics processing unit configured to perform a method for training a handwriting-recognition neural network, the graphics processing unit comprising:

data storage, configured to store one or more graphics textures, the graphics textures describing the handwriting-recognition neural network;

a plurality of pixel shader units configured via pixel shader programming;

to perform repeated forward passes and backward passes of the neural network on handwriting input data;

to store results in the plurality of graphics textures; and

to modify the plurality of textures based on results of the forward and backward passes in order to train the neural network.

18. The graphics processing unit of claim 17, wherein the handwriting-recognition neural network at least in part comprises one convolutional level and one fully-connected level.

19. The graphics processing unit of claim 17, wherein the one or more graphics textures are configured to describe a simplified triangle image, such that all processing performed by the graphics processing unit only requires computation on the part of the pixel shader units.

20. The graphics processing unit of claim 17, wherein the pixel shader units are configured such that summations in the forward passes and backward passes of the neural network are broken up into multiple smaller summations.