System and method for accelerating and optimizing the processing of machine learning techniques using a graphics processing unit

• US 7,219,085 B2
• Filed: 04/30/2004
• Issued: 05/15/2007
• Est. Priority Date: 12/09/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Simard, Patrice Y., Steinkraus, David W., Buck, Ian Andrew
• Primary Examiner(s): Knight; Anthony
• Assistant Examiner(s): Brown, Jr.; Nathan H.
• Application Number:
  US10/837,382
• Publication Number:
  US 20050125369A1
• Time in Patent Office: 1,110 Days
• Field of Search: 706/12
• US Class Current: 706/12
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06V 10/955 using specific electronic processors

Abstract

A system and method for processing machine learning techniques (such as neural networks) and other non-graphics applications using a graphics processing unit (GPU) to accelerate and optimize the processing. The system and method transfers an architecture that can be used for a wide variety of machine learning techniques from the CPU to the GPU. The transfer of processing to the GPU is accomplished using several novel techniques that overcome the limitations and work well within the framework of the GPU architecture. With these limitations overcome, machine learning techniques are particularly well suited for processing on the GPU because the GPU is typically much more powerful than the typical CPU. Moreover, similar to graphics processing, processing of machine learning techniques involves problems with solving non-trivial solutions and large amounts of data.

Claims

1. A computer-implemented method comprising:

processing a machine learning technique using a graphics processing unit to obtain results, wherein processing a machine learning technique further comprises using a pixel shader to compute an inner product that is at least one of;

(a) a vector inner product;

(b) a matrix inner product;

decomposing the inner product into sub-problems and performing multiple passes over the sub-problems using pixel shaders;

using the results to provide solutions for use by a computer application.

2. The computer-implemented method of claim 1, wherein processing a machine learning technique using a graphics processing unit further comprises at least one of:

(a) training learnable parameters of the machine learning technique using the graphics processing unit;

(b) testing and computing functions of an input and trained learnable parameters using the graphics processing unit;

(c) training learnable parameters and testing and computing functions of an input and the trained learnable parameters using the graphics processing unit.

3. The computer-implemented method of claim 1, wherein the results are trained learnable parameters, and further comprising:

training learnable parameters of the machine learning technique using the graphic processing unit to obtain the trained learnable parameters; and

using the trained learnable parameters to obtain the solutions.

4. The computer-implemented method of claim 1, further comprising:

obtaining trained learnable parameters using a central processing unit; and

obtaining results from testing and computing functions of an input and the trained learnable parameters using the graphics processing unit.

5. The computer-implemented method of claim 1, further comprising:

obtaining trained learnable parameters of the machine learning technique using the graphic processing unit; and

testing and computing functions of an input and the trained learnable parameters using the graphics processing unit.

6. The computer-implemented method of claim 1, wherein the computer application is an interactive use computer application that is one of:

(a) a speech recognition application;

(b) a handwriting recognition application.

7. The computer-implemented method of claim 1, wherein the machine learning technique is a neural network.

8. The computer-implemented method of claim 1, wherein processing a machine learning technique further comprises using a pixel shader to compute an outer product.

9. The computer-implemented method of claim 1, wherein processing a machine learning technique further comprises using a pixel shader to perform a matrix transpose.

10. A computer-readable medium having computer-executable instructions for performing the computer-implemented method recited in claim 1.

11. The computer-implemented method of claim 3, further comprising storing at least some of the trained learnable parameters on the graphics processing unit.

12. The computer-implemented method of claim 8, further comprising computing the outer product using texture mapping.

13. The computer-implemented method of claim 9, further comprising performing the matrix transpose using texture mapping.

14. A process for accelerating and optimizing a machine learning technique using a graphics processing unit, comprising:

training learnable parameters of the machine learning technique to obtain trained learnable parameters;

using the trained learnable parameters to obtain results from the machine learning technique;

using pixel shaders to perform at least one of the following;

(a) training learnable parameters;

(b) using the trained learnable parameters; and

outputting the results for use by an application;

wherein the machine learning technique uses an Expectation Maximization (EM) technique.

15. The process as set forth in claim 14, wherein the pixel shaders reside on a graphic card.

16. The process as set forth in claim 14, further comprising locating at least some of the trained learnable parameters on the graphics processing unit.

17. The process as set forth in claim 14, further comprising using pixel shaders to compute at least one of:

(a) a vector inner product;

(b) a matrix inner product;

(c) an outer product;

(d) a matrix transpose.

18. The process as set forth in claim 14 wherein the machine learning technique is a neural network.

19. The process as set forth in claim 14, wherein the machine learning technique is at least one of:

(a) a K-means technique;

(b) a Learning Vector Quantization (LVQ) technique.

20. The process as set forth in claim 14, further comprising preprocessing data using the graphics processing unit.

21. One or more computer-readable media having computer-readable instructions thereon which, when executed by one or more processors, cause the one or more processors to implement the process of claim 14.

22. The process as set forth in claim 18, wherein the neural network is at least one of:

(a) a multi-layer, fully connected neural network;

(b) a convolutional neural network.

23. The process as set forth in claim 20, wherein preprocessing comprises at least one of:

(a) normalizing data to put it in better form for input;

(b) extracting information and features from the data;

(c) obtaining input data and loading the input data onto the graphics processing unit.

24. A computer-readable medium having computer-executable instructions for processing a machine learning technigue using a graphics processing unit, comprising:

preprocessing query data to obtain input data;

loading the input data onto the graphics processing unit;

loading shaders onto the graphics processing unit;

calling the shaders to process a trained function on the graphic processing unit using the machine learning technigue; and

obtaining results for use by an interactive use application.

25. The computer-readable medium of claim 24, further comprising using the shaders to compute primitive operations on the graphics processing unit.

26. The computer-readable medium of claim 24, wherein the machine learning technique is a neural network.

27. The computer-readable medium of claim 24, wherein the machine learning technique is a gradient descent technique.

28. The computer-readable medium of claim 24, wherein the interactive use application is a handwriting recognition application.

29. The computer-readable medium of claim 24, wherein the interactive use application is an optical character recognition application.

30. The computer-readable medium of claim 24, wherein the machine learning technique is a gradient descent technique and further comprising:

storing gradient descent parameters on the graphics processing unit; and

extending the precision of the gradient descent parameters by storing each parameter in two storage locations, one storage location containing the more-significant digits of the parameter value and the other storage location containing the less-significant digits.

31. The computer-readable medium of claim 25, wherein the primitive operations include at least one of:

(a) a vector inner product;

(b) a matrix inner product;

(c) an outer product;

(d) a matrix transpose.

32. The computer-readable medium of claim 27, further comprising having at least some of adjustable parameters of the gradient descent technique reside on the graphics processing unit.

33. The computer-readable medium of claim 28, wherein the machine learning technique is a neural network and further comprising performing classification using the neural network.

34. The computer-readable medium of claim 29, wherein the machine learning technique is a neural network and further comprising performing classification using the neural network.

35. The computer-readable medium of claim 24, further comprising having at least some learnable parameters reside on the graphics processing unit.