DEEP NEURAL NETWORKS TRAINING FOR SPEECH AND PATTERN RECOGNITION
First Claim
1. A computer-readable medium storing computer-executable instructions that are executable to cause one or more processors to perform acts comprising:
- providing a pipelined algorithm to train deep neural networks (DNNs) for performing data analysis based on training data, the DNNs being one of context-dependent DNNs or context-independent DNNs;
partitioning the training data into sample batches of a specific batch size based on rates of data transfers between processors for executing the pipelined algorithm and an execution speed of each processor; and
pipelining an execution of the pipelined algorithm on the DNNs through the processors to train the DNNs using the sample batches.
4 Assignments
0 Petitions
Accused Products
Abstract
The use of a pipelined algorithm that performs parallelized computations to train deep neural networks (DNNs) for performing data analysis may reduce training time. The DNNs may be one of context-independent DNNs or context-dependent DNNs. The training may include partitioning training data into sample batches of a specific batch size. The partitioning may be performed based on rates of data transfers between processors that execute the pipelined algorithm, considerations of accuracy and convergence, and the execution speed of each processor. Other techniques for training may include grouping layers of the DNNs for processing on a single processor, distributing a layer of the DNNs to multiple processors for processing, or modifying an execution order of steps in the pipelined algorithm.
-
Citations
20 Claims
-
1. A computer-readable medium storing computer-executable instructions that are executable to cause one or more processors to perform acts comprising:
-
providing a pipelined algorithm to train deep neural networks (DNNs) for performing data analysis based on training data, the DNNs being one of context-dependent DNNs or context-independent DNNs; partitioning the training data into sample batches of a specific batch size based on rates of data transfers between processors for executing the pipelined algorithm and an execution speed of each processor; and pipelining an execution of the pipelined algorithm on the DNNs through the processors to train the DNNs using the sample batches. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9)
-
-
10. A computer-implemented method, comprising:
-
providing a pipelined algorithm to train deep neural networks (DNNs) for performing data analysis based on training data, the DNNs being one of context-dependent DNNs or context-independent DNNs and including multiple layers; distributing a top layer of the DNNs across multiple processors through model striping for parallelized processing by the pipelined algorithm; and pipelining an execution of the pipelined algorithm on the DNNs through a plurality of processors to train the DNNs using the sample batches from the training data. - View Dependent Claims (11, 12, 13, 14, 15)
-
-
16. A system, comprising:
-
a plurality of processors; a memory that includes a plurality of computer-executable components that are executable by the plurality of processors, comprising; a batch generation component that partitions a training into sample batches of a specific batch size; and an algorithm execution component that pipelines an execution of an pipelined algorithm through the plurality of processors to train deep neural networks (DNNs) using the sample batches, the execution including executing a model update prior to an input data forward propagation in a computation iteration of the pipelined algorithm, the DNNs being one of context-dependent DNNs or context-independent DNNs. - View Dependent Claims (17, 18, 19, 20)
-
Specification