DISCRIMINATIVE PRETRAINING OF DEEP NEURAL NETWORKS

• US 20130138436A1
• Filed: 11/26/2011
• Published: 05/30/2013
• Est. Priority Date: 11/26/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Yu, Dong, Deng, Li, Seide, Frank Torsten Bernd, Li, Gang
• Granted Patent: US 9,235,799 B2
• Time in Patent Office: Days
• US Class Current: 704/232
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

Discriminative pretraining technique embodiments are presented that pretrain the hidden layers of a Deep Neural Network (DNN). In general, a one-hidden-layer neural network is trained first using labels discriminatively with error back-propagation (BP). Then, after discarding an output layer in the previous one-hidden-layer neural network, another randomly initialized hidden layer is added on top of the previously trained hidden layer along with a new output layer that represents the targets for classification or recognition. The resulting multiple-hidden-layer DNN is then discriminatively trained using the same strategy, and so on until the desired number of hidden layers is reached. This produces a pretrained DNN. The discriminative pretraining technique embodiments have the advantage of bringing the DNN layer weights close to a good local optimum, while still leaving them in a range with a high gradient so that they can be fine-tuned effectively.

Claims

1. A computer-implemented process for pretraining deep neural network (DNN), comprising:

using a computer to perform the following process actions;

(a) training a single hidden layer neural network (NN) comprising an input layer into which training data is input, an output layer from which an output is generated, and a first hidden layer which is interconnected with the input and output layers with randomly initialized weights, wherein said training comprises,

accessing a set of training data entries, each data entry of which has a corresponding label assigned thereto,

inputting each data entry of said set one by one into the input layer until all the data entries have been input at least once to produce an initial NN, such that after the inputting of each data entry, said weights associated with the first hidden layer are set via an error BP procedure so that the output generated from the output layer matches the label associated with the training data entry;

(b) discarding the current output layer and adding a new hidden layer which is interconnected with the last previously trained hidden layer and a new output layer with randomly initialized weights to produce a new multiple hidden layer deep neural network;

(c) inputting each data entry of said set one by one into the input layer until all the data entries have been input at least once to produce a revised multiple hidden layer deep neural network, such that after the inputting of each data entry, said weights associated with the new hidden layer and each previously trained hidden layer are set via the error BP procedure to produce an output from the new output layer that matches the label associated with the training data entry;

(d) repeating actions (b) and (c) until a prescribed number of hidden layers have been added; and

(e) designating the last produced revised multiple layer DNN to be said pretrained DNN.

2. The process of claim 1, wherein each output layer employed uses a softmax function to match its output to the label associated with a currently entered training data entry.

3. The process of claim 1, wherein the process action of accessing a set of training data entries, each data entry of which has a corresponding label assigned thereto, comprises accessing a set of speech frames each of which corresponds to a senone label.

4. The process of claim 1, wherein the process action of inputting each data entry of said set one by one into the input layer until all the data entries have been input at least once to produce an initial deep neural network, comprises inputting each data entry of the set just once.

5. The process of claim 1, wherein the process action of inputting each data entry of said set one by one into the input layer until all the data entries have been input at least once to produce a revised multiple hidden layer deep neural network, comprises inputting each data entry of the set just once.

6. The process of claim 1, wherein the error BP procedure used to set the weights associated with the first hidden layer employs a prescribed learning rate that ranges between 0.01 and 0.20.

7. The process of claim 1, wherein the error BP procedure used to set the weights associated with each new hidden layer and each previously trained hidden layer employs a prescribed learning rate that ranges between 0.01 and 0.20.

8. A system for training a context-dependent deep neural network (CD-DNN), comprising:

a computing device;

a computer program comprising program modules executable by the computing device, comprising,

a hidden layer generator program module wherein the computing device is directed by the hidden layer generator program module to,

initially generate a single hidden layer neural network comprising an input layer into which training data is input, an output layer from which an output is generated, and a first hidden layer which is interconnected with the input and output layers with randomly initialized weights,

whenever a pretrained version of the single hidden layer neural network is produced, discard the current output layer and add a new hidden layer which is interconnected with the first hidden layer and a new output layer with randomly initialized weights to produce a multiple hidden layer deep neural network, and

whenever a pretrained version of a last produced multiple hidden layer deep neural network is produced and is designated as lacking a prescribed number of hidden layers, discard the current output layer and add a new hidden layer which is interconnected with the last previously added hidden layer and a new output layer with randomly initialized weights to produce a new multiple hidden layer deep neural network,

a pretraining program module wherein the computing device is directed by the pretraining program module to,

access a set of training data entries, each data entry of which has a corresponding label assigned thereto,

access the single hidden layer neural network once it is generated,

input each data entry of said set one by one into the input layer of the single hidden layer neural network until all the data entries have been input at least once to produce the pretrained version of the single hidden layer neural network, such that after the inputting of each data entry, said weights associated with the first hidden layer are set via an error backpropagation procedure to produce an output from the output layer that matches the label associated with the training data entry;

access each multiple hidden layer deep neural network at the time it is produced,

for each multiple hidden layer deep neural network accessed, input each data entry of said set of training data entries one by one into the input layer until all the data entries have been input at least once to produce a pretrained version of the accessed multiple hidden layer deep neural network, such that after the inputting of each data entry, said weights associated with the last added hidden layer and each previously trained hidden layer are set via the error BP procedure to produce an output from the output layer that matches the label associated with the training data entry, and

a DNN module wherein the computing device is directed by the DNN module to,

each time a pretrained version of a multiple hidden layer DNN is produced, determining whether it includes said prescribed number of hidden layers, and

whenever it is determined the last produced pretrained multiple hidden layer deep neural network does not include the prescribed number of hidden layers, designating it as lacking the prescribed number of hidden layers, and

whenever it is determined the last produced pretrained multiple hidden layer deep neural network does include the prescribed number of hidden layers, designating it to be a pretrained DNN.

9. The system of claim 8, further comprising a fine-tuning module wherein the computing device is directed by the fine-tuning module to iteratively train the pretrained DNN until the weights associated with the each hidden layer do not vary between iterations by more than a prescribed training threshold to produce said trained DNN, wherein each training iteration comprises inputting each data entry of said set of training data entries one by one into the input layer until all the data entries have been input once to produce a new fine-tuned version of the pretrained DNN, such that after the inputting of each data entry, said weights associated with the hidden layers are set via the error back-propagation procedure so that the output generated from the output layer matches the label associated with the training data entry.

10. The system of claim 8, wherein said accessing of a set of training data entries, each data entry of which has a corresponding label assigned thereto, comprises accessing a set of speech frames each of which corresponds to a senone label.

11. The system of claim 8, wherein the pretraining program module directs the computing device to input each data entry of said set one by one into the input layer of the single hidden layer neural network until all the data entries have been input only once to produce the pretrained version of the single hidden layer DNN.

12. The system of claim 8, wherein the pretraining program module directs the computing device to, for each multiple hidden layer DNN accessed, input each data entry of said set of training data entries one by one into the input layer until all the data entries have been input only once to produce a pretrained version of the accessed multiple hidden layer DNN.

13. The system of claim 8, wherein the pretraining program module directs the computing device to set said weights associated with the first hidden layer via an error backpropagation procedure employing a prescribed learning rate that ranges between 0.01 and 0.20 to produce an output from the output layer that matches the label associated with the training data entry.

14. The system of claim 8, wherein the pretraining program module directs the computing device to set said weights associated with the last added hidden layer and each previously trained hidden layer via an error BP procedure employing a prescribed learning rate that ranges between 0.01 and 0.20 to produce an output from the output layer that matches the label associated with the training data entry.

15. A computer-readable storage medium having computer-executable instructions stored thereon for training a deep neural network (DNN), said computer-executable instructions comprising:

(a) training a single hidden layer neural network (NN) comprising an input layer into which training data is input, an output layer from which an output is generated, and a first hidden layer which is interconnected with the input and output layers with randomly initialized weights, wherein said training comprises,

accessing a set of training data entries, each data entry of which has a corresponding label assigned thereto,

inputting each data entry of said set one by one into the input layer until all the data entries have been input once to produce an initial NN, such that after the inputting of each data entry, said weights associated with the first hidden layer are set via an error backpropagation procedure which employs a prescribed learning rate that ranges between 0.01 and 0.20 to produce an output from the output layer that matches the label associated with the training data entry;

(b) discarding the current output layer and adding a new hidden layer which is interconnected with the last previously trained hidden layer and a new output layer with randomly initialized weights to produce a new multiple hidden layer deep neural network;

(c) training the last produced new multiple hidden layer deep neural network, wherein said training comprises, inputting each data entry of said set one by one into the input layer until all the data entries have been input once to produce a revised multiple hidden layer deep neural network, such that after the inputting of each data entry, said weights associated with the new hidden layer and each previously trained hidden layer are set via the error backpropagation procedure which employs said prescribed learning rate so that the output generated from the output layer matches the label associated with the training data entry;

(d) repeating instructions (b) and (c) until a prescribed number of hidden layers have been added; and

(e) designating the last produced revised multiple layer DNN to be a pretrained DNN.

16. The computer-readable storage medium of claim 15, wherein the instruction for training the single hidden layer NN comprises each output layer employing a softmax function to match its output to the label associated with a currently entered training data entry.

17. The computer-readable storage medium of claim 15, wherein the instruction for training the last produced new multiple hidden layer deep neural network comprises each output layer employing a softmax function to match its output to the label associated with a currently entered training data entry.

18. The computer-readable storage medium of claim 15, wherein the instruction for accessing a set of training data entries, each data entry of which has a corresponding label assigned thereto, comprises accessing a set of speech frames each of which corresponds to a senone label.

19. The computer-readable storage medium of claim 15, further comprising an instruction for iteratively training the pretrained DNN a prescribed number of times to produce said trained DNN, wherein each training iteration comprises inputting each data entry of a set of training data entries one by one into the input layer until all the data entries have been input once to produce a new fine-tuned version of the pretrained DNN, such that after the inputting of each data entry, said weights associated with the hidden layers are set via the error backpropagation procedure to produce an output from the output layer that matches the label associated with the training data entry.

20. The computer-readable storage medium of claim 19, wherein the instruction for iteratively training the pretrained DNN a prescribed number of times to produce said trained DNN, comprises training the pretrained DNN four times to produce said trained DNN.