COMPUTER-IMPLEMENTED DEEP TENSOR NEURAL NETWORK

• US 20140067735A1
• Filed: 08/29/2012
• Published: 03/06/2014
• Est. Priority Date: 08/29/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Yu, Dong, Deng, Li, Seide, Frank
• Granted Patent: US 9,292,787 B2
• Time in Patent Office: Days
• US Class Current: 706/20
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent

Abstract

A deep tensor neural network (DTNN) is described herein, wherein the DTNN is suitable for employment in a computer-implemented recognition/classification system. Hidden layers in the DTNN comprise at least one projection layer, which includes a first subspace of hidden units and a second subspace of hidden units. The first subspace of hidden units receives a first nonlinear projection of input data to a projection layer and generates the first set of output data based at least in part thereon, and the second subspace of hidden units receives a second nonlinear projection of the input data to the projection layer and generates the second set of output data based at least in part thereon. A tensor layer, which can converted into a conventional layer of a DNN, generates the third set of output data based upon the first set of output data and the second set of output data.

Claims

1. A method, comprising:

at a recognition system, receiving a sample; and

assigning a label to the sample through employment of a computer-implemented deep tensor neural network, the deep tensor neural network comprising a plurality of layers, the plurality of layers comprising;

a plurality of hidden layers that comprise a projection layer and a tensor layer, the projection comprising a plurality of subspaces, each subspace in the plurality of subspaces comprising respective hidden units, and wherein each subspace is provided with a respective projection of input data from a first immediately adjacent layer in the deep tensor neural network, wherein projections provided to respective subspaces are nonlinear projections of the input data, and wherein the tensor layer receives respective output data from each subspace in the plurality of subspaces and generates an output vector for provision to a second immediately adjacent layer in the deep tensor neural network, the output vector being based at least in part upon the respective output data from subspaces in the projection layer.

2. The method of claim 1, wherein the second immediately adjacent layer is a second projection layer.

3. The method of claim 1, wherein a sigmoid function is applied to at least one projection of the input data.

4. The method of claim 1, wherein the sample is one of a voice sample or a handwriting sample.

5. The method of claim 1, wherein the plurality of hidden layers comprise a plurality of projection layers and a plurality of tensor layers.

6. The method of claim 1, further comprising learning weights between hidden layers in the deep neural network through backpropagation.

7. The method of claim 1, wherein the label assigned to the sample is one of a word, a phone, or a senone

8. The method of claim 1, wherein the first immediately adjacent layer and the second immediately adjacent layer are hidden layers in the deep tensor neural network.

9. The method of claim 1, wherein the tensor layer is converted into a conventional layer by converting the weight tensor to a weight matrix, the weight matrix defining the synapses'"'"' connection strengths between the projection layer and the second immediately adjacent layer in the deep tensor neural network.

10. The method of claim 1 configured for execution in a mobile computing device.

11. The method of claim 10, wherein the mobile computing device is one of a mobile telephone or a tablet computing device.

12. A computing apparatus, comprising:

a processor; and

a memory, the memory comprising a plurality of components that are executed by the processor, the plurality of components comprising;

a receiver component that receives a sample from a user; and

a recognizer component that assigns a label to the sample, the recognizer component comprising a deep tensor neural network that comprises a plurality of hidden layers, the plurality of hidden layers comprising a plurality of projection layers and a plurality of tensor layers, wherein each projection layer comprises at least two subspaces of hidden units that are provided with respective projections of input data received from a first immediately adjacent layer in the deep tensor neural network, and wherein each tensor layer receives respective output data from each subspace in an immediately adjacent projection layer and generates a respective output vector for receipt at a second immediately adjacent layer in the deep tensor neural network based at least in part upon the output data received from each of the subspaces.

13. The computing apparatus of claim 12 being a mobile computing device.

14. The computing apparatus of claim 13, wherein the mobile computing device is one of a mobile telephone or a tablet computing device.

15. The computing apparatus of claim 12, wherein the respective projections are nonlinear projections of the input data.

16. The computing apparatus of claim 15, wherein a sigmoid function is employed to generate at least one nonlinear projection.

17. The computing apparatus of claim 12, wherein the sample is an utterance spoken by the user, and wherein the label is indicative of at least one word in the spoken utterance.

18. The computing apparatus of claim 12, wherein the respective projections are provided to the respective subspaces based at least in part upon respective sets of weighted synapses.

19. The computing apparatus of claim 12, wherein weights of parameters of the deep tensor neural network are learned by way of a backpropagation algorithm.

20. A computer-readable medium comprising instructions that, when executed by a processor, cause the processor to perform acts comprising:

receiving, by way of a microphone, a spoken utterance from an individual;

utilizing a deep tensor neural network to assign a label to at least a portion of the spoken utterance, the label being indicative of at least one word in the spoken utterance, and wherein the deep neural network comprises a plurality of hidden layers, the plurality of hidden layers comprising a plurality of projection layers and a plurality of tensor layers, wherein each projection layer is coupled to a respective tensor layer, wherein each projection layer comprises at least two subspaces of hidden units that are provided with respective nonlinear projections of input data received from a first immediately adjacent layer in the deep tensor neural network, and wherein each tensor layer maps at least two output vectors generated by the at least two subspaces of hidden units into output data for provision to a second immediately adjacent layer in the deep tensor neural network.