Feature-augmented neural networks and applications of same

• US 9,519,858 B2
• Filed: 02/10/2013
• Issued: 12/13/2016
• Est. Priority Date: 02/10/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Zweig, Geoffrey G., Mikolov, Tomas
• Primary Examiner(s): Goddard, Tammy Paige
• Assistant Examiner(s): Leland, III, Edwin S
• Application Number:
  US13/763,701
• Publication Number:
  US 20140229158A1
• Time in Patent Office: 1,402 Days
• Field of Search: 704/9
• US Class Current: 1/1
• CPC Class Codes: G06F 40/44 Statistical methods, e.g. probability models
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

A system is described herein which uses a neural network having an input layer that accepts an input vector and a feature vector. The input vector represents at least part of input information, such as, but not limited to, a word or phrase in a sequence of input words. The feature vector provides supplemental information pertaining to the input information. The neural network produces an output vector based on the input vector and the feature vector. In one implementation, the neural network is a recurrent neural network. Also described herein are various applications of the system, including a machine translation application.

Claims

1. A method performed using one or more processing devices, the method comprising:

receiving a word input vector at an input layer of a neural network, the word input vector representing an individual word from an input sequence of words;

receiving a topic feature vector at the input layer of the neural network, the topic feature vector being separate from the word input vector and representing topics expressed in the input sequence of words;

using the neural network to generate an output vector at an output layer of the neural network based at least on the word input vector and the topic feature vector, wherein using the neural network includes, by a hidden layer of the neural network;

modifying the word input vector using a first learned matrix; and

modifying the topic feature vector using a second learned matrix that is separate from the first learned matrix,

wherein the output vector represents a word probability given the word input vector and the topic feature vector; and

performing a natural language processing operation based at least on the word probability represented by the output vector.

2. The method of claim 1, wherein using the neural network includes:

by the hidden layer of the neural network, modifying a time-delayed hidden-state vector with a third learned matrix, wherein the time-delayed hidden-state vector represents an output of the hidden layer in a prior time instance,

wherein the word input vector, the topic feature vector, and the time-delayed hidden-state vector are separate vectors, and

wherein the first learned matrix, the second learned matrix, and the third learned matrix are separate matrices.

3. The method of claim 2, wherein using the neural network includes, by the output layer of the neural network:

modifying the output of the hidden layer with a fourth learned matrix; and

modifying the topic feature vector with a fifth learned matrix,

wherein the first learned matrix, the second learned matrix, the third learned matrix, the fourth learned matrix, and the fifth learned matrix are separate matrices.

4. The method of claim 2, wherein using the neural network includes, by the hidden layer:

performing a first multiplication operation of the word input vector by the first learned matrix to generate a first multiplication output;

performing a second multiplication operation of the topic feature vector by the second learned matrix to generate a second multiplication output;

performing a third multiplication operation of the time-delayed hidden-state vector by the third learned matrix to generate a third multiplication output; and

summing the first multiplication output, the second multiplication output, and the third multiplication output to generate the output of the hidden layer.

5. The method of claim 1, further comprising:

generating the topic feature vector using a Latent Dirichlet Allocation (LDA) technique; and

as subsequent words from the input sequence are processed using the neural network, incrementally generating next topic feature vectors based at least on previous feature vectors.

6. The method of claim 5, wherein said incrementally generating comprises applying a decay factor to previous topic feature vectors for previous words that have already been processed by the neural network.

7. The method of claim 1, wherein the input sequence of words is part of an input document.

8. A system comprising:

at least one processing device; and

at least one computer readable medium storing instructions which, when executed by the at least one processing device, cause the at least one processing device to;

receive a word input vector at an input layer of a neural network, the word input vector representing an individual word from an input sequence of words;

receive a topic feature vector at the input layer of the neural network, the topic feature vector being separate from the word input vector and representing topics expressed in the input sequence of words;

use the neural network to generate an output vector at an output layer of the neural network based at least on the word input vector and the topic feature vector, wherein using the neural network includes, by a hidden layer of the neural network;

modifying the word input vector using a first learned matrix; and

modifying the topic feature vector using a second learned matrix that is separate from the first learned matrix,

wherein the output vector represents a word probability given the word input vector and the topic feature vector; and

perform a natural language processing operation based at least on the word probability represented by the output vector.

9. The system of claim 8, wherein the instructions, when executed by the at least one processing device, cause the at least one processing device to:

by the hidden layer of the neural network, modify a time-delayed hidden-state vector with a third learned matrix,

wherein the time-delayed hidden-state vector represents an output of the hidden layer in a prior time instance,

wherein the word input vector, the topic feature vector, and the time-delayed hidden-state vector are separate vectors, and

wherein the first learned matrix, the second learned matrix, and the third learned matrix are separate matrices.

10. The system of claim 9, wherein the instructions, when executed by the at least one processing device, cause the at least one processing device to:

by the output layer of the neural network;

modify the output of the hidden layer with a fourth learned matrix; and

modify the topic feature vector with a fifth learned matrix,

wherein the first learned matrix, the second learned matrix, the third learned matrix, the fourth learned matrix, and the fifth learned matrix are separate matrices.

11. The system of claim 9, wherein the instructions, when executed by the at least one processing device, cause the at least one processing device to:

by the hidden layer of the neural network;

perform a first multiplication operation of the word input vector by the first learned matrix to generate a first multiplication output;

perform a second multiplication operation of the topic feature vector by the second learned matrix to generate a second multiplication output;

perform a third multiplication operation of the time-delayed hidden-state vector by the third learned matrix to generate a third multiplication output; and

sum the first multiplication output, the second multiplication output, and the third multiplication output to generate the output of the hidden layer.

12. The system of claim 8, wherein the instructions, when executed by the at least one processing device, cause the at least one processing device to:

generate the topic feature vector using a Latent Dirichlet Allocation (LDA) technique; and

as subsequent words from the input sequence are processed using the neural network, incrementally generate next topic feature vectors based at least on previous feature vectors.

13. The system of claim 12, wherein the instructions, when executed by the at least one processing device, cause the at least one processing device to:

apply a decay factor to previous topic feature vectors for previous words that have already been processed by the neural network.

14. The system of claim 8, wherein the input sequence of words is part of an input document.

15. At least one computer readable storage medium storing instructions which, when executed by at least one processing device, cause the at least one processing device to perform acts comprising:

receiving a word input vector at an input layer of a neural network, the word input vector representing an individual word from an input sequence of words;

receiving a topic feature vector at the input layer of the neural network, the topic feature vector being separate from the word input vector and representing topics expressed in the input sequence of words;

using the neural network to generate an output vector at an output layer of the neural network based at least on the word input vector and the topic feature vector, wherein using the neural network includes, by a hidden layer of the neural network;

modifying the word input vector using a first learned matrix; and

modifying the topic feature vector using a second learned matrix that is separate from the first learned matrix,

wherein the output vector represents a word probability given the word input vector and the topic feature vector; and

performing a natural language processing operation based at least on the word probability represented by the output vector.

16. The at least one computer readable storage medium of claim 15, wherein using the neural network includes:

by the hidden layer of the neural network, modifying a time-delayed hidden-state vector with a third learned matrix, wherein the time-delayed hidden-state vector represents an output of the hidden layer in a prior time instance,

wherein the word input vector, the topic feature vector, and the time-delayed hidden-state vector are separate vectors, and

wherein the first learned matrix, the second learned matrix, and the third learned matrix are separate matrices.

17. The at least one computer readable storage medium of claim 16, wherein using the neural network includes, by the output layer of the neural network:

modifying the output of the hidden layer with a fourth learned matrix; and

modifying the topic feature vector with a fifth learned matrix,

wherein the first learned matrix, the second learned matrix, the third learned matrix, the fourth learned matrix, and the fifth learned matrix are separate matrices.

18. The at least one computer readable storage medium of claim 16 wherein using the neural network includes, by the hidden layer:

performing a first multiplication operation of the word input vector by the first learned matrix to generate a first multiplication output;

performing a second multiplication operation of the topic feature vector by the second learned matrix to generate a second multiplication output;

performing a third multiplication operation of the time-delayed hidden-state vector by the third learned matrix to generate a third multiplication output; and

summing the first multiplication output, the second multiplication output, and the third multiplication output to generate the output of the hidden layer.

19. The at least one computer readable storage medium of claim 15, the acts further comprising:

generating the topic feature vector using a Latent Dirichlet Allocation (LDA) technique; and

as subsequent words from the input sequence are processed using the neural network, incrementally generating next topic feature vectors based at least on previous feature vectors.

20. The at least one computer readable storage medium of claim 19, wherein said incrementally generating comprises applying a decay factor to previous topic feature vectors for previous words that have already been processed by the neural network.