Feature-Augmented Neural Networks and Applications of Same

• US 20140229158A1
• Filed: 02/10/2013
• Published: 08/14/2014
• Est. Priority Date: 02/10/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zweig, Geoffrey G., Acero, Alejandro, Mikolov, Tomas
• Granted Patent: US 9,519,858 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 40/44 Statistical methods, e.g. probability models
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

A system is described herein which uses a neural network having an input layer that accepts an input vector and a feature vector. The input vector represents at least part of input information, such as, but not limited to, a word or phrase in a sequence of input words. The feature vector provides supplemental information pertaining to the input information. The neural network produces an output vector based on the input vector and the feature vector. In one implementation, the neural network is a recurrent neural network. Also described herein are various applications of the system, including a machine translation application.

Claims

1. A method for mapping input information into output information using a neural network, comprising:

receiving an input vector at an input layer of the neural network, the input vector representing at least part of the input information;

receiving a feature vector at the input layer of the neural network, the feature vector representing supplemental information pertaining to the input information; and

using the neural network to generate an output vector at an output layer of the neural network, based on the input vector and the feature vector,

the output vector representing the output information.

2. The method of claim 1, wherein the neural network comprises a recurrent network.

3. The method of claim 2, wherein the recurrent neural network has a hidden layer that operates on:

the input vector, as modified by a first matrix;

the feature vector, as modified by a second matrix; and

a time-delayed hidden-state vector that represents an output of the hidden layer in a prior time instance.

4. The method of claim 3, wherein the output layer of the neural network operates on:

the output of the hidden layer, as modified by a fourth matrix; and

the feature vector, as modified by a fifth matrix.

5. The method of claim 1, wherein the neural network is a feed-forward neural network.

6. The method of claim 1, further comprising generating the feature vector using a Latent Dirichlet Allocation (LDA) technique.

7. The method of claim 6, wherein said generating comprises:

identifying at least one word associated with a block of words that is used to compute the feature vector;

for each of said at least one word, identifying an unigram distribution of topics, to provide at least one unigram distribution of topics; and

using said at least one unigram distribution of topics to generate the feature vector.

8. The method of claim 1, further comprising generating the feature vector using a Latent Semantic Analysis (LSA) technique.

9. The method of claim 1, further comprising generating the feature vector in incremental fashion, based, in part, on a weighted version of a feature vector generated in a prior time instance.

10. The method of claim 1, wherein the feature vector represents one or more aspects of the input information that are derivable from the input information itself.

11. The method of claim 1, wherein:

the input vector represents at least one identified word in a sequence of words provided by the input information; and

the feature vector represents at least one aspect of a block of words in the sequence of words.

12. The method of claim 1, wherein the feature vector describes one or more of:

at least one characteristic of a user who is associated with the input information;

at least one characteristic of an activity performed by the user; and

at least one characteristic of an environment in which the input information is provided.

13. The method of claim 1, wherein:

the input vector represents input language information;

the feature vector represents at least one aspect of the input language information; and

the output vector represents a reformulation of the input language information, as conditioned by the supplemental information.

14. The method of claim 1, wherein:

the input vector represents at least part of language information that accompanies media content;

the feature vector describes characteristics of the media information; and

the output vector represents an interpretation of the language information, as conditioned by the supplemental information.

15. The method of claim 1, wherein:

the input vector represents source language information;

the feature vector represents supplemental information pertaining to the source language information; and

the output vector represents a translation of the source language information into a target language.

16. The method of claim 1, wherein:

the input vector represents at least one translation hypothesis, corresponding to a possible translation of source language information into a target language;

the feature vector represents supplemental information pertaining to the source language information; and

the output vector represents a translation of the source language information into the target language, as conditioned by the supplemental information.

17. A recurrent neural network, comprising:

an input layer for receiving;

an input vector, the input vector representing at least part of input information;

a feature vector, the feature vector representing supplemental information pertaining to the input information; and

a time-delayed hidden-state vector that represents an output from a hidden layer in a prior time instance;

the hidden layer, the hidden layer for operating on;

the input vector, as modified by a first matrix;

the feature vector, as modified by a second matrix;

the time-delayed hidden-state vector, as modified by a third matrix; and

an output layer for providing an output vector by operating on;

an output hidden-state vector provided by the hidden layer, as modified by a fourth matrix; and

the feature vector, as modified by a fifth matrix,

the output vector providing output information, as conditioned by the supplemental information.

18. The recurrent neural network of claim 17, wherein:

the input vector represents source language information;

the feature vector represents supplemental information pertaining to the source language information; and

the output vector represents a translation of the source language information into a target language.

19. The recurrent neural network of claim 17, wherein:

the input vector represents at least one translation hypothesis, corresponding to a possible translation of source language information into a target language;

the feature vector represents supplemental information pertaining to the source language information; and

the output vector represents a translation of the source language information into the target language, as conditioned by the supplemental information.

20. A system for processing linguistic information, comprising:

an input information-providing module configured to provide an input vector, the input vector representing at least one identified word in a sequence of words;

a feature information-providing module configured to provide a feature vector, the feature vector describing one or more of;

at least one aspect of a block of words in the sequence of words; and

information that pertains to the sequence of words obtained from a source other than the sequence of words; and

a recurrent neural network configured to generate an output vector based on the input vector and the feature vector.