Systems and Methods for Neural Language Modeling

• US 20160247061A1
• Filed: 02/18/2016
• Published: 08/25/2016
• Est. Priority Date: 02/19/2015
• Status: Active Grant

• Current Assignee: Digital Reasoning Systems, Inc (Smarsh Incorporated)
• Original Assignee: Digital Reasoning Systems, Inc (Smarsh Incorporated)
• Inventors: Trask, Andrew, Gilmore, David, Russell, Matthew
• Granted Patent: US 10,339,440 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/09 Supervised learning
• CPC Class Codes: G06N 3/098 Distributed learning, e.g. federated learning

Abstract

In some aspects, the present disclosure relates to neural language modeling. In one embodiment, a computer-implemented neural network includes a plurality of neural nodes, where each of the neural nodes has a plurality of input weights corresponding to a vector of real numbers. The neural network also includes an input neural node corresponding to a linguistic unit selected from an ordered list of a plurality of linguistic units, and an embedding layer with a plurality of embedding node partitions. Each embedding node partition includes one or more neural nodes. Each of the embedding node partitions corresponds to a position in the ordered list relative to a focus term, is configured to receive an input from an input node, and is configured to generate an output. The neural network also includes a classifier layer with a plurality of neural nodes, each configured to receive the embedding outputs from the embedding layer, and configured to generate an output corresponding to a probability that a particular linguistic unit is the focus term.

Claims

1. A computer-implemented neural network, comprising:

a plurality of neural nodes, each of the neural nodes having a plurality of input weights corresponding to a vector of real numbers;

an input neural node corresponding to a linguistic unit selected from an ordered list of a plurality of linguistic units;

an embedding layer comprising a plurality of embedding node partitions, each embedding node partition comprising one or more neural nodes, wherein each of the embedding node partitions corresponds to a position in the ordered list relative to a focus term, is configured to receive an input from an input node, and is configured to generate an output; and

a classifier layer comprising a plurality of neural nodes, each neural node in the classifier layer configured to receive the embedding outputs from the embedding layer, and configured to generate an output corresponding to a probability that a particular linguistic unit is the focus term.

2. The computer-implemented neural network of claim 1, wherein the linguistic unit is a character.

3. The computer-implemented neural network of claim 1, wherein the linguistic unit is a word.

4. The computer-implemented neural network of claim 1, wherein the positions relative to a focus term of the embedding node partitions are window positions relative to the focus term.

5. The computer-implemented neural network of claim 1, wherein the positions relative to a focus term of the embedding node partitions are directions relative to the focus term.

6. The computer-implemented neural network of claim 1, wherein the neural network is trained by performing functions that comprise:

removing the focus term from the ordered list of linguistic units;

selecting a partition from each remaining linguistic unit'"'"'s embeddings based on that linguistic unit'"'"'s position relative to the focus term;

concatenating the partitions;

propagating the partitions through the classifier layer, and

updating weights for one or more neural nodes in the classifier layer and embedding layer such that a probability of the presence of the focus term is approximately 100% and other randomly sampled linguistic units are approximately 0%.

7. The computer-implemented neural network of claim 1, wherein the neural network is trained by performing functions that comprise:

training a first partition of the embedding node partitions by;

removing the focus term from the ordered list of linguistic units;

selecting a partition from each remaining linguistic unit'"'"'s embeddings based on that respective linguistic unit'"'"'s position relative to the focus term; and

updating weights for each neural node in the classifier layer and embedding layer such that the probability of the presence of the focus term is about 100% and other randomly sampled linguistic units are 0%.

8. The computer-implemented neural network of claim 7, wherein the neural network is trained by performing functions that further comprise:

training a second partition by;

removing the focus term from the ordered list of linguistic units;

selecting a partition from each remaining linguistic unit'"'"'s embedding based on that linguistic unit'"'"'s position relative to the focus term; and

updating the weights to model the probability of the presence of the focus term is about 100% and other randomly sampled linguistic units are 0%,

wherein the steps of training a first partition and training a second partition are performed in parallel.

9. The computer-implemented neural network of claim 1, wherein particular linguistic units are selected from the ordered list of the plurality of linguistic units and multiple linguistic domains associated with the selected particular linguistic units are modeled into a common vector space, wherein each of the multiple linguistic domains corresponds to a different language.

10. A computer-implemented neural network, comprising:

a computer-implemented data structure including;

an input matrix syn0 of N rows and M columns, wherein each row corresponds to a linguistic unit and all columns are partitioned into P partitions, wherein each partition is a vector of an equal number of numerical values, and

an output matrix syn1 of M rows and N columns, wherein each column corresponds to a linguistic unit and all rows are partitioned into P partitions of equal number and size to the partitions in syn0, wherein each partition is a vector of an equal number of numerical values,

wherein the neural network is trained to perform functions that include predicting a linguistic unit given its surrounding context linguistic units within a sentence, wherein each context linguistic unit relates to a partition of its corresponding row in syn0 corresponding to its relative position relative to the focus term and multiples by the corresponding partition in syn1 corresponding to this same relative partition in the focus term.

11. The computer-implemented neural network of claim 10, wherein each selected partition in syn0 and syn1 is updated to make correct predictions true and incorrect predictions false.

12. The computer-implemented neural network of claim 10, wherein the neural network is trained by performing functions that comprise:

removing the focus term from the ordered list of linguistic units;

selecting a partition from each remaining linguistic unit'"'"'s embedding based on that linguistic unit'"'"'s position relative to the focus term;

concatenating the partitions;

propagating the partitions through syn1; and

updating the weights for each neural node in the classifier layer and embedding layer such that the probability of the presence of the focus term is approximately 100% and the other randomly sampled linguistic units are approximately 0%.

13. The computer-implemented neural network of claim 10, wherein each partition is trained by performing functions that comprise:

removing the focus term from the ordered list of linguistic units;

selecting a partition from each remaining linguistic unit'"'"'s embedding based on that linguistic unit'"'"'s position relative to the focus term; and

updating the weights for each neural node in syn1 and syn0 such that the probability of the presence of the focus term approaches 100% and the other randomly sampled linguistic units is about 0%.

14. The computer-implemented neural network of claim 10, wherein in accordance with an interpolation of the syn0 vectors, the linguistic units are characters according to the following procedure:

interpolating character embeddings in a bucketed fashion, such that the final word embedding is comprised of a predetermined number of buckets; and

weighting each character, via summing, into each final word embedding based on proximity to the bucket position given the respective character position in the word, such that the distance is calculated according to a squared percent distance across the word,

wherein character embeddings do not overlap across the bucketed partitions in the final word embedding.

15. A system having one or more processors configured to implement:

a plurality of neural nodes, each of the neural nodes having a plurality of input weights corresponding to a vector of real numbers;

an input neural node corresponding to a linguistic unit selected from an ordered list of a plurality of linguistic units;

an embedding layer comprising a plurality of embedding node partitions, each embedding node partition comprising one or more neural nodes, wherein each of the embedding node partitions corresponds to a position in the ordered list relative to a focus term, is configured to receive an input from an input node, and is configured to generate an output; and

a classifier layer comprising a plurality of neural nodes, each neural node in the classifier layer configured to receive the embedding outputs from the embedding layer, and configured to generate an output corresponding to a probability that a particular linguistic unit is the focus term.

16. The system of claim 15, wherein the linguistic unit is a character or a word.

17. The system of claim 15, wherein the positions relative to a focus term of the embedding node partitions are window positions relative to the focus term or directions relative to the focus term.

18. The system of claim 15, wherein the plurality of neural nodes, input neural node, embedding layer, and classifier layer form a neural network, the neural network trained by performing functions that comprise:

training a first partition of the embedding node partitions by;

removing the focus term from the ordered list of linguistic units;

selecting a partition from each remaining linguistic unit'"'"'s embeddings based on that respective linguistic unit'"'"'s position relative to the focus term; and

updating weights for each neural node in the classifier layer and embedding layer such that the probability of the presence of the focus term is about 100% and other randomly sampled linguistic units are 0%.

19. The system of claim 18, wherein the neural network is trained by performing functions that further comprise:

training a second partition by;

removing the focus term from the ordered list of linguistic units;

selecting a partition from each remaining linguistic unit'"'"'s embedding based on that linguistic unit'"'"'s position relative to the focus term; and

updating the weights to model the probability of the presence of the focus term is about 100% and other randomly sampled linguistic units are 0%,

wherein the steps of training a first partition and training a second partition are performed in parallel.

20. The system of claim 15, wherein particular linguistic units are selected from the ordered list of the plurality of linguistic units and multiple linguistic domains associated with the selected particular linguistic units are modeled into a common vector space, wherein each of the multiple linguistic domains corresponds to a different language.