SYSTEM AND METHOD FOR LEARNING WORD EMBEDDINGS USING NEURAL LANGUAGE MODELS

• US 20150095017A1
• Filed: 11/08/2013
• Published: 04/02/2015
• Est. Priority Date: 09/27/2013
• Status: Abandoned Application

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: MNIH, Andriy, KAVUKCUOGLU, Koray
• Application Number:
  US14/075,166
• Publication Number:
  US 20150095017A1
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/242 Dictionaries
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks

Abstract

A system and method are provided for learning natural language word associations using a neural network architecture. A word dictionary comprises words identified from training data consisting a plurality of sequences of associated words. A neural language model is trained using data samples selected from the training data defining positive examples of word associations, and a statistically small number of negative samples defining negative examples of word associations that are generated from each selected data sample. A system and method of predicting a word association is also provided, using a word association matrix including data defining representations of words in a word dictionary derived from a trained neural language model, whereby a word association query is resolved without applying a word position-dependent weighting.

Claims

1. A method of learning natural language word associations using a neural network architecture, comprising processor implemented steps of:

storing data defining a word dictionary comprising words identified from training data consisting a plurality of sequences of associated words;

selecting a predefined number of data samples from the training data, the selected data samples defining positive examples of word associations;

generating a predefined number of negative samples for each selected data sample, the negative samples defining negative examples of word associations, wherein the number of negative samples generated for each data sample is a statistically small proportion of the number of words in the word dictionary; and

training a neural language model using said data samples and said generated negative samples.

2. The method of claim 1, wherein the negative samples for each selected data sample are generated by replacing one or more words in the data sample with a respective one or more replacement words selected from the word dictionary.

3. The method of claim 2, wherein the one or more replacement words are pseudo-randomly selected from the word dictionary based on frequency of occurrence of words in the training data.

4. The method of claim 1, wherein the number of negative samples generated for each data sample is between 1/10000 and 1/100000 of the number of words in the word dictionary.

5. The method of claim 1, wherein the neural language model is configured to output a word representation for an input word, representative of the association between the input word and other words in the word dictionary.

6. The method of claim 5, further comprising generating a word association matrix comprising a plurality of vectors, each vector defining a representation of a word in the word dictionary output by the trained neural language model.

7. The method of claim 6, further comprising using the word association matrix to resolve a word association query.

8. The method of claim 7, further comprising resolving the query without applying a word position-dependent weighting.

9. The method of claim 1, wherein the neural language model is trained without applying a word position-dependent weighting.

10. The method of claim 1, wherein the data samples each include a target word and a plurality of context words that are associated with the target word, and label data identifying the data sample as a positive example of word association.

11. The method of claim 10, wherein the negative samples each include a target word selected from the word dictionary and the plurality of context words from a data sample, and label data identifying the negative sample as a negative example of word association.

12. The method of claim 1, wherein the training samples and negative samples are fixed-length contexts.

13. The method of claim 1, wherein the neural language model is configured to receive a representation of the target word and representations of the plurality of context words of an input sample, and to output a probability value indicative of the likelihood that the target word is associated with the context words.

14. The method of claim 1, wherein the neural language model is further configured to receive a representation of the target word and representations of at least one context word of an input sample, and to output a probability value indicative of the likelihood that at least one context word is associated with the target word.

15. The method of claim 13, wherein training the neural language model comprises adjusting parameters based on a calculated error value derived from the output probability value and the label associated with the sample.

16. The method of claim 1, further comprising generating the word dictionary based on the training data, wherein the word dictionary includes calculated values of the frequency of occurrence of each word within the training data.

17. The method of claim 1, further comprising normalizing the training data.

18. The method of claim 1, wherein the training data comprises a plurality of sequences of associated words.

25. The method of claim 19, further comprising training the neural language model according to claim 1.

26. The method of claim 25, wherein the training samples each include a target word and a plurality of context words that are associated with the target word, and label data identifying the sample as a positive example of word association.

27. The method of claim 26, wherein the negative samples each include a target word and a plurality of context words that are selected from the word dictionary, and label data identifying the sample as a negative example of word association.

28. The method of claim 27, wherein the data samples and negative samples have fixed-length contexts.

29. The method of claim 27, wherein the negative samples are pseudo-randomly selected based on frequency of occurrence of words in the training data.

30. The method of claim 29, further comprising receiving a representation of the target word and representations of the plurality of context words of an input sample, and outputting a probability value indicative of the likelihood that the target word is associated with the context words.

31. The method of claim 29, further comprising receiving a representation of the target word and representations of at least one context word of an input sample, and outputting a probability value indicative of the likelihood that at least one context word is associated with the target word.

32. The method of claim 30, further comprising training the neural language model by adjusting parameters based on a calculated error value derived from the output probability value and the label associated with the sample.

33. The method of claim 25, further comprising generating the word dictionary based on training data, wherein the word dictionary includes calculated values of the frequency of occurrence of each word within the training data.

34. The method of claim 25, further comprising normalizing the training data.

38. A non-transitive storage medium comprising machine readable instructions stored thereon for causing a computer system to perform a method in accordance with claim 1.

39. The method of claim 14, wherein training the neural language model comprises adjusting parameters based on a calculated error value derived from the output probability value and the label associated with the sample.

40. The method of claim 31, further comprising training the neural language model by adjusting parameters based on a calculated error value derived from the output probability value and the label associated with the sample.

19. A method of predicting a word association between words in a word dictionary, comprising processor implemented steps of:

storing data defining a word association matrix including a plurality of vectors, each vector defining a representation of a word derived from a trained neural language model;

receiving a plurality of query words;

retrieving the associated representations of the query words from the word association matrix;

calculating a candidate representation based on the retrieved representations; and

determining at least one word in the word dictionary that matches the candidate representation, wherein the determination is made based on the word association matrix and without applying a word position-dependent weighting.

20. The method of claim 19, wherein the candidate representation is calculated as the average representation of the retrieved representations.

21. The method of claim 19, wherein calculating the representation comprises subtracting one or more retrieved representations from one or more other retrieved representations.

22. The method of claim 19, further comprising excluding one or more query words from the word dictionary before calculating the candidate representation.

23. The method of claim 19, wherein the trained neural language model is configured to output a word representation for an input word, representative of the association between the input word and other words in the word dictionary.

24. The method of claim 23, further comprising generating the word association matrix from representations of words in the word dictionary output by the trained neural language model.

35. The method of claim 19, wherein the query is an analogy-based word similarity query.

41. A non-transitive storage medium comprising machine readable instructions stored thereon for causing a computer system to perform a method in accordance with claim 19.

36. A system for learning natural language word associations using a neural network architecture, comprising one or more processors configured to:

store data defining a word dictionary comprising words identified from training data consisting of a plurality of sequences of associated words;

select a predefined number of data samples from the training data, the selected data samples defining positive examples of word associations;

generate a predefined number of negative samples for each selected data sample, the negative samples defining negative examples of word associations, wherein the number of negative samples generated for each data sample is a statistically small proportion of the number of wherein the number of negative samples generated for each data sample is a statistically small proportion of the number of words in the word dictionary; and

train a neural language model using said data samples and said generated negative samples.

37. A data processing system for resolving a word similarity query, comprising one or more processors configured to:

store data defining a word association matrix including a plurality of vectors, each vector defining a representation of a word derived from a trained neural language model;

receive a plurality of query words;

retrieve the associated representations of the query words from the word association matrix;

calculate a candidate representation based on the retrieved representations; and

determine at least one word that matches the candidate representation, wherein the determination is made based on the word association matrix and without applying a word position-dependent weighting.