Word hash language model

• US 10,482,875 B2
• Filed: 09/20/2018
• Issued: 11/19/2019
• Est. Priority Date: 12/19/2016
• Status: Active Grant

• Current Assignee: Asapp, Inc.
• Original Assignee: Asapp, Inc.
• Inventors: Henry, Shawn
• Primary Examiner(s): Chawan, Vijay B
• Application Number:
  US16/136,959
• Publication Number:
  US 20190019503A1
• Time in Patent Office: 425 Days
• Field of Search: 704 1- 10, 704251, 704256, 704255, 704257
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/253 Grammatical analysis; Style critique
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G10L 15/14 using statistical models, e.g. Hidden Markov Models [HMMs] G10L15/18 takes precedence
• CPC Class Codes: G10L 15/148 Duration modelling in HMMs, e.g. semi HMM, segmental models or transition probabilities
• CPC Class Codes: G10L 15/16 using artificial neural networks
• CPC Class Codes: G10L 15/197 Probabilistic grammars, e.g. word n-grams
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications

Abstract

A language model may be used in a variety of natural language processing tasks, such as speech recognition, machine translation, sentence completion, part-of-speech tagging, parsing, handwriting recognition, or information retrieval. A natural language processing task may use a vocabulary of words, and a word hash vector may be created for each word in the vocabulary. A sequence of input words may be received, and a hash vector may be obtained for each word in the sequence. A language model may process the hash vectors for the sequence of input words to generate an output hash vector that describes words that are likely to follow the sequence of input words. One or words may then be selected using the output word hash vector and used for a natural language processing task.

Claims

1. A computer-implemented method for implementing a neural network language model, the method comprising:

obtaining a word hash vector for each word of a vocabulary of words;

receiving a first sequence of words for processing by the neural network language model to select a word to follow the first sequence of words;

generating a first sequence of word hash vectors by retrieving a word hash vector for each word of the first sequence of words;

processing the first sequence of word hash vectors with a layer of the neural network language model to compute a first output vector;

quantizing the first output vector to obtain a first output word hash vector;

determining a distance between the first output word hash vector and a first hash vector for a first word in the vocabulary; and

selecting the first word from the vocabulary using the distance between the first output word hash vector and the first hash vector for the first word.

2. The computer-implemented method of claim 1, further comprising using the selected first word to perform at least one of:

speech recognition, machine translation, sentence completion, part-of-speech tagging, parsing, handwriting recognition, or information retrieval.

3. The computer-implemented method of claim 1, further comprising:

generating a second sequence of word hash vectors using the first sequence of word hash vectors and the first hash vector for the selected first word;

processing the second sequence of word hash vectors with the layer of the neural network language model to compute a second output vector;

quantizing the second output vector to obtain a second output word hash vector;

determining a distance between the second output word hash vector and a second hash vector for a second word in the vocabulary; and

selecting the second word from the vocabulary using the distance between the second output word hash vector and the second hash vector for the second word.

4. The computer-implemented method of claim 1, wherein the layer of the neural network language model corresponds to a fixed-size language model.

5. The computer-implemented method of claim 1, wherein each element of the first output word hash vector comprises a Boolean value.

6. The computer-implemented method of claim 1, wherein the first output word hash vector is equal to the first hash vector for the selected first word.

7. The computer-implemented method of claim 1, wherein the distance is a Hamming distance.

8. The computer-implemented method of claim 1, wherein selecting the first word comprises using an inverted index.

9. A system, comprising:

at least one computer comprising at least one processor and at least one memory, the at least one computer configured to;

obtain a word hash vector for each word of a vocabulary of words;

receive a sequence of words;

generate a sequence of word hash vectors by retrieving a word hash vector for each word of the sequence of words;

process the sequence of word hash vectors with a layer of a language model to compute an output vector;

quantize the output vector to obtain an output word hash vector;

determine a distance between the output word hash vector and a first hash vector for a first word in the vocabulary; and

select the first word from the vocabulary using the distance between the output word hash vector and the first hash vector for the first word.

10. The system of claim 9, wherein the at least one computer is further configured to:

compute a word embedding for each word of the vocabulary of words; and

compute a word hash vector for each word using the word embedding for the word.

11. The system of claim 9, wherein the at least one computer is further configured to:

select a plurality of words from the vocabulary using the output word hash vector; and

add the selected plurality of words to a search graph.

12. The system of claim 11, wherein the at least one computer is further configured to:

prune the search graph; and

select at least one best scoring path through the search graph.

13. The system of claim 12, wherein the at least one computer is further configured to:

perform at least one of;

speech recognition, machine translation, sentence completion, part-of-speech tagging, parsing, handwriting recognition, or information retrieval;

wherein the performing utilizes the at least one best scoring path.

14. The system of claim 9, wherein the at least one computer is further configured to quantize the output vector using a linear kernel support vector machine.

15. The system of claim 9, wherein the layer of the language model comprises a recursive language model.

16. The system of claim 9, wherein the at least one computer is further configured to compute a word hash vector by performing a rotation of a word embedding.

17. One or more non-transitory, computer-readable media comprising computer executable instructions that, when executed, cause at least one processor to perform actions comprising:

receiving a sequence of words;

generating a sequence of word hash vectors by obtaining a word hash vector for each word of the sequence of words;

processing the sequence of word hash vectors with a layer of a language model to compute an output vector;

quantizing the output vector to obtain an output word hash vector;

determining a distance between the output word hash vector and a first hash vector for a first word in a vocabulary; and

selecting the first word from the vocabulary using the output word hash vector and the first hash vector for the first word.

18. The one or more non-transitory, computer-readable media of claim 17, wherein the layer of the language model comprises a recurrent neural network with long short-term memory units.

19. The one or more non-transitory, computer-readable media of claim 17, wherein each element of the output word hash vector comprises a Boolean value.

20. The one or more non-transitory, computer-readable media of claim 17, wherein the distance is a Hamming distance.