Sentence compression using recurrent neural networks

• US 10,229,111 B1
• Filed: 02/03/2017
• Issued: 03/12/2019
• Est. Priority Date: 02/03/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Filippova, Ekaterina, Alfonseca, Enrique, Colmenares Rojas, Carlos Alberto, Kaiser, Lukasz Mieczyslaw, Vinyals, Oriol
• Primary Examiner(s): Leland, III, Edwin S
• Application Number:
  US15/423,852
• Time in Patent Office: 767 Days
• Field of Search: 704 9
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/42 Data-driven translation
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks

Abstract

Methods, systems, apparatus, including computer programs encoded on computer storage medium, for generating a sentence summary. In one aspect, the method includes actions of tokenizing the sentence into a plurality of tokens, processing data representative of each token in a first order using an LSTM neural network to initialize an internal state of a second LSTM neural network, processing data representative of each token in a second order using the second LSTM neural network, comprising, for each token in the sentence: processing the data representative of the token using the second LSTM neural network in accordance with a current internal state of the second LSTM neural network to (i) generate an LSTM output for the token, and (ii) to update the current internal state of the second LSTM neural network, and generating the summarized version of the sentence using the outputs of the second LSTM neural network for the tokens.

Claims

1. A method of generating a summarized version of a sentence, the method comprising:

tokenizing the sentence into a plurality of tokens;

processing data representative of each token in a first order using a first long short-term memory (LSTM) neural network to initialize an internal state of a second LSTM neural network;

processing data representative of each token in a second order using the second long short-term memory (LSTM) neural network, comprising, for data representative of each token in the sentence;

processing the data representative of the token using the second LSTM neural network in accordance with a current internal state of the second LSTM neural network to (i) generate an LSTM output for the token, wherein the LSTM output indicates whether the token should be included in the summarized version of the sentence, and (ii) to update the current internal state of the second LSTM neural network; and

generating the summarized version of the sentence using the LSTM outputs for the tokens.

2. The method of claim 1, wherein each of the plurality of tokens is a word or a punctuation mark.

3. The method of claim 1, wherein the LSTM output for each token includes:

data indicating that a word corresponding to the token should be included in the summarized version of the sentence;

or

data indicating that a word corresponding to the token should not be included in the summarized version of the sentence.

4. The method of claim 3, further comprising:

for each of the plurality of tokens, associating the LSTM output for the token with a word corresponding to the token.

5. The method of claim 4, wherein generating the summarized version of the sentence using the LSTM outputs for the tokens further comprises:

selecting only a subset of words that correspond to the tokens that are each associated with data indicating that the word corresponding to the token should be included in the summarized version of the sentence.

6. The method of claim 5, wherein the selected subset of words that correspond to the tokens used to generate the summarized version of the sentence remain in the same order that each respective word appeared in the sentence prior to tokenization.

7. The method of claim 1, wherein the first order of tokens is an order of tokens corresponding to words in the sentence that is the reverse of the order the words corresponding to the tokens appeared in the sentence.

8. The method of claim 1, wherein the second order of tokens is an order of tokens corresponding to words in the sentence that is the same order that the words corresponding to the tokens appeared in the sentence.

9. The method of claim 1, wherein processing data representative of each token in a second order using the second long short-term memory (LSTM) neural network, further comprises, for each token in the sentence:

processing data indicative of whether (1) the preceding token has been processed and a word corresponding to the preceding token should be included in the summarized sentence, or (2) the preceding token has been processed and a word corresponding to the preceding token should be removed from the summarized sentence.

10. The method of claim 1, the method further comprising:

parsing the sentence in order to generate a dependency tree,

wherein processing data representative of each token in a second order using the second long short-term memory (LSTM) neural network, further comprises, for each token in the sentence;

processing the data representative of the token and a corresponding parent token in the dependency tree.

11. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by one or more computers, to cause the one or more computers to perform the operations comprising;

tokenizing the sentence into a plurality of tokens;

processing data representative of each token in a first order using a first long short-term memory (LSTM) neural network to initialize an internal state of a second LSTM neural network;

processing data representative of each token in a second order using the second long short-term memory (LSTM) neural network, comprising, for data representative of each token in the sentence;

processing the data representative of the token using the second LSTM neural network in accordance with a current internal state of the second LSTM neural network to (i) generate an LSTM output for the token, wherein the LSTM output indicates whether the token should be included in the summarized version of the sentence, and (ii) to update the current internal state of the second LSTM neural network; and

generating the summarized version of the sentence using the LSTM outputs for the tokens.

12. The system of claim 11, wherein the LSTM output for each token includes:

data indicating that a word corresponding to the token should be included in the summarized version of the sentence;

or

data indicating that a word corresponding to the token should not be included in the summarized version of the sentence.

13. The system of claim 12, further comprising:

for each of the plurality of tokens, associating the LSTM output for the token with a word corresponding to the token.

14. The system of claim 13, wherein generating the summarized version of the sentence using the LSTM outputs for the tokens further comprises:

selecting only a subset of words that correspond to the tokens that are each associated with data indicating that the word corresponding to the token should be included in the summarized version of the sentence.

15. The system of claim 14, wherein the selected subset of words that correspond to the tokens used to generate the summarized version of the sentence remain in the same order that each respective word appeared in the sentence prior to tokenization.

16. A non-transitory computer-readable medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operations comprising:

tokenizing the sentence into a plurality of tokens;

processing data representative of each token in a first order using a first long short-term memory (LSTM) neural network to initialize an internal state of a second LSTM neural network;

processing data representative of each token in a second order using the second long short-term memory (LSTM) neural network, comprising, for data representative of each token in the sentence;

processing the data representative of the token using the second LSTM neural network in accordance with a current internal state of the second LSTM neural network to (i) generate an LSTM output for the token, wherein the LSTM output indicates whether the token should be included in the summarized version of the sentence, and (ii) to update the current internal state of the second LSTM neural network; and

generating the summarized version of the sentence using the LSTM outputs for the tokens.

17. The non-transitory computer-readable medium of claim 16, wherein the LSTM output for each token includes:

data indicating that a word corresponding to the token should be included in the summarized version of the sentence;

or

data indicating that a word corresponding to the token should not be included in the summarized version of the sentence.

18. The non-transitory computer-readable medium of claim 17, further comprising:

for each of the plurality of tokens, associating the LSTM output for the token with a word corresponding to the token.

19. The non-transitory computer-readable medium of claim 18, wherein generating the summarized version of the sentence using the LSTM outputs for the tokens further comprises:

selecting only a subset of words that correspond to the tokens that are each associated with data indicating that the word corresponding to the token should be included in the summarized version of the sentence.

20. The non-transitory computer-readable medium of claim 19, wherein the selected subset of words that correspond to the tokens used to generate the summarized version of the sentence remain in the same order that each respective word appeared in the sentence prior to tokenization.