DEEP REINFORCED MODEL FOR ABSTRACTIVE SUMMARIZATION

• US 20190311002A1
• Filed: 06/25/2019
• Published: 10/10/2019
• Est. Priority Date: 04/14/2017
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Paulus, Romain
• Granted Patent: US 10,521,465 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/345 Summarisation for human users
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

A system for text summarization includes an encoder for encoding input tokens of a document and a decoder for emitting summary tokens which summarize the document based on the encoded input tokens. At each iteration the decoder generates attention scores between a current hidden state of the decoder and previous hidden states of the decoder, generates a current decoder context from the attention scores and the previous hidden states of the decoder, and selects a next summary token based on the current decoder context and a current encoder context of the encoder. The attention scores penalize candidate summary tokens having high attention scores in previous iterations. In some embodiments, the attention scores include an attention score for each of the previous hidden states of the decoder. In some embodiments, the selection of the next summary token prevents emission of repeated summary phrases in a summary of the document.

Claims

1. A text summarization system comprising:

an encoder for encoding input tokens of a document to be summarized; and

a decoder for emitting summary tokens which summarize the document based on the encoded input tokens, wherein at each iteration the decoder;

generates attention scores between a current hidden state of the decoder and previous hidden states of the decoder;

generates a current decoder context from the attention scores and the previous hidden states of the decoder; and

selects a next summary token based on the current decoder context and a current encoder context of the encoder;

wherein the attention scores penalize candidate summary tokens having high attention scores in previous iterations.

2. The text summarization system of claim 1, wherein the attention scores include an attention score for each of the previous hidden states of the decoder.

3. The text summarization system of claim 1, wherein at each iteration, the decoder further normalizes the attention scores.

4. The text summarization system of claim 3, wherein the decoder normalizes the attention scores using a softmax layer.

5. The text summarization system of claim 1, wherein the current decoder context is a convex combination of the attention scores and the previous hidden states of the decoder.

6. The text summarization system of claim 1, wherein the selection of the next summary token prevents emission of repeated summary phrases in a summary of the document.

7. The text summarization system of claim 1, wherein the next summary token is further based on the current hidden state of the decoder.

8. A method for summarizing text, the method comprising:

receiving a document to be summarized;

encoding, using an encoder, input tokens of the document;

generating, using a decoder, attention scores between a current hidden state of the decoder and previous hidden states of the decoder;

generating, using the decoder, a current decoder context from the attention scores and the previous hidden states of the decoder; and

selecting, using the decoder, a next summary token based on the current decoder context and a current encoder context of the encoder;

wherein;

the next summary token from each iteration summarizes the document; and

the attention scores penalize candidate summary tokens having high attention scores in previous iterations.

9. The method of claim 8, wherein the attention scores include an attention score for each of the previous hidden states of the decoder.

10. The method of claim 8, further comprising, at each iteration, normalizing the attention scores.

11. The method of claim 10, wherein normalizing the attention scores comprises using a softmax layer.

12. The method of claim 8, wherein generating the current decoder context comprises generating a convex combination of the attention scores and the previous hidden states of the decoder.

13. The method of claim 8, wherein the selecting of the next summary token prevents emission of repeated summary phrases in a summary of the document.

14. The method of claim 8, wherein selecting the next summary token is further based on the current hidden state of the decoder.

15. A tangible non-transitory computer readable storage medium impressed with computer program instructions that, when executed on a processor, implement a method comprising:

receiving a document to be summarized;

encoding, using an encoder, input tokens of the document;

generating, using a decoder, attention scores between a current hidden state of the decoder and previous hidden states of the decoder;

generating, using the decoder, a current decoder context from the attention scores and the previous hidden states of the decoder; and

selecting, using the decoder, a next summary token based on the current decoder context and a current encoder context of the encoder;

wherein;

the next summary token from each iteration summarizes the document; and

the attention scores penalize candidate summary tokens having high attention scores in previous iterations.

16. The tangible non-transitory computer readable storage medium of claim 15, wherein the attention scores include an attention score for each of the previous hidden states of the decoder.

17. The tangible non-transitory computer readable storage medium of claim 15, further comprising, at each iteration, normalizing the attention scores.

18. The tangible non-transitory computer readable storage medium of claim 15, wherein generating the current decoder context comprises generating a convex combination of the attention scores and the previous hidden states of the decoder.

19. The tangible non-transitory computer readable storage medium of claim 15, wherein the selecting of the next summary token prevents emission of repeated summary phrases in a summary of the document.

20. The tangible non-transitory computer readable storage medium of claim 15, wherein selecting the next summary token is further based on the current hidden state of the decoder.