NEURAL PARAPHRASE GENERATOR

• US 20180329883A1
• Filed: 05/14/2018
• Published: 11/15/2018
• Est. Priority Date: 05/15/2017
• Status: Active Grant

• Current Assignee: Thomson Reuters Enterprise Centre GmbH (The Woodbridge Co. Ltd.)
• Original Assignee: Thomson Reuters Global Resources Unlimited Company (The Woodbridge Co. Ltd.)
• Inventors: Leidner, Jochen L., Plachouras, Vasileios, Petroni, Fabio
• Granted Patent: US 10,733,380 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/2237 Vectors, bitmaps or matrices
• CPC Class Codes: G06F 16/3347 using vector based model
• CPC Class Codes: G06F 17/18 for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06F 40/117 Tagging; Marking up details of markup languages G06F40/143; Designating a block; Setting of attributes style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154
• CPC Class Codes: G06F 40/247 Thesauruses; Synonyms
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/56 Natural language generation
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/082 modifying the architecture, e.g. adding, deleting or silencing nodes or connections
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition

Abstract

A neural paraphrase generator receives a sequence of tuples comprising a source sequence of words, each tuple comprising word data element and structured tag element representing a linguistic attribute about the word data element. An RNN encoder receives a sequence of vectors representing a source sequence of words, and RNN decoder predicts a probability of a target sequence of words representing a target output sentence based on a recurrent state in the decoder. An input composition component includes a word embedding matrix and a tag embedding matrix, and receives and transforms the input sequence of tuples into a sequence of vectors by 1) mapping word data elements to word embedding matrix to generate word vectors, 2) mapping structured tag elements to tag embedding matrix to generate tag vectors, and 3) concatenating word vectors and tag vectors. An output decomposition component outputs a target sequence of tuples representing predicted words and structured tag elements, the probability of each single tuple from the output is predicted based on a recurrent state of the decoder.

Claims

1. In a computer-based system comprising a processor in electrical communication with a memory, the memory adapted to store data and instructions for executing by the processor, a neural paraphrase generator comprising:

an input adapted to receive a sequence of tuples (t=(t₁, . . . , t_n)) comprising a source sequence of words, each tuple (t_i=(w_i,p_i)) comprising a word data element (w_i) and a structured tag element (p_i), the structured tag element representing a linguistic attribute about the word data element;

a recurrent neural network (RNN) comprising an encoder and a decoder, wherein the encoder is adapted to receive a sequence of vectors representing a source sequence of words, and the decoder is adapted to predict a probability of a target sequence of words representing a target output sentence based on a recurrent state in the decoder, a set of previous words and a context vector;

an input composition component connected to the input and comprising a word embedding matrix and a tag embedding matrix, the input composition component being adapted to receive and transform the input sequence of tuples into a sequence of vectors by

     1) mapping the word data elements to the word embedding matrix to generate word vectors,

     2) mapping the structured tag elements to the tag embedding matrix to generate tag vectors, and

     3) respectively concatenating together the word vectors and the tag vectors; and

an output decomposition component connected to the decoder and adapted to output a target sequence of tuples representing predicted words and structured tag elements, wherein the probability of each single tuple from the output target sequence of tuples is predicted based on a recurrent state of the decoder.

2. The neural paraphrase generator of claim 1 further comprising an attention module adapted to generate a custom context vector for each prediction based at least in part on an attention function.

5. The neural paraphrase generator of claim 1 wherein the word embedding matrix and the tag embedding matrix are populated with pretrained values.

6. The neural paraphrase generator of claim 1 wherein the structured tag element is a part-of-speech tag.

7. The neural paraphrase generator of claim 1 further comprising a loss function adapted to learn to predict tuples of words and structured tags and comprising a custom objective function that jointly considers word and structured tag predictions.

8. The neural paraphrase generator of claim 7, wherein the custom objective function J_t is formulated as follows:

9. The neural paraphrase generator of claim 1, wherein the RNN includes Long Short Term Memory (LSTM) cells.

10. The neural paraphrase generator of claim 1 further comprising at least one additional linguistic attribute in addition to the structured tag.

3. The neural paraphrase generator of claim 3 wherein the attention module is further adapted to generate an attentional vector by concatenating the decoder state and the context vector.

4. The neural paraphrase generator of claim 3 wherein the attentional vector is passed through a softmax layer to produce a probability distribution over a word vocabulary data set.