Semantic parsing using deep neural networks for predicting canonical forms

• US 9,858,263 B2
• Filed: 05/05/2016
• Issued: 01/02/2018
• Est. Priority Date: 05/05/2016
• Status: Active Grant

• Current Assignee: Centre National De La Recherche Scientifique, Conduent Business Services, LLC (Conduent, Inc.)
• Original Assignee: Centre National De La Recherche Scientifique, Conduent Business Services, LLC (Conduent, Inc.)
• Inventors: Xiao, Chunyang, Dymetman, Marc, Gardent, Claire
• Primary Examiner(s): ROBERTS, SHAUN A
• Application Number:
  US15/147,222
• Publication Number:
  US 20170323636A1
• Time in Patent Office: 607 Days
• Field of Search: 704 9
• CPC Class Codes: G06F 16/332 Query formulation
• CPC Class Codes: G06F 40/274 Converting codes to words; Guess-ahead of partial word inputs
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G10L 15/16 using artificial neural networks

Abstract

A method for predicting a canonical form for an input text sequence includes predicting the canonical form with a neural network model. The model includes an encoder, which generates a first representation of the input text sequence based on a representation of n-grams in the text sequence and a second representation of the input text sequence generated by a first neural network. The model also includes a decoder which sequentially predicts terms of the canonical form based on the first and second representations and a predicted prefix of the canonical form. The canonical form can be used, for example, to query a knowledge base or to generate a next utterance in a discourse.

Claims

1. A method comprising:

providing a neural network model which has been trained to predict a canonical form, containing a sequence of words, for an input text sequence, containing a sequence of words, the neural network model comprising;

an encoder which generates a first representation of the input text sequence based on a representation of n-grams in the text sequence, the encoder including a first neural network which reads the input text sequence and generates a second representation of the input text sequence, and

a decoder which sequentially predicts a next term of the canonical form, based on the first and second representations and a predicted prefix of the canonical form, the prefix containing a sequence of at least one word;

receiving an input text sequence, containing a sequence of words;

with a processor, predicting a canonical form, containing a sequence of words, for the input text sequence with the trained neural network model; and

outputting information based on the predicted canonical form.

2. The method of claim 1, further comprising:

parsing the predicting canonical form to generate a logical form; and

generating a query based on the logical form.

3. The method of claim 2, further comprising:

querying a knowledge base with the query; and

retrieving a response to the query from the knowledge base, the output information being based on the response.

4. The method of claim 1, further comprising:

training the neural network model on training data, the training data comprising training pairs, each training pair including a canonical form and a corresponding text sequence.

5. The method of claim 4, further comprising:

generating the training pairs comprising collecting text sequences for a set of canonical forms using crowdsourcing.

6. The method of claim 1, wherein the first neural network comprises a first recurrent neural network.

7. The method of claim 6, wherein the first recurrent neural network is a first long short-term memory neural network.

8. The method of claim 1, wherein the decoder comprises a second neural network which sequentially predicts the prefix of the canonical form.

9. The method of claim 8, wherein the second neural network comprises a second long short-term memory neural network.

10. The method of claim 8, wherein the decoder comprises a multilayer perceptron which sequentially generates a next term of the canonical form based on the first and second representations and the predicted prefix of the canonical form.

11. The method of claim 1, wherein the next term of the canonical form is estimated as:

P(y_t|u_l,u_b,c_l,t-1)=s′

(W′

₂(s′

(W′

₁(z)))) 

 

(3),

where z is a combined representation generated from the first and second representations and the predicted prefix c_l,t-1 of the canonical form, W′

₁, W′

₂ are parameter matrices that are learned during training, and s′

is a non-linear activation function.

12. The method of claim 1, wherein the neural network model includes at least one embedding layer which converts words of the input text sequence to vectorial representations.

13. A system comprising memory which stores instructions for performing the method of claim 1 and a processor in communication with the memory which executes the instructions.

14. A computer program product comprising a non-transitory storage medium storing instructions which, when executed by a computer, perform the method of claim 1.

15. A method comprising:

providing a neural network model which has been trained to predict a canonical form for an input text sequence, the neural network model comprising;

an encoder which comprises a first multilayer perceptron which generates a first representation of the input text sequence based on a representation of n-grams in the text sequence, and

a first recurrent neural network which reads the input text sequence and generates a second representation of the input text sequence, and

a decoder which sequentially predicts a next term of the canonical form, based on the first and second representations and a predicted prefix of the canonical form;

receiving an input text sequence;

with a processor, predicting a canonical form for the input text sequence with the trained neural network model; and

outputting information based on the predicted canonical form.

16. A system comprising:

memory which stores a neural network model which has been trained to predict a canonical form for an input text sequence, the neural network model comprising;

an encoder which generates a first representation of the input text sequence based on a representation of n-grams in the text sequence and a second representation of the input text sequence generated by a first neural network, and

a decoder which sequentially predicts terms of the canonical form based on the first and second representations and a predicted prefix of the canonical form;

a prediction component which predicts a canonical form for an input text sequence with the trained neural network model;

a semantic parser which generates a logical form based on the predicted canonical form;

an output component which outputs information based on the predicted canonical form; and

a processor which implements the prediction component and the output component.

17. The system of claim 16, further comprising a learning component which trains the neural network model on training data, the training data comprising training pairs, each training pair including a canonical form and a corresponding text sequence.

18. The system of claim 17, further comprising a querying component which queries a knowledge base with a query based on the logical form for retrieving responsive information.

19. A method for predicting a canonical form comprising:

providing training data, the training data comprising a collection of training pairs, each training pair in the collection including a canonical form, containing a sequence of words, and a corresponding text sequence, containing a sequence of words;

with the training data, training a neural network model to predict a canonical form, containing a sequence of words, for an input text sequence, the neural network model comprising;

an encoder which generates a first representation of the input text sequence based on a representation of n-grams in the text sequence and a second representation of the input text sequence generated by a first neural network, and

a decoder which sequentially predicts terms of the canonical form based on the first and second representations and a predicted prefix of the canonical form, each of the terms of the canonical form including at least one word;

receiving an input text sequence, containing a sequence of words;

with a processor, predicting a canonical form, containing a sequence of words, for the input text sequence with the trained neural network model; and

outputting information based on the predicted canonical form.