Generation of a grammatically diverse test set for deep question answering systems

• US 10,394,950 B2
• Filed: 08/22/2016
• Issued: 08/27/2019
• Est. Priority Date: 08/22/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Bethard, Sean L., Katz, Edward G., Phipps, Christopher
• Primary Examiner(s): Spooner, Lamont M
• Application Number:
  US15/242,807
• Publication Number:
  US 20180052818A1
• Time in Patent Office: 1,100 Days
• Field of Search: 704 1, 704 9, 704 10
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

A grammatically diverse test set of natural language sentences for a deep question answering system is provided by analyzing a given sentence to characterize its syntactical classification, and adding the sentence to the test set if its classification is sufficiently different from other sentences already in the test set. A particular sentence may be selected for inclusion according to a desired syntactic distribution. Multiple sentences having the exact same classification may be allowed subject to a maximum number of such sentences. The test set is adapted to an element of interest by characterizing each syntactical classification relative to the element of interest. The analysis derives a parse tree, identifies a particular node of the tree corresponding to the element of interest, and extracts syntactic information by traversing the tree starting at the particular node and ending at the root node of the tree according to different traversal schemes.

Claims

1. A method of providing a test set of natural language sentences for a deep question answering system comprising:

receiving a plurality of natural language sentences in computer-readable form wherein each natural language sentence has a composition of certain words, by executing first instructions in a computer system;

analyzing the natural language sentences to characterize different syntactical classifications of the sentences, by executing second instructions in the computer system;

selecting a subset of the natural language sentences according to a desired syntactic distribution based on the different syntactical classifications wherein each natural language sentence in the subset is unchanged from its composition, by executing third instructions in the computer system;

storing the selected subset as the test set, by executing fourth instructions in the computer system;

applying the test set to the deep question answering system to train the deep question answering system, by executing fifth instructions in the computer system;

receiving a query from a user, by executing sixth instructions in the computer system;

using the deep question answering system to generate an answer to the query, by executing seventh instructions in the computer system; and

presenting the answer to the user, by executing eighth instructions in the computer system.

2. The method of claim 1 wherein said selecting includes:

determining that a syntactical classification for a candidate natural language sentence is different from syntactical classifications of other natural language sentences already in the subset according to at least one predetermined similarity criterion; and

responsive to said determining, adding the candidate natural language sentence to the subset.

3. The method of claim 1 wherein said selecting provides a predetermined minimum number of differently classified sentences in the subset.

4. The method of claim 1 wherein said selecting allows multiple sentences having the exact same syntactical classification in the subset subject to a predetermined maximum number.

5. The method of claim 1 wherein the test set is adapted to an element of interest by characterizing each syntactical classification relative to a location of the element of interest in a respective one of the natural language sentences.

6. The method of claim 5 wherein said analyzing includes:

parsing a given one of the natural language sentences to derive a parse tree having a plurality of nodes;

identifying a particular one of the nodes that corresponds to the element of interest;

extracting syntactic information from the parse tree relative to particular node corresponding to the element of interest; and

recording the syntactic information as the classification.

7. A computer system comprising:

one or more processors which process program instructions;

a memory device connected to said one or more processors; and

program instructions residing in said memory device for providing a test set of natural language sentences for a deep question answering system by receiving a plurality of natural language sentences wherein each natural language sentence has a composition of certain words, analyzing the natural language sentences to characterize different syntactical classifications of the sentences, selecting a subset of the natural language sentences according to a desired syntactic distribution based on the different syntactical classifications wherein each natural language sentence in the subset is unchanged from its composition, storing the selected subset as the test set, applying the test set to the deep question answering system to train the deep question answering system, receiving a query from a user, using the deep question answering system to generate an answer to the query, and presenting the answer to the user.

8. The computer system of claim 7 wherein said program instructions further determine that a syntactical classification for a candidate natural language sentence is different from syntactical classifications of other natural language sentences already in the subset according to at least one predetermined similarity criterion, and responsively adds the candidate natural language sentence to the subset.

9. The computer system of claim 7 wherein the selecting provides a predetermined minimum number of differently classified sentences in the subset.

10. The computer system of claim 7 wherein the selecting allows multiple sentences having the exact same syntactical classification in the subset subject to a predetermined maximum number.

11. The computer system of claim 7 wherein the test set is adapted to an element of interest by characterizing each syntactical classification relative to a location of the element of interest in a respective one of the natural language sentences.

12. The computer system of claim 11 wherein the analyzing includes:

parsing a given one of the natural language sentences to derive a parse tree having a plurality of nodes;

identifying a particular one of the nodes that corresponds to the element of interest;

extracting syntactic information from the parse tree relative to particular node corresponding to the element of interest; and

recording the syntactic information as the classification.

13. A computer program product comprising:

a computer readable storage medium; and

program instructions residing in said storage medium for providing a test set of natural language sentences for a deep question answering system by receiving a plurality of natural language sentences wherein each natural language sentence has a composition of certain words, analyzing the natural language sentences to characterize different syntactical classifications of the sentences, selecting a subset of the natural language sentences according to a desired syntactic distribution based on the different syntactical classifications wherein each natural language sentence in the subset is unchanged from its composition, storing the selected subset as the test set, applying the test set to the deep question answering system to train the deep question answering system, receiving a query from a user, using the deep question answering system to generate an answer to the query, and presenting the answer to the user.

14. The computer program product of claim 13 wherein said program instructions further determine that a syntactical classification for a candidate natural language sentence is different from syntactical classifications of other natural language sentences already in the subset according to at least one predetermined similarity criterion, and responsively adds the candidate natural language sentence to the subset.

15. The computer program product of claim 13 wherein the selecting provides a predetermined minimum number of differently classified sentences in the sub set.

16. The computer program product of claim 13 wherein the selecting allows multiple sentences having the exact same syntactical classification in the subset subject to a predetermined maximum number.

17. The computer program product of claim 13 wherein the test set is adapted to an element of interest by characterizing each syntactical classification relative to a location of the element of interest in a respective one of the natural language sentences.

18. The computer program product of claim 17 wherein the analyzing includes:

parsing a given one of the natural language sentences to derive a parse tree having a plurality of nodes;

identifying a particular one of the nodes that corresponds to the element of interest;

extracting syntactic information from the parse tree relative to particular node corresponding to the element of interest; and

recording the syntactic information as the classification.