Crowdsource Reasoning Process to Facilitate Question Answering

• US 20160196497A1
• Filed: 01/07/2015
• Published: 07/07/2016
• Est. Priority Date: 01/07/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Allen, Corville O., Will, Scott A., Woods, Eric
• Granted Patent: US 9,373,086 B1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24522 Translation of natural language queries to structured queries
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 5/046 Forward inferencing; Production systems

Abstract

Mechanisms are provided, in association with a Question and Answer (QA) system, for implementing corpus identified reasoning when evaluating questions. An input question is received and an answer to the input question and supporting evidence for the answer are determined based on an evaluation of a corpus of information. The supporting evidence is analyzed to identify reasoning criteria specifying a reason as to why the answer is considered to be a correct answer by a provider of the supporting evidence. Rules are generated based on the identified reasoning criteria with the rules correlating the answer with the identified reasoning criteria and features of the input question. The rules are stored as entries in a reasoning data structure a subsequent question is processed based on the reasoning data structure.

Claims

1. A method, in a data processing system having a processor and a memory and which implements a Question and Answer (QA) system, the method comprising:

receiving, by the QA system, an input question;

analyzing, by the QA system, the input question to extract one or more semantic or syntactic features of the input question;

determining, by the QA system, an answer to the input question and supporting evidence for the answer based on an evaluation of a corpus of information;

analyzing, by the QA system, natural language content of the supporting evidence to identify reasoning key words or reasoning key phrases and associated reasoning criteria, present in the natural language content of the supporting evidence, the reasoning criteria specifying a reason as to why the answer is considered to be a correct answer by a provider of the supporting evidence;

generating, by the QA system, one or more rules based on the identified reasoning criteria, wherein the one or more rules correlate the answer with the identified reasoning criteria and the one or more semantic or syntactic features of the input question;

storing, by the QA system, the one or more rules as one or more entries in a reasoning data structure; and

processing, by the QA system, a subsequent question based on the reasoning data structure.

2. The method of claim 1, wherein the corpus of information comprises natural language crowdsource information and wherein the provider of the supporting evidence is a crowdsourcing source.

3. The method of claim 1, wherein analyzing the natural language content of the supporting evidence to identify reasoning key words or reasoning key phrases and associated reasoning criteria comprises performing a search of text of the supporting evidence for reasoning key terms or reasoning key phrases indicative of a natural language reasoning statement in the text of the supporting evidence linking at least one reasoning criterion to the answer.

4. The method of claim 3, wherein performing a search of the text of the supporting evidence comprises generating a logical parse tree and evaluating nodes of the logical parse tree to determine which nodes of the logical parse tree comprise the reasoning key terms or reasoning key phrases indicative of a natural language reasoning statement.

5. The method of claim 3, wherein analyzing the natural language content of the supporting evidence to identify reasoning key words or reasoning key phrases and associated reasoning criteria further comprises, for each instance of a reasoning key term or reasoning key phrase found in the text of the supporting evidence:

identifying reasoning criteria specified in association with the reasoning key term or reasoning key phrase; and

generating a distribution of reasoning criteria indicative of reasoning for the answer to the input question.

6. The method of claim 5, wherein generating one or more rules based on the identified reasoning criteria comprises:

selecting, based on the distribution of reasoning criteria, one or more reasoning criteria that are most often identified in association with the answer to the input question; and

generating the one or more rules based on the selected one or more reasoning criteria.

8. The method of claim 1, wherein processing the subsequent question based on the reasoning data structure comprises:

searching the reasoning data structure based on features of the subsequent question to identify a matching entry in the reasoning data structure having one or more semantic or syntactic features similar to the features of the subsequent question;

retrieving a rule associated with the matching entry; and

applying the rule to evidence identified in the corpus of information corresponding to a focus of the subsequent question to determine if the identified reasoning criteria of the rule are satisfied by the evidence identified in the corpus of information corresponding to the focus of the subsequent question.

9. The method of claim 8, wherein processing the subsequent question based on the reasoning data structure further comprises:

in response to the rule being satisfied, returning an answer corresponding to the matching entry as an answer to the subsequent question; and

in response to the rule not being satisfied, processing the subsequent question by the QA system to generate another answer to the subsequent question.

10. The method of claim 1, further comprising, in response to generating one or more rules:

generating one or more additional natural language statements specifying reasoning of the one or more rules in association with the answer; and

inserting the one or more additional natural language statements into the corpus of information.

7. (canceled)

11. A computer program product comprising a non-transitory computer readable medium having a computer readable program stored therein, wherein the computer readable program, when executed on a data processing system implementing a Question and Answer (QA) system, causes the data processing system to:

receive, by the QA system, an input question;

analyze, by the QA system, the input question to extract one or more semantic or syntactic features of the input question;

determine, by the QA system, an answer to the input question and supporting evidence for the answer based on an evaluation of a corpus of information;

analyze, by the QA system, natural language content of the supporting evidence to identify reasoning key words or reasoning key phrases and associated reasoning criteria, present in the natural language content of the supporting evidence, the reasoning criteria specifying a reason as to why the answer is considered to be a correct answer by a provider of the supporting evidence;

generate, by the QA system, one or more rules based on the identified reasoning criteria, wherein the one or more rules correlate the answer with the identified reasoning criteria and the one or more semantic or syntactic features of the input question;

store, by the QA system, the one or more rules as one or more entries in a reasoning data structure; and

process, by the QA system, a subsequent question based on the reasoning data structure.

12. The computer program product of claim 11, wherein the corpus of information comprises natural language crowdsource information and wherein the provider of the supporting evidence is a crowdsourcing source.

13. The computer program product of claim 11, wherein the computer readable program further causes the data processing system to analyze the natural language content of the supporting evidence to identify reasoning key words or reasoning key phrases and associated reasoning criteria at least by performing a search of text of the supporting evidence for reasoning key terms or reasoning key phrases indicative of a natural language reasoning statement in the text of the supporting evidence linking at least one reasoning criterion to the answer.

14. The computer program product of claim 13, wherein the computer readable program further causes the data processing system to perform a search of the text of the supporting evidence at least by generating a logical parse tree and evaluating nodes of the logical parse tree to determine which nodes of the logical parse tree comprise the reasoning key terms or reasoning key phrases indicative of a natural language reasoning statement.

15. The computer program product of claim 13, wherein the computer readable program further causes the data processing system to analyze the natural language content of the supporting evidence to identify reasoning key words or reasoning key phrases and associated reasoning criteria further at least by, for each instance of a reasoning key term or reasoning key phrase found in the text of the supporting evidence:

identifying reasoning criteria specified in association with the reasoning key term or reasoning key phrase; and

generating a distribution of reasoning criteria indicative of reasoning for the answer to the input question.

16. The computer program product of claim 15, wherein the computer readable program further causes the data processing system to generate one or more rules based on the identified reasoning criteria at least by:

selecting, based on the distribution of reasoning criteria, one or more reasoning criteria that are most often identified in association with the answer to the input question; and

generating the one or more rules based on the selected one or more reasoning criteria.

18. The computer program product of claim 11, wherein the computer readable program further causes the data processing system to process the subsequent question based on the reasoning data structure at least by:

searching the reasoning data structure based on features of the subsequent question to identify a matching entry in the reasoning data structure having one or more semantic or syntactic features similar to the features of the subsequent question;

retrieving a rule associated with the matching entry; and

applying the rule to evidence identified in the corpus of information corresponding to a focus of the subsequent question to determine if the identified reasoning criteria of the rule are satisfied by the evidence identified in the corpus of information corresponding to the focus of the subsequent question.

19. The computer program product of claim 18, wherein the computer readable program further causes the data processing system to process the subsequent question based on the reasoning data structure at least by:

in response to the rule being satisfied, returning an answer corresponding to the matching entry as an answer to the subsequent question; and

in response to the rule not being satisfied, processing the subsequent question by the QA system to generate another answer to the subsequent question.

20. The computer program product of claim 11, wherein the computer readable program further causes the data processing system to, in response to generating one or more rules:

generate one or more additional natural language statements specifying reasoning of the one or more rules in association with the answer; and

insert the one or more additional natural language statements into the corpus of information.

17. (canceled)

21. An apparatus comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

receive an input question;

analyze the input question to extract one or more semantic or syntactic features of the input question;

determine an answer to the input question and supporting evidence for the answer based on an evaluation of a corpus of information;

analyze natural language content of the supporting evidence to identify reasoning key words or reasoning key phrases and associated reasoning criteria, present in the natural language content of the supporting evidence, the reasoning criteria specifying a reason as to why the answer is considered to be a correct answer by a provider of the supporting evidence;

generate one or more rules based on the identified reasoning criteria, wherein the one or more rules correlate the answer with the identified reasoning criteria and the one or more semantic or syntactic features of the input question;

store the one or more rules as one or more entries in a reasoning data structure; and

process a subsequent question based on the reasoning data structure.