Justifying Passage Machine Learning for Question and Answer Systems

• US 20140297571A1
• Filed: 03/28/2014
• Published: 10/02/2014
• Est. Priority Date: 03/29/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Whitley, Michael D., Yates, Robert L., Beamon, Bridget B.
• Granted Patent: US 9,613,317 B2
• Time in Patent Office: Days
• US Class Current: 706/12
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/61 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition

Abstract

Mechanisms are provided for generating an answer to an input question. An input question is received and a set of candidate answers is generated along with, for each candidate answer in the set of candidate answers, a corresponding selection of one or more selected evidence portions from a corpus of information providing evidence in support of the candidate answer being a correct answer for the input question. The candidate answers are ranked based on an application of a justifying passage model (JPM) to the selected evidence portions for each of the candidate answers in the set of candidate answers. The JPM identifies whether a candidate answer is justified by a selected evidence passage corresponding to the candidate answer. A candidate answer is output as the correct answer for the input question based on the ranking of the candidate answers.

Claims

1. A method, in a data processing system comprising a processor and a memory configured to implement a question and answer system (QA), for generating answers to an input question, comprising:

receiving, in the data processing system, the input question;

generating, by the data processing system, a set of candidate answers for the input question and, for each candidate answer in the set of candidate answers, a corresponding selection of one or more selected evidence portions from a corpus of information providing evidence in support of the candidate answer being a correct answer for the input question;

ranking, by the data processing system, the candidate answers based on an application of a justifying passage model (JPM) to the selected evidence portions for each of the candidate answers in the set of candidate answers, wherein the JPM identifies whether a candidate answer is justified by a selected evidence passage corresponding to the candidate answer; and

outputting, by the data processing system, a candidate answer in the set of candidate answers as the correct answer for the input question based on the ranking of the candidate answers.

2. The method of claim 1, wherein the JPM is trained based on a JPM ground truth data structure that comprises a justification indicator, for each question-answer-evidence passage (QAP) triplet in a plurality of QAP triplets of the JPM ground truth data structure, indicating whether an answer in the QAP triplet is justified, by the evidence passage of the QAP triplet, as being a correct answer for a question in the QAP triplet.

3. The method of claim 2, wherein the justification indicator for each QAP triplet in the plurality of QAP triplets is set to a justification value by a subject matter expert that evaluates the QAP triplet.

4. The method of claim 1, wherein application of the JPM to the selected evidence portions comprises applying one or more weights of the JPM to evidence scores associated with the selected evidence portions to thereby generate a confidence score associated with the candidate answer, wherein the one or more weights of the JPM indicate a degree of relevance of the selected evidence portions to the input question.

5. The method of claim 1, wherein application of the JPM to the selected evidence portions comprises:

ranking evidence portions in the selected evidence portions according to evidence scores generated based on the application of the JPM to the selected evidence portions; and

selecting a subset of the ranked evidence portions in the selected evidence portions, for use in ranking the candidate answers, based on one or more filter criteria.

6. The method of claim 1, wherein application of the JPM to the selected evidence portions comprises, for each candidate answer in the set of candidate answers:

weighting an evidence score associated with an evidence portion associated with the candidate answer based on a weight value, in the JPM, associated with the evidence portion; and

calculating a weighted confidence score for the candidate answer based on a combination of weighted evidence scores for evidence portions associated with the candidate answer.

7. The method of claim 1, wherein application of the JPM to the selected evidence portions comprises, for each candidate answer:

performing first context dependent scoring of the evidence portions corresponding to the candidate answer, wherein the first context dependent scoring scores the candidate answer based on a correlation between the candidate answer and the evidence portions corresponding to the candidate answer; and

performing context independent scoring of the candidate answer, wherein the context independent scoring of the candidate answer scores the candidate answer independently of the evidence portions corresponding to the candidate answer.

8. The method of claim 7, wherein application of the JPM to the selected evidence portions comprises:

weighting, based on the JPM, context independent scores associated with the candidate answers to generate weighted context independent scores;

filtering, based on the weighted context independent scores of the candidate answers, candidate answers from the set of candidate answers to generate a subset of candidate answers; and

performing, for each candidate answer in the subset of candidate answers, additional evidence portion retrieval operations for retrieving additional evidence portions from the corpus that support the candidate answer as being a correct answer for the input question.

9. The method of claim 8, wherein application of the JPM to the selected evidence portions further comprises:

performing second context dependent scoring on the additional evidence portions retrieved as a result of the additional evidence portion retrieval operations;

combining results of the first context dependent scoring and the second context dependent scoring; and

generating a final ranking of candidate answers based on the combined results of the first context dependent scoring and the second context dependent scoring.

10. The method of claim 7, wherein performing the first context dependent scoring and performing the context independent scoring are performed in parallel by parallel QA system pipeline paths on candidate answers generated by the QA system pipeline.

11. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a data processing system implementing a question and answer system (QA), causes the data processing system to:

receive the input question;

generate a set of candidate answers for the input question and, for each candidate answer in the set of candidate answers, a corresponding selection of one or more selected evidence portions from a corpus of information providing evidence in support of the candidate answer being a correct answer for the input question;

rank the candidate answers based on an application of a justifying passage model (JPM) to the selected evidence portions for each of the candidate answers in the set of candidate answers, wherein the JPM identifies whether a candidate answer is justified by a selected evidence passage corresponding to the candidate answer; and

output a candidate answer in the set of candidate answers as the correct answer for the input question based on the ranking of the candidate answers.

12. The computer program product of claim 11, wherein the JPM is trained based on a JPM ground truth data structure that comprises a justification indicator, for each question-answer-evidence passage (QAP) triplet in a plurality of QAP triplets of the JPM ground truth data structure, indicating whether an answer in the QAP triplet is justified, by the evidence passage of the QAP triplet, as being a correct answer for a question in the QAP triplet.

13. The computer program product of claim 12, wherein the justification indicator for each QAP triplet in the plurality of QAP triplets is set to a justification value by a subject matter expert that evaluates the QAP triplet.

14. The computer program product of claim 11, wherein the computer readable program further causes the data processing system to apply the JPM to the selected evidence portions at least by applying one or more weights of the JPM to evidence scores associated with the selected evidence portions to thereby generate a confidence score associated with the candidate answer, wherein the one or more weights of the JPM indicate a degree of relevance of the selected evidence portions to the input question.

15. The computer program product of claim 11, wherein the computer readable program further causes the data processing system to apply the JPM to the selected evidence portions at least by:

ranking evidence portions in the selected evidence portions according to evidence scores generated based on the application of the JPM to the selected evidence portions; and

selecting a subset of the ranked evidence portions in the selected evidence portions, for use in ranking the candidate answers, based on one or more filter criteria.

16. The computer program product of claim 11, wherein the computer readable program further causes the data processing system to apply the JPM to the selected evidence portions at least by, for each candidate answer in the set of candidate answers:

weighting an evidence score associated with an evidence portion associated with the candidate answer based on a weight value, in the JPM, associated with the evidence portion; and

calculating a weighted confidence score for the candidate answer based on a combination of weighted evidence scores for evidence portions associated with the candidate answer.

17. The computer program product of claim 11, wherein the computer readable program further causes the data processing system to apply the JPM to the selected evidence portions at least by, for each candidate answer:

performing first context dependent scoring of the evidence portions corresponding to the candidate answer, wherein the first context dependent scoring scores the candidate answer based on a correlation between the candidate answer and the evidence portions corresponding to the candidate answer; and

performing context independent scoring of the candidate answer, wherein the context independent scoring of the candidate answer scores the candidate answer independently of the evidence portions corresponding to the candidate answer.

18. The computer program product of claim 17, wherein the computer readable program further causes the data processing system to apply the JPM to the selected evidence portions at least by:

weighting, based on the JPM, context independent scores associated with the candidate answers to generate weighted context independent scores;

filtering, based on the weighted context independent scores of the candidate answers, candidate answers from the set of candidate answers to generate a subset of candidate answers; and

performing, for each candidate answer in the subset of candidate answers, additional evidence portion retrieval operations for retrieving additional evidence portions from the corpus that support the candidate answer as being a correct answer for the input question.

19. The computer program product of claim 18, wherein the computer readable program further causes the data processing system to apply the JPM to the selected evidence portions further at least by:

performing second context dependent scoring on the additional evidence portions retrieved as a result of the additional evidence portion retrieval operations;

combining results of the first context dependent scoring and the second context dependent scoring; and

generating a final ranking of candidate answers based on the combined results of the first context dependent scoring and the second context dependent scoring.

20. A data processing system configured to implement a question and answer system (QA), comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

receive the input question;

generate a set of candidate answers for the input question and, for each candidate answer in the set of candidate answers, a corresponding selection of one or more selected evidence portions from a corpus of information providing evidence in support of the candidate answer being a correct answer for the input question;

rank the candidate answers based on an application of a justifying passage model (JPM) to the selected evidence portions for each of the candidate answers in the set of candidate answers, wherein the JPM identifies whether a candidate answer is justified by a selected evidence passage corresponding to the candidate answer; and

output a candidate answer in the set of candidate answers as the correct answer for the input question based on the ranking of the candidate answers.

21. A method, in a data processing system comprising a processor and a memory configured to implement a question and answer system (QA), for generating answers to an input question, comprising:

receiving, in the data processing system, the input question;

generating, by the data processing system, a set of candidate answers for the input question and, for each candidate answer in the set of candidate answers, a corresponding selection of one or more selected evidence portions from a corpus of information providing evidence in support of the candidate answer being a correct answer for the input question;

for each candidate answer in the set of candidate answers;

applying a justifying passage model (JPM) to selected evidence portions associated with the candidate answer to generate an evidence score for each selected evidence portion, wherein the JPM identifies whether a candidate answer is justified by a selected evidence passage corresponding to the candidate answer;

selecting a subset of the selected evidence portions for use in ranking the candidate answers, based on one or more filter criteria; and

ranking the candidate answer relative to other candidate answers in the set of candidate answers based on the selected subset of the ranked evidence portions; and

outputting, by the data processing system, a candidate answer in the set of candidate answers as the correct answer for the input question based on the ranking of the candidate answers.