Justifying passage machine learning for question and answer systems
First Claim
1. A method, in a data processing system comprising a processor and a memory configured to implement a question and answer system (QA), for generating answers to an input question, comprising:
- training a justifying passage model (JPM) based on a JPM ground truth data structure that comprises a justification indicator for each question-answer-evidence passage (QAP) triplet in a plurality of QAP triplets of the JPM ground truth data structure, wherein the justification indicator indicates whether or not an answer in the QAP triplet is justified by the evidence passage of the QAP triplet as being a correct answer for a question in the QAP triplet, and wherein an answer of the QAP triplet is justified by the evidence passage of the QAP triplet when content of the evidence passage explicitly states the answer to be a correct answer for the question of the QAP triplet;
receiving, in the data processing system, the input question;
generating, by the data processing system, a set of candidate answers for the input question and, for each candidate answer in the set of candidate answers, a corresponding selection of one or more selected evidence portions from a corpus of information providing evidence in support of the candidate answer being a correct answer for the input question;
ranking, by the data processing system, the candidate answers based on an application of the trained JPM to the selected evidence portions for each of the candidate answers in the set of candidate answers, wherein the JPM identifies whether a candidate answer is justified by a selected evidence portion corresponding to the candidate answer, and wherein application of the trained JPM to the selected evidence portions causes the ranking of the candidate answers to be modified based on whether or not a selected evidence portion is justifying of a corresponding candidate answer; and
outputting, by the data processing system, a candidate answer in the set of candidate answers as the correct answer for the input question based on the ranking of the candidate answers.
1 Assignment
0 Petitions
Accused Products
Abstract
Mechanisms are provided for generating an answer to an input question. An input question is received and a set of candidate answers is generated along with, for each candidate answer in the set of candidate answers, a corresponding selection of one or more selected evidence portions from a corpus of information providing evidence in support of the candidate answer being a correct answer for the input question. The candidate answers are ranked based on an application of a justifying passage model (JPM) to the selected evidence portions for each of the candidate answers in the set of candidate answers. The JPM identifies whether a candidate answer is justified by a selected evidence passage corresponding to the candidate answer. A candidate answer is output as the correct answer for the input question based on the ranking of the candidate answers.
35 Citations
21 Claims
-
1. A method, in a data processing system comprising a processor and a memory configured to implement a question and answer system (QA), for generating answers to an input question, comprising:
-
training a justifying passage model (JPM) based on a JPM ground truth data structure that comprises a justification indicator for each question-answer-evidence passage (QAP) triplet in a plurality of QAP triplets of the JPM ground truth data structure, wherein the justification indicator indicates whether or not an answer in the QAP triplet is justified by the evidence passage of the QAP triplet as being a correct answer for a question in the QAP triplet, and wherein an answer of the QAP triplet is justified by the evidence passage of the QAP triplet when content of the evidence passage explicitly states the answer to be a correct answer for the question of the QAP triplet; receiving, in the data processing system, the input question; generating, by the data processing system, a set of candidate answers for the input question and, for each candidate answer in the set of candidate answers, a corresponding selection of one or more selected evidence portions from a corpus of information providing evidence in support of the candidate answer being a correct answer for the input question; ranking, by the data processing system, the candidate answers based on an application of the trained JPM to the selected evidence portions for each of the candidate answers in the set of candidate answers, wherein the JPM identifies whether a candidate answer is justified by a selected evidence portion corresponding to the candidate answer, and wherein application of the trained JPM to the selected evidence portions causes the ranking of the candidate answers to be modified based on whether or not a selected evidence portion is justifying of a corresponding candidate answer; and outputting, by the data processing system, a candidate answer in the set of candidate answers as the correct answer for the input question based on the ranking of the candidate answers. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 20)
-
-
10. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a data processing system implementing a question and answer system (QA), causes the data processing system to:
-
train a justifying passage model (JPM) based on a JPM ground truth data structure that comprises a justification indicator for each question-answer-evidence passage (QAP) triplet in a plurality of QAP triplets of the JPM ground truth data structure, wherein the justification indicator indicates whether or not an answer in the QAP triplet is justified by the evidence passage of the QAP triplet as being a correct answer for a question in the QAP triplet, and wherein an answer of the QAP triplet is justified by the evidence passage of the QAP triplet when content of the evidence passage explicitly states the answer to be a correct answer for the question of the QAP triplet; receive the input question; generate a set of candidate answers for the input question and, for each candidate answer in the set of candidate answers, a corresponding selection of one or more selected evidence portions from a corpus of information providing evidence in support of the candidate answer being a correct answer for the input question; rank the candidate answers based on an application of the trained JPM to the selected evidence portions for each of the candidate answers in the set of candidate answers, wherein the JPM identifies whether a candidate answer is justified by a selected evidence portion corresponding to the candidate answer, and wherein application of the trained JPM to the selected evidence portions causes the ranking of the candidate answers to be modified based on whether or not a selected evidence portion is justifying of a corresponding candidate answer; and output a candidate answer in the set of candidate answers as the correct answer for the input question based on the ranking of the candidate answers. - View Dependent Claims (11, 12, 13, 14, 15, 16, 17, 21)
-
-
18. A data processing system configured to implement a question and answer system (QA), comprising:
-
a processor; and a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to; train a justifying passage model (JPM) based on a JPM ground truth data structure that comprises a justification indicator for each question-answer-evidence passage (QAP) triplet in a plurality of QAP triplets of the JPM ground truth data structure, wherein the justification indicator indicates whether or not an answer in the QAP triplet is justified by the evidence passage of the QAP triplet as being a correct answer for a question in the QAP triplet, and wherein an answer of the QAP triplet is justified by the evidence passage of the QAP triplet when content of the evidence passage explicitly states the answer to be a correct answer for the question of the QAP triplet; receive the input question; generate a set of candidate answers for the input question and, for each candidate answer in the set of candidate answers, a corresponding selection of one or more selected evidence portions from a corpus of information providing evidence in support of the candidate answer being a correct answer for the input question; rank the candidate answers based on an application of the trained JPM to the selected evidence portions for each of the candidate answers in the set of candidate answers, wherein the JPM identifies whether a candidate answer is justified by a selected evidence portion corresponding to the candidate answer, and wherein application of the trained JPM to the selected evidence portions causes the ranking of the candidate answers to be modified based on whether or not a selected evidence portion is justifying of a corresponding candidate answer; and output a candidate answer in the set of candidate answers as the correct answer for the input question based on the ranking of the candidate answers.
-
-
19. A method, in a data processing system comprising a processor and a memory configured to implement a question and answer system (QA), for generating answers to an input question, comprising:
-
receiving, in the data processing system, the input question; generating, by the data processing system, a set of candidate answers for the input question and, for each candidate answer in the set of candidate answers, a corresponding selection of one or more selected evidence portions from a corpus of information providing evidence in support of the candidate answer being a correct answer for the input question; for each candidate answer in the set of candidate answers; applying a justifying passage model (JPM) to selected evidence portions associated with the candidate answer to generate an evidence score for each selected evidence portion, wherein the JPM identifies whether a candidate answer is justified by a selected evidence passage corresponding to the candidate answer and modifies the evidence score according to whether or not the candidate answer is justified by the selected evidence passage, and wherein the candidate answer is justified by the selected evidence portion when content of the selected evidence portion explicitly states the candidate answer to be a correct answer for the input question; selecting a subset of the selected evidence portions for use in ranking the candidate answers, based on one or more filter criteria; and ranking the candidate answer relative to other candidate answers in the set of candidate answers based on the selected subset of the ranked evidence portions; and outputting, by the data processing system, a candidate answer in the set of candidate answers as the correct answer for the input question based on the ranking of the candidate answers.
-
Specification