Avoiding supporting evidence processing when evidence scoring does not affect final ranking of a candidate answer

• US 10,108,906 B2
• Filed: 12/19/2014
• Issued: 10/23/2018
• Est. Priority Date: 12/19/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Britt, Michael W., Lange-Pearson, Adam C., Pitzen, Thomas P., Stevens, Richard J.
• Primary Examiner(s): Hill, Stanley K.
• Application Number:
  US14/576,893
• Publication Number:
  US 20160180244A1
• Time in Patent Office: 1,404 Days
• Field of Search: None
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: H04L 45/08 Learning-based routing, e.g. using neural networks or artificial intelligence

Abstract

Systems and computer program products to provide selective supporting evidence processing by applying a first machine learning (ML) model to a first candidate answer to generate a first confidence score that does not consider supporting evidence for the first candidate answer, determining, from a second ML model, an expected contribution of processing supporting evidence for the first candidate answer, and upon determining that the expected contribution does not exceed a specified threshold, skipping supporting evidence processing for the first candidate answer.

Claims

1. A system, comprising:

one or more computer processors; and

a memory containing a program, which when executed by the one or more computer processors, performs an operation to provide selective supporting evidence processing in a question answering (QA) system, the operation comprising;

initiating processing of a first question using an execution pipeline of the QA system, wherein the execution pipeline comprises (i) a first stage configured to determine candidate answers to the first question without consideration of any items of supporting evidence and (ii) a second stage, downstream from the first stage, configured to process items of supporting evidence relating to the candidate answers in order to generate updated confidence scores;

upon determining a first candidate answer for a first question during the first stage of the execution pipeline and prior to executing the second stage of the execution pipeline;

generating, by the QA system, using a first machine learning (ML) model, a first confidence score value for the first candidate answer, wherein the first confidence score value reflects a degree to which the first candidate answer is a correct response to the first question, wherein the first ML model does not consider supporting evidence features for the first candidate answer;

generating, by the QA system, using a second ML model, a measure of expected change to the first confidence score value based at least in part on supporting evidence features for the first candidate answer; and

upon determining that the measure of expected change does not exceed a first threshold, returning, by the QA system, at least the first candidate answer in response to the first question, without processing the first question using the second stage of the execution pipeline.

2. The system of claim 1, wherein the second ML model specifies:

(i) a weighted coefficient value for processing items of supporting evidence for the first candidate answer, and (ii) a plurality of supporting evidence feature score values, wherein each supporting evidence feature score value of the plurality of supporting evidence feature score values was observed during a training session of the QA system by processing items of supporting evidence for each of a plurality of candidate answers generated during the training session.

3. The system of claim 2, wherein the measure of expected change to the first confidence score value is generated by computing a product of the weighted coefficient value and one of:

(i) an average of the plurality of supporting evidence feature score values in the second (ML) model, (ii) a maximum value of the plurality of supporting evidence feature score values, and (iii) a selected one of the plurality of supporting evidence feature score values that is greater than a threshold percentage of the plurality of supporting evidence feature score values.

4. The system of claim 3, wherein the plurality of threshold comprises at least one of:

(i) a difference between the confidence score value of a second candidate answer and the first confidence score value of the first candidate answer, wherein the confidence score value of the second candidate answer is generated by applying the first ML model to the second candidate answer, (ii) a difference between a confidence score threshold for a first confidence tier of a plurality of confidence tiers and the first confidence score value, and (iii) a difference between the first confidence score value and a minimum confidence score value, wherein the minimum confidence score value is associated with a minimum confidence tier of the plurality of confidence tiers.

5. The system of claim 1, the operation further comprising:

generating, by the QA system, using the first machine learning (ML) model, a second confidence score value for a second candidate answer;

generating, by the QA system, using the second ML model, a second measure of expected change to the first confidence score value based at least in part on supporting evidence features for the second candidate answer;

determining that the measure of expected change exceeds a second threshold;

processing items of supporting evidence for the second candidate answer;

scoring the second candidate answer; and

ranking the second candidate answer relative to a set of other candidate answers based on a respective score for each candidate answer.

6. The system of claim 5, the operation further comprising:

prior to processing items of supporting evidence for the second candidate answer, determining that the second confidence score value exceeds a minimum confidence threshold.

7. The system of claim 1, the operation further comprising:

generating, by the QA system, the first and second ML models during a training session of the QA system.

8. A computer program product to provide selective supporting evidence processing in a question answering (QA) system, comprising:

a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation comprising;

initiating processing of a first question using an execution pipeline of the QA system, wherein the execution pipeline comprises (i) a first stage configured to determine candidate answers to the first question without consideration of any items of supporting evidence and (ii) a second stage, downstream from the first stage, configured to process items of supporting evidence relating to the candidate answers in order to generate updated confidence scores;

upon determining a first candidate answer for a first question during the first stage of the execution pipeline and prior to executing the second stage of the execution pipeline;

generating, by the QA system, using a first machine learning (ML) model, a first confidence score value for the first candidate answer, wherein the first confidence score value reflects a degree to which the first candidate answer is a correct response to the first question, wherein the first ML model does not consider supporting evidence features for the first candidate answer;

generating, by the QA system, using a second ML model, a measure of expected change to the first confidence score value based at least in part on supporting evidence features for the first candidate answer; and

upon determining that the measure of expected change does not exceed a first threshold, returning, by the QA system, at least the first candidate answer in response to the first question, without processing the first question using the second stage of the execution pipeline.

9. The computer program product of claim 8, wherein the second ML model specifies:

(i) a weighted coefficient value for processing items of supporting evidence for the first candidate answer, and (ii) a plurality of supporting evidence feature score values, wherein each supporting evidence feature score value of the plurality of supporting evidence feature score values was observed during a training session of the QA system by processing items of supporting evidence for each of a plurality of candidate answers generated during the training session.

10. The computer program product of claim 9, wherein the measure of expected change is generated by computing a product of the weighted coefficient value and one of:

(i) an average of the plurality of supporting evidence feature score values in the second ML model, (ii) a maximum value of the plurality of supporting evidence feature score values, and (iii) a selected one of the plurality of supporting evidence feature score values that is greater than a threshold percentage of the plurality of supporting evidence feature score values.

11. The computer program product of claim 10, wherein the first threshold comprises at least one of:

(i) a difference between the confidence score value of a second candidate answer and the first confidence score value of the first candidate answer, wherein the confidence score value of the second candidate answer is generated by applying the first ML model to the second candidate answer, (ii) a difference between a confidence score threshold for a first confidence tier of a plurality of confidence tiers and the first confidence score value, and (iii) a difference between the first confidence score value and a minimum confidence score value, wherein the minimum confidence score value is associated with a minimum confidence tier of the plurality of confidence tiers.

12. The computer program product of claim 9, the operation further comprising:

generating, by the QA system, using the first machine learning (ML) model, a second confidence score value for a second candidate answer;

generating, by the QA system, using the second ML model, a second measure of expected change to the first confidence score value based at least in part on supporting evidence features for the second candidate answer;

determining that the measure of expected change exceeds a second threshold;

processing items of supporting evidence for the second candidate answer;

scoring the second candidate answer; and

ranking the second candidate answer relative to a set of other candidate answers based on a respective score for each candidate answer.

13. The computer program product of claim 12, the operation further comprising:

generating, by the QA system, the first and second ML models during a training session of the QA system.

14. The computer program product of claim 8, the operation further comprising:

generating, by applying the first ML model to a second candidate answer, a second confidence score value for the second candidate answer;

generating, by applying the second ML model to the second candidate answer, a measure of expected change to the second confidence score value based at least in part on supporting evidence features;

determining that the measure of expected change to the second confidence score value exceeds a second threshold; and

processing items of supporting evidence for the second candidate answer.

15. The computer program product of claim 14, wherein the second ML model specifies:

(i) a weighted coefficient value for processing items of supporting evidence for the first candidate answer, and (ii) a plurality of supporting evidence feature score values, wherein each supporting evidence feature score value of the plurality of supporting evidence feature score values was observed during a training session of the QA system using supporting evidence processing for each of a plurality of candidate answers generated during the training session.

16. The computer program product of claim 15, wherein the measure of expected change is generated by computing a product of the weighted coefficient value and a first one of:

(i) an average of the plurality of supporting evidence feature score values in the second model, (ii) a maximum value of the plurality of supporting evidence feature score values, and (iii) a selected one of the plurality of supporting evidence feature score values that is greater than a threshold percentage of the plurality of supporting evidence feature score values, wherein value corresponding to the expected change to the first confidence score value is determined by computing a product of the weighted coefficient value and a second one of;

(i) the average of the plurality of supporting evidence feature score values in the second machine learning model, (ii) the maximum value of the plurality of supporting evidence feature score values, and (iii) the selected one of the plurality of supporting evidence feature score values that is greater than the threshold percentage of the plurality of supporting evidence feature score values.

17. The computer program product of claim 16, wherein the first threshold comprises one of:

(i) a difference between the confidence score value of the second candidate answer and the first confidence score value of the first candidate answer, (ii) a difference between a confidence score threshold for a first confidence tier of a plurality of confidence tiers and the first confidence score value, and (iii) a difference between the first confidence score value and a minimum confidence score value, wherein the minimum confidence score value is associated with a minimum confidence tier of the plurality of confidence tiers.

18. The computer program product of claim 17, wherein the second threshold comprises a difference between the confidence score threshold for a second confidence tier of the plurality of confidence tiers and the second confidence score value.