AVOIDING SUPPORTING EVIDENCE PROCESSING WHEN EVIDENCE SCORING DOES NOT AFFECT FINAL RANKING OF A CANDIDATE ANSWER

• US 20160180244A1
• Filed: 12/19/2014
• Published: 06/23/2016
• Est. Priority Date: 12/19/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: BRITT, Michael W., LANGE-PEARSON, Adam C., PITZEN, Thomas P., STEVENS, Richard J.
• Granted Patent: US 10,108,906 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: H04L 45/08 Learning-based routing, e.g. using neural networks or artificial intelligence

Abstract

Systems and computer program products to provide selective supporting evidence processing by applying a first machine learning (ML) model to a first candidate answer to generate a first confidence score that does not consider supporting evidence for the first candidate answer, determining, from a second ML model, an expected contribution of processing supporting evidence for the first candidate answer, and upon determining that the expected contribution does not exceed a specified threshold, skipping supporting evidence processing for the first candidate answer.

Claims

1-7. -7. (canceled)

8. A system, comprising:

one or more computer processors; and

a memory containing a program, which when executed by the one or more computer processors, performs an operation to provide selective supporting evidence processing, the operation comprising;

applying a first machine learning (ML) model to a first candidate answer to generate a first confidence score that does not consider supporting evidence for the first candidate answer;

determining, from a second ML model, an expected contribution of processing supporting evidence for the first candidate answer; and

upon determining that the expected contribution does not exceed a specified threshold, skipping supporting evidence processing for the first candidate answer.

9. The system of claim 8, wherein the second ML model specifies:

(i) a weighted coefficient of processing supporting evidence for the first candidate answer, and (ii) a range of supporting evidence feature scores, wherein each supporting evidence feature score in the range of supporting evidence scores was observed during a training session.

10. The system of claim 9, wherein the expected contribution comprises a product of the weighted coefficient and at least one supporting evidence feature score, of the range of supporting evidence feature scores.

11. The system of claim 8, wherein the threshold comprises at least one of:

(i) a difference between the confidence score of a second candidate answer and the confidence score of the first candidate answer, wherein the confidence score of the second candidate answer is generated by applying the first ML model to the second candidate answer, and (ii) a difference between a confidence score threshold and the first confidence score.

12. The system of claim 8, the operation further comprising:

upon determining that the expected contribution exceeds the specified threshold, processing supporting evidence for the first candidate answer;

scoring the first candidate answer; and

ranking the first candidate answer relative to a set of other candidate answers based on a respective score for each candidate answer.

13. The system of claim 12, the operation further comprising:

prior to processing supporting evidence for the first candidate answer, determining that the first confidence score exceeds a minimum confidence threshold.

14. The system of claim 8, the operation further comprising:

producing the first and second ML models during a training session.

15. A computer program product to provide selective supporting evidence processing, comprising:

a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to;

apply a first machine learning (ML) model to a first candidate answer to generate a first confidence score that does not consider supporting evidence for the first candidate answer;

determine, from a second ML model, an expected contribution of processing supporting evidence for the first candidate answer; and

upon determining that the expected contribution does not exceed a specified threshold, skipping supporting evidence processing for the first candidate answer.

16. The computer program product of claim 15, wherein the second ML model specifies:

(i) a weighted coefficient of processing supporting evidence for the first candidate answer, and (ii) a range of supporting evidence feature scores, wherein each supporting evidence feature score in the range of supporting evidence scores was observed during a training session.

17. The computer program product of claim 16, wherein the expected contribution comprises a product of the weighted coefficient and at least one supporting evidence feature score, of the range of supporting evidence feature scores.

18. The computer program product of claim 15, wherein the threshold comprises at least one of:

(i) a difference between the confidence score of a second candidate answer and the confidence score of the first candidate answer, wherein the confidence score of the second candidate answer is generated by applying the first ML model to the second candidate answer, and (ii) a difference between a confidence score threshold and the first confidence score.

19. The computer program product of claim 16, wherein the computer-readable program code is further executable to:

upon determining that the expected contribution exceeds the specified threshold, process supporting evidence for the first candidate answer;

score the first candidate answer; and

rank the first candidate answer relative to a set of other candidate answers based on a respective score for each candidate answer.

20. The computer program product of claim 19, wherein the first and second ML models are produced during a training session, wherein the computer-readable program code is further executable to:

produce the first and second ML models during the training session.