AVOIDING SUPPORTING EVIDENCE PROCESSING WHEN EVIDENCE SCORING DOES NOT AFFECT FINAL RANKING OF A CANDIDATE ANSWER

• US 20160180249A1
• Filed: 09/22/2015
• Published: 06/23/2016
• Est. Priority Date: 12/19/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: BRITT, Michael W., PITZEN, Thomas P., STEVENS, Richard J., LANGE-PEARSON, Adam C.
• Granted Patent: US 10,147,050 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: H04L 45/08 Learning-based routing, e.g. using neural networks or artificial intelligence

Abstract

Methods to provide selective supporting evidence processing by applying a first machine learning (ML) model to a first candidate answer to generate a first confidence score that does not consider supporting evidence for the first candidate answer, determining, from a second ML model, an expected contribution of processing supporting evidence for the first candidate answer, and upon determining that the expected contribution does not exceed a specified threshold, skipping supporting evidence processing for the first candidate answer.

Claims

1. A method to provide selective supporting evidence, comprising:

applying a first machine learning (ML) model to a first candidate answer to generate a first confidence score that does not consider supporting evidence for the first candidate answer;

determining, from a second ML model, an expected contribution of processing supporting evidence for the first candidate answer; and

upon determining that the expected contribution does not exceed a specified threshold, skipping supporting evidence processing for the first candidate answer.

2. The method of claim 1, wherein the second ML model specifies:

(i) a weighted coefficient of processing supporting evidence for the first candidate answer, and (ii) a range of supporting evidence feature scores, wherein each supporting evidence feature score in the range of supporting evidence scores was observed during a training session.

3. The method of claim 2, wherein the expected contribution comprises a product of the weighted coefficient and at least one supporting evidence feature score, of the range of supporting evidence feature scores.

4. The method of claim 1, wherein the threshold comprises at least one of:

(i) a difference between the confidence score of a second candidate answer and the confidence score of the first candidate answer, wherein the confidence score of the second candidate answer is generated by applying the first ML model to the second candidate answer, and (ii) a difference between a confidence score threshold and the first confidence score.

5. The method of claim 1, further comprising:

upon determining that the expected contribution exceeds the specified threshold, processing supporting evidence for the first candidate answer;

scoring the first candidate answer; and

ranking the first candidate answer relative to a set of other candidate answers based on a respective score for each candidate answer.

6. The method of claim 5, further comprising:

prior to processing supporting evidence for the first candidate answer, determining that the first confidence score exceeds a minimum confidence threshold.

7. The method of claim 1, further comprising:

producing the first and second ML models during a training session.