Avoidance of supporting evidence processing based on key attribute predictors

• US 9,652,717 B2
• Filed: 12/19/2014
• Issued: 05/16/2017
• Est. Priority Date: 12/19/2014
• Status: Active Grant

• Current Assignee: AirBNB Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Lange-Pearson, Adam C., Pitzen, Thomas P., Britt, Michael W., Stevens, Richard J.
• Primary Examiner(s): Hill, Stanley K
• Assistant Examiner(s): Kim, David H
• Application Number:
  US14/576,854
• Publication Number:
  US 20160180233A1
• Time in Patent Office: 879 Days
• Field of Search: None
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06N 5/048 Fuzzy inferencing

Abstract

Systems, methods, and computer program products to perform an operation comprising receiving a case, generating a set of candidate answers for the case, excluding a first candidate answer from the set of candidate answers upon determining that a first attribute in the case precludes returning the first candidate answer as a valid response to the case, and processing supporting evidence for the remaining candidate answers in the set of candidate answers.

Claims

1. A method, comprising:

receiving a case;

generating a set of candidate answers for the case based on a corpus of information;

excluding a first candidate answer from the set of candidate answers upon determining that a first attribute in the case precludes returning the first candidate answer as a valid response to the case based on a first rule, of a plurality of rules for processing supporting evidence;

processing supporting evidence for the remaining candidate answers in the set of candidate answers by searching for items of evidence in the corpus of information that include passages supporting at least one candidate answer for each candidate answer in the set of candidate answers; and

foregoing processing supporting evidence for the first candidate answer by refraining from searching for items of evidence in the corpus of information that include passages supporting the first candidate answer.

2. The method of claim 1, wherein the set of candidate answers specifies candidate answers for which supporting evidence will be processed, wherein supporting evidence is not processed for the first candidate answer, wherein the first rule comprises a minimum confidence threshold for a confidence score of each of the candidate answers, wherein the attribute of the first candidate answer comprises the confidence score for the first candidate answer, wherein the first candidate answer is excluded from the set of candidate answers upon determining that the confidence score for the first candidate answer is below the minimum confidence threshold, wherein foregoing processing supporting evidence for the first candidate answer is based upon the determination that the confidence score for the first candidate answer is below the minimum confidence threshold.

3. The method of claim 2, wherein an output of a candidate answer scoring algorithm specifies that the first attribute and the first rule precludes returning the first candidate answer as a valid response to the case, wherein the first candidate answer is further excluded from the set of candidate answers based on the output of the candidate answer scoring algorithm.

4. The method of claim 3, further comprising:

outputting, to a user presenting the case, an indication that supporting evidence for the first candidate answer is not included as part of a response to the case at least in part due to the first attribute.

5. The method of claim 4, further comprising:

excluding a second candidate answer from the set of candidate answers upon determining that at least one of the first attribute and a second attribute preclude returning the second candidate answer as a valid response to the case.

6. The method of claim 1, further comprising:

storing an indication that the first attribute is relevant to returning a valid response to a class of cases including the received case.

7. A system, comprising:

one or more computer processors; and

a memory containing a program, which when executed by the one or more computer processors, performs an operation comprising;

receiving a case;

generating a set of candidate answers for the case based on a corpus of information;

excluding a first candidate answer from the set of candidate answers upon determining that a first attribute in the case precludes returning the first candidate answer as a valid response to the case based on a first rule, of a plurality of rules for processing supporting evidence;

processing supporting evidence for the remaining candidate answers in the set of candidate answers by searching for items of evidence in the corpus of information that include passages supporting at least one candidate answer for each candidate answer in the set of candidate answers; and

foregoing processing supporting evidence for the first candidate answer by refraining from searching for items of evidence in the corpus of information that include passages supporting the first candidate answer.

8. The system of claim 7, wherein the set of candidate answers specifies candidate answers for which supporting evidence will be processed, wherein supporting evidence is not processed for the first candidate answer, wherein the first rule comprises a minimum confidence threshold for a confidence score of each of the candidate answers, wherein the attribute of the first candidate answer comprises the confidence score for the first candidate answer, wherein the first candidate answer is excluded from the set of candidate answers upon determining that the confidence score for the first candidate answer is below the minimum confidence threshold, wherein foregoing processing supporting evidence for the first candidate answer is based upon the determination that the confidence score for the first candidate answer is below the minimum confidence threshold.

9. The system of claim 8, wherein an output of a candidate answer scoring algorithm specifies that the first attribute and the first rule precludes returning the first candidate answer as a valid response to the case, wherein the first candidate answer is further excluded from the set of candidate answers based on the output of the candidate answer scoring algorithm.

10. The system of claim 9, the operation further comprising:

outputting, to a user presenting the case, an indication that supporting evidence for the first candidate answer is not included as part of a response to the case at least in part due to the first attribute.

11. The system of claim 10, the operation further comprising:

excluding a second candidate answer from the set of candidate answers upon determining that at least one of the first attribute and a second attribute preclude returning the second candidate answer as a valid response to the case.

12. The system of claim 7, the operation further comprising:

storing an indication that the first attribute is relevant to returning a valid response to a class of cases including the received case.

13. A computer program product, comprising:

a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation comprising;

receiving a case;

generating a set of candidate answers for the case based on a corpus of information;

excluding a first candidate answer from the set of candidate answers upon determining that a first attribute in the case precludes returning the first candidate answer as a valid response to the case based on a first rule, of a plurality of rules for processing supporting evidence;

processing supporting evidence for the remaining candidate answers in the set of candidate answers by searching for items of evidence in the corpus of information that include passages supporting at least one candidate answer for each candidate answer in the set of candidate answers; and

foregoing processing supporting evidence for the first candidate answer by refraining from searching for items of evidence in the corpus of information that include passages supporting the first candidate answer.

14. The computer program product of claim 13, wherein the set of candidate answers specifies candidate answers for which supporting evidence will be processed, wherein supporting evidence is not processed for the first candidate answer, wherein the first rule comprises a minimum confidence threshold for a confidence score of each of the candidate answers, wherein the attribute of the first candidate answer comprises the confidence score for the first candidate answer, wherein the first candidate answer is excluded from the set of candidate answers upon determining that the confidence score for the first candidate answer is below the minimum confidence threshold, wherein foregoing processing supporting evidence for the first candidate answer is based upon the determination that the confidence score for the first candidate answer is below the minimum confidence threshold.

15. The computer program product of claim 14, wherein an output of a candidate answer scoring algorithm specifies that the first attribute and the first rule precludes returning the first candidate answer as a valid response to the case, wherein the first candidate answer is further excluded from the set of candidate answers based on the output of the candidate answer scoring algorithm.

16. The computer program product of claim 15, wherein the operation further comprises:

outputting, to a user presenting the case, an indication that supporting evidence for the first candidate answer is not included as part of a response to the case at least in part due to the first attribute.

17. The computer program product of claim 16, wherein the operation further comprises:

excluding a second candidate answer from the set of candidate answers upon determining that at least one of the first attribute and a second attribute preclude returning the second candidate answer as a valid response to the case.

18. The computer program product of claim 17, the operation further comprising:

determining, during a training phase of a question answering system, that the first attribute impacts the confidence score of the first candidate answer beyond an impact threshold.

19. The computer program product of claim 18, wherein the plurality of rules are defined by:

(i) a user, and (ii) the question answering system based on the corpus of information.