Scoring attributes in deep question answering systems based on algorithmic source code influences

• US 10,127,284 B2
• Filed: 12/18/2014
• Issued: 11/13/2018
• Est. Priority Date: 12/18/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Allen, Corville O., Delima, Roberto, Eggebraaten, Thomas J., Setnes, Marie L.
• Primary Examiner(s): Rifkin, Ben
• Application Number:
  US14/574,861
• Publication Number:
  US 20160179862A1
• Time in Patent Office: 1,426 Days
• Field of Search: 706 45
• CPC Class Codes: G06F 16/23 Updating
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: G06N 5/04 Inference or reasoning models

Abstract

Systems and computer program products to perform an operation comprising: identifying a first attribute of a source code in a deep question answering system, computing an influence score for the first attribute based on a rule in the source code used to compute a confidence score for each of a plurality of candidate answers generated by the deep question answering system, computing an importance score for the first attribute based at least in part on the computed influence score, and upon determining that the importance score exceeds a predefined threshold, storing an indication that the first attribute is an important attribute relative to other attributes specified in the source code.

Claims

1. A system, comprising:

one or more computer processors; and

a memory containing a program which when executed by the processors performs an operation comprising;

identifying a first variable in a source code of a question answering (QA) system;

upon determining that a weight applied to a value of the first variable by a first rule in the source code increases a confidence score for candidate answers generated by the QA system beyond a threshold;

computing an influence score for the first variable based on;

(i) the weight applied to the value of the first variable by the first rule in the source code, (ii) a number of rules specifying weights applied to values of the first attribute, (iii) a location of the first attribute in each rule, (iv) a number of times the first variable is used in each rule, (v) a type of operation applied to the value of the first variable by each respective rule, and (vi) an identified phase of a processing pipeline of the QA system in which each respective rule is applied;

computing an importance score for the first variable based at least in part on the computed influence score; and

upon determining that the importance score exceeds a predefined threshold, storing an indication that the first variable is an important variable relative to other variables specified in the source code;

receiving, by the QA system, a case that does not specify a value for the first variable; and

refraining, by the QA system, from processing the case.

2. The system of claim 1, the operation further comprising:

outputting, to a user, a request to provide a value for the first variable;

receiving, from the user, a first value for the variable; and

processing, by the QA system, the case using the first value for the variable.

3. The system of claim 1, wherein the influence score for the first variable is further based on a respective weight applied to each of a plurality of possible values for the first variable specified in the first rule.

4. The system of claim 1, wherein the first variable is identified during a code scan of the source code of each of a plurality of scorers configured to compute confidence scores for candidate answers in the QA system.

5. The system of claim 1, wherein the confidence score specifies a level of confidence that a response to a case generated by the deep question answering system is correct, wherein the source code comprises a current source code of the QA system.

6. The system of claim 1, the operation further comprising:

identifying a second variable in the source code of the QA system;

determining that a weight applied to a value of the second variable by a second rule in the source code increases the confidence score for candidate answers generated by the QA system beyond the threshold;

computing an influence score for the second variable based on the weight applied to the value of the second variable by the second rule in the source code;

computing an importance score for the second variable based at least in part on the computed influence score for the second variable; and

determining that the importance score exceeds the predefined threshold.

7. The system of claim 6, the operation further comprising:

receiving, by the QA system, a case that does not specify values for at least the first and second variables;

determining that the importance score for the first variable is greater than the importance for the second variable; and

outputting, to a user, a request to provide values for the first and second variables, wherein the request is ordered based on the importance score of the first variable being greater than the importance score of the second variable.

8. The system of claim 7, wherein the case does not specify values for a plurality of variables including the first and second variables, the operation further comprising:

generating an ordered list of the plurality of variables, wherein the list is ordered according to the importance score of each variable;

outputting the ordered list to the user;

receiving, from the user, values for each of the plurality of variables; and

processing, by the QA system, the case based on the values for each of the plurality of variables provided by the user.

9. A computer program product, comprising:

a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation comprising;

identifying a first variable in a source code of a question answering (QA) system;

upon determining that a weight applied to a value of the first variable by a first rule in the source code increases a confidence score for candidate answers generated by the QA system beyond a threshold;

computing an influence score for the first variable based on;

(i) the weight applied to the value of the first variable by the first rule in the source code, (ii) a number of rules specifying weights applied to values of the first attribute, (iii) a location of the first attribute in each rule, (iv) a number of times the first variable is used in each rule, (v) a type of operation applied to the value of the first variable by each respective rule, and (vi) an identified phase of a processing pipeline of the QA system in which each respective rule is applied;

computing an importance score for the first variable based at least in part on the computed influence score; and

upon determining that the importance score exceeds a predefined threshold, storing an indication that the first variable is an important variable relative to other variables specified in the source code;

receiving, by the QA system, a case that does not specify a value for the first variable; and

refraining, by the QA system, from processing the case.

10. The computer program product of claim 9, the operation further comprising:

outputting, to a user, a request to provide a value for the first variable;

receiving, from the user, a first value for the variable; and

processing, by the QA system, the case using the first value for the variable.

11. The computer program product of claim 9, wherein the influence score for the first variable is further computed based on a respective weight applied to each of a plurality of possible values for the first variable specified in the first rule, wherein the first variable is identified during a code scan of the source code of each of a plurality of scorers configured to compute confidence scores for candidate answers in the QA system.

12. The computer program product of claim 9, wherein the confidence score specifies a level of confidence that a response to a case generated by the deep question answering system is correct, wherein the source code comprises a current source code of the QA system.

13. The computer program product of claim 9, the operation further comprising:

identifying a second variable in the source code of the QA system;

determining that a weight applied to a value of the second variable by a second rule in the source code increases the confidence score for candidate answers generated by the QA system beyond the threshold;

computing an influence score for the second variable based on the weight applied to the value of the second variable by the second rule in the source code;

computing an importance score for the second variable based at least in part on the computed influence score for the second variable; and

determining that the importance score exceeds the predefined threshold.

14. The computer program product of claim 13, the operation further comprising:

receiving, by the QA system, a case that does not specify values for at least the first and second variables;

determining that the importance score for the first variable is greater than the importance for the second variable; and

outputting, to a user, a request to provide values for the first and second variables, wherein the request is ordered based on the importance score of the first variable being greater than the importance score of the second variable.

15. The computer program product of claim 14, wherein the case does not specify values for a plurality of variables including the first and second variables, the operation further comprising:

generating an ordered list of the plurality of variables, wherein the list is ordered according to the importance score of each variable;

outputting the ordered list to the user;

receiving, from the user, values for each of the plurality of variables; and

processing, by the QA system, the case based on the values for each of the plurality of variables provided by the user.