SCORING ATTRIBUTES IN DEEP QUESTION ANSWERING SYSTEMS BASED ON ALGORITHMIC SOURCE CODE INFLUENCES

• US 20160179811A1
• Filed: 09/23/2015
• Published: 06/23/2016
• Est. Priority Date: 12/18/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: ALLEN, Corville O., DELIMA, Roberto, EGGEBRAATEN, Thomas J., SETNES, Marie L.
• Granted Patent: US 10,120,910 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/23 Updating
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: G06N 5/04 Inference or reasoning models

Abstract

Methods to perform an operation comprising: identifying a first attribute of a source code in a deep question answering system, computing an influence score for the first attribute based on a rule in the source code used to compute a confidence score for each of a plurality of candidate answers generated by the deep question answering system, computing an importance score for the first attribute based at least in part on the computed influence score, and upon determining that the importance score exceeds a predefined threshold, storing an indication that the first attribute is an important attribute relative to other attributes specified in the source code.

Claims

1. A method, comprising:

identifying a first attribute in a source code of a deep question answering system;

computing an influence score for the first attribute based on a rule in the source code used to compute a confidence score for each of a plurality of candidate answers generated by the deep question answering system;

computing an importance score for the first attribute based at least in part on the computed influence score; and

upon determining that the importance score exceeds a predefined threshold, storing an indication that the first attribute is an important attribute relative to other attributes specified in the source code.

2. The method of claim 1, wherein the source code comprises a plurality of rules, wherein the influence score for the attribute is further determined based on:

(i) a number of rules specifying the first attribute, (ii) a location of the first attribute in each rule, (iii) a number of times the first attribute is used in each rule, (iv) a type of each respective rule, and (v) an identified phase of a processing pipeline of the deep question answering system in which each respective rule is applied.

3. The method of claim 1, further comprising:

receiving, by the deep question answering system, a case that does not specify a value for the first attribute; and

outputting, to a user, a request to specify a value for the first attribute.

4. The method of claim 1, wherein the influence score for the first attribute is further based on a weight the rule applies to the first attribute to compute the confidence score for each candidate answer.

5. The method of claim 1, wherein the influence score for the first attribute is further based on a respective weight of each of a plurality of possible values for the first attribute specified in the rule.

6. The method of claim 1, wherein the first attribute is identified during a code scan of the source code of each scorer in the deep question answering system.

7. The method of claim 1, wherein the confidence score specifies a level of confidence that a response returned by the deep question answering system is correct, wherein the source code comprises a current source code of the deep question answering system.