Multi-dimensional feature merging for supporting evidence in a question and answering system

• US 9,773,044 B2
• Filed: 03/11/2013
• Issued: 09/26/2017
• Est. Priority Date: 12/17/2012
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Dubbels, Joel C., Eggebraaten, Thomas J., Megerian, Mark G., Rapp, William C., Stevens, Richard J., Wildt, Patrick M., Will, Eric W.
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Lamardo, Viker
• Application Number:
  US13/793,287
• Publication Number:
  US 20140172879A1
• Time in Patent Office: 1,660 Days
• Field of Search: None
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems

Abstract

Method, system, and computer program product to analyze a plurality of candidate answers identified as responsive to a question presented to a deep question answering system, by computing a first feature score for a first feature of an item of evidence, of a plurality of items of evidence, the first feature score being based on at least one attribute of the first feature, the item of evidence relating to a first candidate answer, of the plurality of candidate answers, and computing a merged feature score for the first candidate answer by applying the first feature score to a second feature score for a second feature of the item of evidence.

Claims

1. A method to analyze a plurality of candidate answers for a case received by a question answering system, the method comprising:

defining a first set of a plurality of features for;

(i) the candidate answers, and (ii) a plurality of items of evidence, wherein defining the first set of features further specifies to merge the first set of features of the plurality of items of evidence based on a set of attributes of each feature in the first set of features;

defining a second set of the plurality of features, different than the first set of features, wherein defining the second set of features further specifies to merge the second set of features of the plurality of items of evidence based on a set of attributes of each feature in the second set of features;

computing a first feature score for the merged first set of features for each of the plurality of items of evidence for a first candidate answer of the plurality of candidate answers, wherein the first feature score is based on the set of attributes of each feature in the first set of features and each feature in the merged first set of features of the plurality of items of evidence, wherein the first feature score, the set of attributes of each feature in the first set of features, and the merged first set of features of the plurality of items of evidence are related to a first attribute of a plurality of attributes of the plurality of candidate answers and the plurality of items of evidence;

computing a second feature score for the merged second set of features for the plurality of items of evidence for the first candidate answer, wherein the second feature score is based on;

(i) the set of attributes of each feature in the second set of features, (ii) the first feature score for the merged first set of features for the respective item of evidence, and (iii) each feature in the merged second set of features of the plurality of items of evidence, wherein the second feature score, the set of attributes of each feature in the second set of features, and the merged second set of features of the plurality of items of evidence are related to a second attribute of the plurality of attributes;

selecting a first merging technique, from a plurality of merging techniques, for merging the first and second feature scores; and

computing a merged feature score for the first candidate answer based on the first merging technique applied to the first and second feature scores for each of the plurality of items of evidence.

2. The method of claim 1, wherein the first feature score, the second feature score, and the merged feature score are of a plurality of possible values for each respective score, wherein the plurality of merging techniques comprise:

(i) weighting the merged feature score based on the first feature score, (ii) weighting the merged feature score based on the second feature score, (iii) not considering the first feature score upon determining the first feature score does not exceed a minimum weight threshold for the first feature score, (iv) considering the first feature score upon determining the first feature score exceeds the minimum weight threshold for the first feature score, (v) not considering the second feature score upon determining the second feature score does not exceed a minimum weight threshold for the second feature score, and (vi) considering the second feature score upon determining the second feature score exceed the minimum weight threshold for the second feature score, wherein computing the first feature score merges the first set of features of the plurality of items of evidence, wherein computing the second feature score merges the second set of features of the plurality of items of evidence, wherein the plurality of features comprise;

(i) a quality, (ii) a relevance, (iii) a sentiment, and (iv) a source of each candidate answer and each item of evidence, wherein computing the second feature score for a first item of evidence of the plurality of items of evidence based on the first feature score of the first item of evidence comprises;

not considering the first feature score of the first item of evidence when computing the second feature score for the first item of evidence upon determining that the first feature score of the first item of evidence does not exceed the minimum weight threshold for the first feature score; and

weighting the second feature score of the first item of evidence based on the first feature score of the first item of evidence upon determining that the first feature score of the first item of evidence exceeds the minimum weight threshold for the first feature score.

3. The method of claim 2, wherein the first merging technique comprises a combination of at least two of the plurality of merging techniques, the method further comprising:

computing a merged feature score of each of the plurality of candidate answers based on the first merging technique, wherein the merged feature scores of the plurality of candidate answers are used to select one of the plurality of candidate answers as a correct response to a question presented to the question answering system.

4. The method of claim 3, further comprising identifying the first feature and the second feature.

5. The method of claim 4, wherein the computing the first and second feature scores further comprises:

(i) applying a machine learning model to the respective items of evidence, and (ii) computing a weighted sum of a set of values for each of the feature in the respective sets of features.

6. The method of claim 5, further comprising training the question answering system to produce the machine learning model.

7. The method of claim 1, wherein a third feature of each item of evidence is not considered in generating the merged feature score for the first candidate answer.

8. The method of claim 1, further comprising:

computing a third feature score for a third feature of each of the plurality of items of evidence, wherein each of the third feature scores are based on at least one attribute of the third feature in the respective item of evidence, wherein the second feature scores are further computed based on the third feature score of the respective item of evidence.

9. A method to analyze a plurality of candidate answers for a case received by a question answering system, the method comprising:

defining a first set of a plurality of features for;

(i) the candidate answers, and (ii) a plurality of items of evidence, wherein defining the first set of features further specifies to merge the first set of features of the plurality of items of evidence based on a set of attributes of each feature in the first set of features;

defining a second set of the plurality of features, different than the first set of features, wherein defining the second set of features further specifies to merge the second set of features of the plurality of items of evidence based on a set of attributes of each feature in the second set of features;

computing a first feature score for the merged first set of features for each of the plurality of items of evidence for a first candidate answer of the plurality of candidate answers, wherein the first feature score is based on the set of attributes of each feature in the first set of features and each feature in the merged first set of features of the plurality of items of evidence, wherein the first feature score, the set of attributes of each feature in the first set of features, and the merged first set of features of the plurality of items of evidence are related to a first attribute of a plurality of attributes of the plurality of candidate answers and the plurality of items of evidence;

computing a second feature score for the merged second set of features for the plurality of items of evidence for the first candidate answer, wherein the second feature score is based on;

(i) the set of attributes of each feature in the second set of features, (ii) the first feature score for the merged first set of features for the respective item of evidence, and (iii) each feature in the merged second set of features of the plurality of items of evidence, wherein the second feature score, the set of attributes of each feature in the second set of features, and the merged second set of features of the plurality of items of evidence are related to a second attribute of the plurality of attributes;

selecting a first merging technique, from a plurality of merging techniques, for merging the first and second feature scores, wherein the first merging technique comprises weighting the merged feature score based on the first and second feature scores; and

computing a merged feature score for the first candidate answer based on the first merging technique applied to the first and second feature scores for each of the plurality of items of evidence.

10. The method of claim 9, wherein the first feature score, the second feature score, and the merged feature score are of a plurality of possible values for each respective score, wherein the plurality of merging techniques further comprise:

(i) weighting the merged feature score based on the first feature score, (ii) weighting the merged feature score based on the second feature score, (iii) not considering the first feature score upon determining the first feature score does not exceed a first weight threshold for the first feature score, (iv) considering the first feature score upon determining the first feature score exceeds a second weight threshold for the first feature score, different than the first weight threshold for the first feature score, (v) not considering the second feature score upon determining the second feature score does not exceed a first weight threshold for the second feature score, (vi) considering the second feature score upon determining the second feature score exceeds a second weight threshold for the second feature score, different than the first weight threshold for the first feature score, wherein the plurality of features comprise;

(i) a quality, (ii) a relevance, (iii) a sentiment, and (iv) a source of each candidate answer and each item of evidence, wherein the operation further comprises prior to computing the merged feature score;

selecting a second merging technique of the plurality of merging techniques, for merging the first and second feature scores,

wherein the merged feature score for the first candidate answer is computed based on the first and second merging techniques applied to the first and second feature scores for each of the plurality of items of evidence.