Automated Answer Scoring Based on Combination of Informativity and Specificity Metrics

• US 20170364519A1
• Filed: 06/15/2016
• Published: 12/21/2017
• Est. Priority Date: 06/15/2016
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Beller, Charles E., Chase, JR., Paul J., Darden, Richard L., Drzewucki, Michael, Katz, Edward G.
• Application Number:
  US15/182,837
• Publication Number:
  US 20170364519A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 40/20 Natural language analysis semantic analysis of natural language G06F40/30
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/041 Abduction
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

A mechanism is provided in a computing device configured with instructions executing on a processor of the computing device to implement a question answering system for answer scoring based on a specificity score. The question answering system, executing on the processor of the computing device and configured with a question answering machine learning model, generates a set of candidate answers for a user-generated input question. For each given candidate answer in the set of candidate answers, a specificity scorer of the question answering system determines a specificity value of each term in the given candidate answer based on a position of the term in a taxonomy data structure and determines a specificity score of the given candidate answer based on the specificity value of the terms in the given candidate answer. The question answering system, determines a confidence score for each candidate answer within the set of candidate answers based on its specificity score. The question answering system ranks the set of candidate answers according to confidence score to form a ranked set of candidate answers and returns the ranked set of candidate answers.

Claims

1. A method, in a computing device configured with instructions executing on a processor of the computing device to implement a question answering system, for answer scoring based on a specificity score, the method comprising:

generating, by the question answering system executing on the processor of the computing device and configured with a question answering machine learning model, a set of candidate answers for a user-generated input question;

for each given candidate answer in the set of candidate answers, determining, by a specificity scorer of the question answering system, a specificity value of each term in the given candidate answer based on a position of the term in a taxonomy data structure and determining a specificity score of the given candidate answer based on the specificity value of the terms in the given candidate answer;

determining, by the question answering system, a confidence score for each candidate answer within the set of candidate answers based on its specificity score;

ranking, by the question answering system, the set of candidate answers according to confidence score to form a ranked set of candidate answers; and

returning, by the question answering system, the ranked set of candidate answers.

2. The method of claim 1, wherein each node of the taxonomy data structure is assigned a specificity value.

3. The method of claim 2, wherein each node of the taxonomy data structure has an associated informativity value;

and

wherein determining the specificity value of each term in the given candidate answer comprises, responsive to the specificity scorer determining that a given term in the given candidate answer does not occur in the taxonomy data structure;

determining an informativity value for the given term using corpus statistics;

aligning the given term with a node in the taxonomy data structure based on informativity value; and

assigning specificity value of the node in the taxonomy data structure to be the specificity value of the given term.

4. The method of claim 3, wherein determining the informativity value for the given term comprises determining an inverse Zipfian ranking of the given term as the informativity value.

5. The method of claim 3, wherein an informativity value of a given taxonomic group within the taxonomy data structure is an average of informativity values of member nodes in the taxonomic group.

6. The method of claim 2, wherein specificity values of the taxonomy data structure are determined heuristically or using a machine learning approach.

7. The method of claim 1, wherein determining the specificity score of the given candidate answer comprises determining a highest specificity value of the terms in the given candidate answer to be the specificity score of the given candidate answer.

8. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program comprises instructions, which when executed on a processor of a computing device causes the computing device to implement a question answering system for answer scoring based on a specificity score, wherein the computer readable program causes the computing device to:

generate, by the question answering system executing on the processor of the computing device and configured with a question answering machine learning model, a set of candidate answers for a user-generated input question;

for each given candidate answer in the set of candidate answers, determine, by a specificity scorer of the question answering system, a specificity value of each term in the given candidate answer based on a position of the term in a taxonomy data structure and determine a specificity score of the given candidate answer based on the specificity value of the terms in the given candidate answer;

determine, by the question answering system, a confidence score for each candidate answer within the set of candidate answers based on its specificity score;

rank, by the question answering system, the set of candidate answers according to confidence score to form a ranked set of candidate answers; and

return, by the question answering system, the ranked set of candidate answers.

9. The computer program product of claim 8, wherein each node of the taxonomy data structure is assigned a specificity value.

10. The computer program product of claim 9, wherein each node of the taxonomy data structure has an associated informativity value;

and

wherein determining the specificity value of each term in the given candidate answer comprises, responsive to the specificity scorer determining that a given term in the given candidate answer does not occur in the taxonomy data structure;

determining an informativity value for the given term using corpus statistics;

aligning the given term with a node in the taxonomy data structure based on informativity value; and

assigning specificity value of the node in the taxonomy data structure to be the specificity value of the given term.

11. The computer program product of claim 10, wherein determining the informativity value for the given term comprises determining an inverse Zipfian ranking of the given term as the informativity value.

12. The computer program product of claim 10, wherein an informativity value of a given taxonomic group within the taxonomy data structure is an average of informativity values of member nodes in the taxonomic group.

13. The computer program product of claim 9, wherein specificity values of the taxonomy data structure are determined heuristically or using a machine learning approach.

14. The computer program product of claim 8, wherein determining the specificity score of the given candidate answer comprises determining a highest specificity value of the terms in the given candidate answer to be the specificity score of the given candidate answer.

15. A computing device comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions, which when executed on a processor of a computing device causes the computing device to implement a question answering system for answer scoring based on a specificity score, wherein the instructions cause the processor to;

generate, by the question answering system executing on the processor of the computing device and configured with a question answering machine learning model, a set of candidate answers for a user-generated input question;

for each given candidate answer in the set of candidate answers, determine, by a specificity scorer of the question answering system, a specificity value of each term in the given candidate answer based on a position of the term in a taxonomy data structure and determine a specificity score of the given candidate answer based on the specificity value of the terms in the given candidate answer;

determine, by the question answering system, a confidence score for each candidate answer within the set of candidate answers based on its specificity score;

rank, by the question answering system, the set of candidate answers according to confidence score to form a ranked set of candidate answers; and

return, by the question answering system, the ranked set of candidate answers.

16. The computing device of claim 15, wherein each node of the taxonomy data structure is assigned a specificity value.

17. The computing device of claim 16, wherein each node of the taxonomy data structure has an associated informativity value;

and

wherein determining the specificity value of each tern in the given candidate answer comprises, responsive to the specificity scorer determining that a given term in the given candidate answer does not occur in the taxonomy data structure;

determining an informativity value for the given term using corpus statistics;

aligning the given term with a node in the taxonomy data structure based on informativity value; and

assigning specificity value of the node in the taxonomy data structure to be the specificity value of the given term.

18. The computing device of claim 17, wherein determining the informativity value for the given term comprises determining an inverse Zipfian ranking of the given term as the informativity value.

19. The computing device of claim 17, wherein an informativity value of a given taxonomic group within the taxonomy data structure is an average of informativity values of member nodes in the taxonomic group.

20. The computing device of claim 15, wherein determining the specificity score of the given candidate answer comprises determining a highest specificity value of the terms in the given candidate answer to be the specificity score of the given candidate answer.