LEXICAL ANSWER TYPE CONFIDENCE ESTIMATION AND APPLICATION

• US 20130282363A1
• Filed: 06/18/2013
• Published: 10/24/2013
• Est. Priority Date: 09/24/2010
• Status: Active Grant

• Current Assignee: Hyundai Motor Company (Hyundai Motor Group), Kia Corp.
• Original Assignee: International Business Machines Corporation
• Inventors: Fan, James J., Ferrucci, David A., Kalyanpur, Aditya A., Lally, Adam P., Murdock, James W., Zadrozny, Wlodek W., Gondek, David C.
• Granted Patent: US 8,943,051 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 40/55 Rule-based translation
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F

Abstract

A system, method and computer program product for automatically estimating the confidence of a detected LAT to provide a more accurate overall score for an obtained candidate answer. A confidence “score” or value of each detected LAT is obtained, and the system and method performs combining the confidence score with a degree of match between a LAT and an AnswerType of the candidate answer to provide improved overall score for the candidate answer.

Claims

1. A method for extracting features from a query comprising a text string, said method comprising:

identifying a syntactic pattern rule associated with said query, said pattern rule identified from a set of pattern rules that define common lexical answer types (LATs), a first feature of said extracted features comprising an identified pattern rule;

checking for prior instances of a detected lexical answer type (LAT) and computing a LAT word frequency based on said prior instances, a second feature of said extracted features comprising a computed frequency of a query word being a candidate LAT;

obtaining a parse tree data structure associated with said query;

identifying grammatical relations amongst words associated with said candidate LAT in said parse tree structure, a third feature of said extracted features comprising a part of speech of the candidate LAT,

determining whether the candidate LAT word co-references some other word in said query recognized as a LAT, a fourth feature of said extracted features comprising a co-reference information, and

applying a model to said extracted first, second, third and fourth features to produce a confidence value representing an estimated accuracy of a detected query LAT,

wherein one or more programmed processor devices performs said identifying a syntactic pattern rule, checking for prior instances, obtaining the parse tree data structure, identifying grammatical relations, and determining LAT word co-references, and model applying.

2. The method of claim 1, wherein said one or more programmed processor devices performs extracting said one or more first, second, third and fourth features in parallel.

3. The method of claim 2, wherein said applied model is one or more of:

a logistic regression model or a decision tree model constructed manually or automatically from prior sets of training instances.

4. The method of claim 2, wherein said identifying grammatical relations amongst words comprises:

traversing said parse tree.

5. The method of claim 2, further comprising:

computing a first score based on said confidence value of a detected LAT associated with said query, wherein said first score is indicative of a quality of said query LAT;

obtaining a candidate answer to said query from an information source;

determining an answer type (AT) for said candidate answer;

comparing said detected query LAT with said candidate answer AT;

computing a second score representing a degree of match between said compared query LAT with said candidate answer AT; and

combining said first score and said second score to provide a total score indicative of a quality of said candidate answer.

wherein one or more programmed processor devices performs one or more of said computing said first score and second score, obtaining candidate answers, determining an AT, comparing and combining.

6. The method of claim 5, wherein said computing a second score representing a degree of match between said compared query LAT with said candidate answer AT includes:

matching said candidate answer against instances in a corpus or a knowledge base;

retrieving a type associated with an instance in the corpus or the knowledge base; and

,

matching said query LAT with said retrieved type to produce a type coercion (TyCor) value, said second score comprising said TyCor value.

7. The method of claim 6, wherein a query has up to n candidate LATs, said combining to provide said total score is a summation according to:

(confidence score_LAT1)(TyCor_{LAT 1})+(confidence score_LAT2)(TyCor_{LAT 2})+ . . . +(confidence score_LATn)(TyCor_LATn)

wherein confidence score_LAT1, confidence score_LAT2, and confidence score_LATn are produced confidence values representing estimated accuracies of each determined LAT of said query, and TyCor_LAT1, TyCor_LAT2, and TyCor_LATn are the produced type coercion (TyCor) values of a candidate answer with each LAT.

8. A system for extracting features from a query comprising a text string, said system comprising:

a memory storage device;

one or more processor devices, each in communication to said memory device and configured to perform a method to;

identify a syntactic pattern rule associated with said query, said pattern rule identified from a set of pattern rules that define common lexical answer types (LATs), a first feature of said extracted features comprising an identified pattern rule;

check for prior instances of a detected lexical answer type (LAT) and computing a LAT word frequency based on said prior instances, a second feature of said extracted features comprising a computed frequency of a query word being a candidate LAT;

obtain a parse tree data structure associated with said query;

identify grammatical relations amongst words associated with said candidate LAT in said parse tree structure, a third feature of said extracted features comprising a part of speech of the candidate LAT,

determine whether the candidate LAT word co-references some other word in said query recognized as a LAT, a fourth feature of said extracted features comprising a co-reference information, and

apply a model to said extracted first, second, third and fourth features to produce a confidence value representing an estimated accuracy of a detected query LAT.

9. The system of claim 8, wherein said one or more programmed processor devices performs extracting said one or more first, second, third and fourth features in parallel.

10. The system of claim 9, wherein said applied model is one or more of:

a logistic regression model or a decision tree model constructed manually or automatically from prior sets of training instances.

11. The system of claim 9, wherein said identifying grammatical relations amongst words comprises:

traversing said parse tree.

12. The system of claim 9, further configured to perform a method to:

compute a first score based on said confidence value of a detected LAT associated with said query, wherein said first score is indicative of a quality of said query LAT;

obtain a candidate answer to said query from an information source;

determine an answer type (AT) for said candidate answer;

compare said detected query LAT with said candidate answer AT;

compute a second score representing a degree of match between said compared query LAT with said candidate answer AT; and

combine said first score and said second score to provide a total score indicative of a quality of said candidate answer.

13. The system of claim 12, wherein to compute a second score representing a degree of match between said compared query LAT with said candidate answer AT includes further configuring said one or more processor devices to:

match said candidate answer against instances in a corpus or a knowledge base;

retrieve a type associated with an instance in the corpus or the knowledge base; and

,

match said query LAT with said retrieved type to produce a type coercion (TyCor) value, said second score comprising said TyCor value.

14. The system of claim 13, wherein a query has up to n candidate LATs, said combining to provide said total score is a summation according to:

(confidence score_LAT1)(TyCor_{LAT 1})+(confidence score_LAT2)(TyCor_{LAT 2})+ . . . +(confidence score_LATn)(TyCor_LATn)

wherein confidence score_LAT1, confidence score_LAT2, and confidence score_LATn are produced confidence values representing estimated accuracies of each determined LAT of said query, and TyCor_LAT1, TyCor_LAT2, and TyCor_LATn are the produced type coercion (TyCor) values of a candidate answer with each LAT.

15. A computer program product for automatically extracting features from a query comprising a text string, the computer program device comprising a storage medium readable by a processing circuit and storing instructions run by the processing circuit for performing a method, the method comprising:

identifying a syntactic pattern rule associated with said query, said pattern rule identified from a set of pattern rules that define common lexical answer types (LATs), a first feature of said extracted features comprising an identified pattern rule;

checking for prior instances of a detected lexical answer type (LAT) and computing a LAT word frequency based on said prior instances, a second feature of said extracted features comprising a computed frequency of a query word being a candidate LAT;

obtaining a parse tree data structure associated with said query;

identifying grammatical relations amongst words associated with said candidate LAT in said parse tree structure, a third feature of said extracted features comprising a part of speech of the candidate LAT,

determining whether the candidate LAT word co-references some other word in said query recognized as a LAT, a fourth feature of said extracted features comprising a co-reference information, and

applying a model to said extracted first, second, third and fourth features to produce a confidence value representing an estimated accuracy of a detected query LAT.

16. The computer program product of claim 15, wherein said one or more programmed processor devices performs extracting said one or more first, second, third and fourth features in parallel.

17. The computer program product of claim 16, wherein said applied model is one or more of:

a logistic regression model or a decision tree model constructed manually or automatically from prior sets of training instances.

18. The computer program product of claim 16, wherein said identifying grammatical relations amongst words comprises:

traversing said parse tree.

19. The computer program product of claim 16, wherein said method further comprises:

computing a first score based on said confidence value of a detected LAT associated with said query, wherein said first score is indicative of a quality of said query LAT;

obtaining a candidate answer to said query from an information source;

determining an answer type (AT) for said candidate answer;

comparing said detected query LAT with said candidate answer AT;

computing a second score representing a degree of match between said compared query LAT with said candidate answer AT; and

combining said first score and said second score to provide a total score indicative of a quality of said candidate answer.

20. The computer program product of claim 19, wherein said computing a second score representing a degree of match between said compared query LAT with said candidate answer AT includes:

matching said candidate answer against instances in a corpus or a knowledge base;

retrieving a type associated with an instance in the corpus or the knowledge base; and

,

matching said query LAT with said retrieved type to produce a type coercion (TyCor) value, said second score comprising said TyCor value.

21. The computer program product of claim 20, wherein a query has up to n candidate LATs, said combining to provide said total score is a summation according to:

(confidence score_LAT1)(TyCor_{LAT 1})+(confidence score_LAT2)(TyCor_{LAT 2})+ . . . +(confidence score_LATn)(TyCor_LATn)

wherein confidence score_LAT1, confidence score_LAT2, and confidence score_LATn are produced confidence values representing estimated accuracies of each determined LAT of said query, and TyCor_LAT1, TyCor_LAT2, and TyCor_LATn are the produced type coercion (TyCor) values of a candidate answer with each LAT.