SYSTEM AND METHOD FOR DOMAIN ADAPTATION IN QUESTION ANSWERING

• US 20120077178A1
• Filed: 09/24/2011
• Published: 03/29/2012
• Est. Priority Date: 05/14/2008
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Bagchi, Sugato, Ferrucci, David A., Gondek, David C., Zadrozny, Wlodek W., Levas, Anthony T.
• Granted Patent: US 9,240,128 B2
• Time in Patent Office: Days
• US Class Current: 434/362
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F

Abstract

The present disclosure relates generally to question answering systems and methods and, particularly, to systems and methods for domain adaptation in question answering.

Claims

1. A method for providing adaptation to a question answering system, wherein the question answering system has associated therewith a question-answer set, the method comprising the steps of:

submitting a set of questions to the question answering system;

receiving back from the question answering system a set of answers generated in response to the set of questions;

comparing the set of answers received back from the question answering system to answers in the question-answer set;

identifying, based on the comparison, a plurality of answers from the set of answers received back from the question answering system that are not correct;

generating a plurality of groups by performing automated grouping on at least one of;

(a) a plurality of questions from the question-answer set that correspond to the identified answers that are not correct; and

(b) a plurality of answers from the question-answer set that correspond to the identified answers that are not correct;

creating a collection of related terms associated with the groups;

obtaining, from a corpus of data, textual information about each of the related terms; and

creating a plurality of textual resources from the obtained information.

2. The method of claim 1, wherein each of the answers from the set of answers received back from the question answering system that are not correct comprises one of:

(a) an answer that is incorrect; and

(b) an answer that is non-existent.

3. The method of claim 1, wherein the generating a plurality of groups comprises using at least one ontology.

4. The method of claim 1, wherein the generating a plurality of groups comprises at least one of:

(a) clustering; and

(b) classifying.

5. The method of claim 1, wherein the generating a plurality of groups comprises performing automated grouping on each of:

(a) a plurality of questions from the question-answer set that correspond to the identified answers that are not correct; and

(b) a plurality of answers from the question-answer set that correspond to identified answers that are not correct.

6. The method of claim 1, wherein the obtaining comprises at least one of:

(a) obtaining from the world wide web; and

(b) obtaining from an e-commerce source.

7. The method of claim 1, wherein the creating a plurality of textual resources from the obtained information comprises creating at least one of:

(a) at least one n-gram collection;

(b) at least one lexicalized relation resource; and

(c) at least one new text document using information obtained from at least one of;

(i) the world wide web; and

(ii) an e-commerce source.

8. The method of claim 1, wherein the question answering system has associated therewith a first corpus of data and the corpus of data from which the textual information is obtained is a second corpus of data.

9. The method of claim 8, wherein the second corpus of data is external relative to the first corpus of data.

33. The method of claim 1, wherein the steps are carried out in the order recited.

10. A method for providing adaptation to a question answering system, wherein the question answering system has associated therewith a question-answer set, the method comprising the steps of:

submitting a set of questions to the question answering system;

receiving back from the question answering system a set of answers generated in response to the set of questions;

comparing the set of answers received back from the question answering system to answers in the question-answer set;

identifying, based on the comparison, a plurality of answers from the set of answers received back from the question answering system that are not correct;

determining for each of the answers from the set of answers received back from the question answering system that are not correct a missing semantic data type, wherein the determining is performed by using a plurality of ontologies; and

automatically creating a software program to run against a corpus of data to create at least one textual resource about each missing semantic data type;

wherein the automatic creation comprises;

(a) for each question with an answer that is not correct, obtaining a corresponding correct answer and computing for each corresponding correct answer an associated semantic data type;

(b) for each semantic data type associated with a corresponding correct answer, finding a collection of words or expressions that are related to the semantic data type; and

(c) for each the words or expressions that are in the collection that is found, obtaining additional related information from the world wide web and creating a new textual resource with the additional information.

11. The method of claim 10, wherein each of the answers from the set of answers received back from the question answering system that are not correct comprises one of:

(a) an answer that is incorrect; and

(b) an answer that is non-existent.

12. The method of claim 10, wherein the automatically creating a software program to run against the first corpus of data comprises automatically creating a software program that uses existing components.

13. The method of claim 10, wherein the software program comprises a composite annotator.

14. The method of claim 10, wherein each corresponding correct answer is obtained from the question-answer set.

15. The method of claim 10, wherein the computation comprises a look up in an ontology.

16. The method of claim 10, wherein the computing comprises finding in a corpus of data one or more expressions and treating a portion of each expression as a semantic type.

17. The method of claim 10, wherein the finding a collection of words or expressions that are related to the semantic data type comprises searching a corpus or the Internet.

18. The method of claim 10, wherein the finding a collection of words or expressions that are related to the semantic data type comprises using an ontology.

34. The method of claim 10, wherein the steps are carried out in the order recited.

19. A method for providing adaptation to a question answering system, wherein the question answering system has associated therewith a plurality of trace data and a question-answer set, the method comprising the steps of:

submitting a set of questions to the question answering system;

receiving back from the question answering system a set of answers generated in response to the set of questions;

comparing the set of answers received back from the question answering system to answers in the question-answer set; and

generating, based on the comparison and the plurality of trace data, an estimate of how much more training data is needed for each of a plurality of question types or answer types;

wherein the training data comprises a plurality of training questions along with a corresponding plurality of correct training answers;

wherein the trace data is generated during the generation of the set of answers; and

wherein the step of generating further comprises the following steps;

(a) for each question type, successively sample an increasing number of question-answer pairs from the question-answer set;

(b) automatically train the question answering system using the sampled question-answer pairs;

(c) automatically compute a functional dependence of the question answering system performance on the remaining questions relative to the size of the sample; and

(d) extrapolate from the functional dependence a required number of training question-answer pairs of each question type required.

20. The method of claim 19, wherein the plot of performance relates to at least one of:

(a) accuracy; and

(b) recall.

35. The method of claim 19, wherein the steps are carried out in the order recited.

21. A method for providing adaptation to a question answering system, wherein the question answering system has associated therewith a plurality of trace data and a question-answer set, the method comprising the steps of:

submitting a set of questions to the question answering system;

receiving back from the question answering system a set of answers generated in response to the set of questions;

comparing the set of answers received back from the question answering system to answers in the question-answer set; and

generating, based on the comparison and the plurality of trace data, a plurality of proposals, wherein each of the proposals identifies at least one available feature and at least one available scorer that should be used in answering each of a plurality of question types;

wherein the trace data is generated during the generation of the set of answers; and

wherein the step of generating further comprises the following steps;

(a) automatically computing a correlation between the use of at least one of;

(i) one or more features and system performance on the set of questions of this type; and

(ii) one or more scorers and system performance on the set of questions of this type; and

(b) automatically identifying at least one of;

(i) one or more features having a correlation with system performance; and

(b) one or more scorers having a correlation with system performance.

22. The method of claim 21 wherein the step of identifying comprises identifying at least one of:

(i) one or more features having a correlation with system performance above a threshold; and

(b) one or more scorers having a correlation with system performance above a threshold.

36. The method of claim 21, wherein the steps are carried out in the order recited.

23. A method for providing adaptation to a question answering system, wherein the question answering system has associated therewith a plurality of trace data and a question-answer set, the method comprising the steps of:

submitting a set of questions to the question answering system;

receiving back from the question answering system a set of answers generated in response to the set of questions;

comparing the set of answers received back from the question answering system to answers in the question-answer set; and

generating, based on the comparison and the plurality of trace data, an estimate of at least one of;

(a) headroom per feature;

(b) headroom per feature set; and

(c) headroom per content type;

wherein the trace data is generated during the generation of the set of answers.

24. The method of claim 23, wherein the headroom per feature comprises a possible improvement relative to a feature having perfect performance on the question-answer set.

25. The method of claim 23, wherein the headroom per feature set comprises a possible improvement relative to a feature set having perfect performance on the question-answer set.

26. The method of claim 23, wherein the headroom per content type comprises a possible improvement relative to possible addition of new content for the question-answer set.

37. The method of claim 23, wherein the steps are carried out in the order recited.

27. A method for providing adaptation to a question answering system, wherein the question answering system has associated therewith a plurality of trace data and a question-answer set, the method comprising the steps of:

submitting a set of questions to the question answering system;

receiving back from the question answering system a set of answers generated in response to the set of questions;

comparing the set of answers received back from the question answering system to answers in the question-answer set;

identifying, based on the comparison, a plurality of answers from the set of answers received back from the question answering system that are not correct;

identifying, based on the answers that are not correct and the plurality of trace data, at least one type of functional adaptation required;

wherein the trace data is generated during the generation of the set of answers; and

wherein the step of identifying at least one functional adaptation further comprises the following steps;

(a) identifying at least one of;

(i) at least one question feature in the answers that are not correct; and

(ii) at least one answer feature in the answers that are not correct;

(b) comparing each identified feature to at least one feature in an annotator; and

(c) based on the comparison made in step b, identifying at least one of;

(i) a question feature that is not covered by at least one annotator; and

(ii) an answer feature that is not covered by at least one annotator.

28. The method of claim 27, wherein the annotator is a scorer.

29. The method of claim 27, wherein each of the answers from the set of answers received back from the question answering system that are not correct comprises one of:

(a) an answer that is incorrect; and

(b) an answer that is non-existent.

30. The method of claim 27, further comprising proposing, based upon the at least one type of functional adaptation identified, at least one of:

(a) a collection of features; and

(b) at least one new annotator.

31. The method of claim 30, wherein the new annotator is a scorer.

38. The method of claim 27, wherein the steps are carried out in the order recited.

32. A method for providing adaptation to a question answering system, wherein the question answering system has associated therewith at least one tool comprising a database containing trace data and a database query system, the method comprising the steps of:

submitting a set of questions to the question answering system;

receiving back from the question answering system a set of answers generated in response to the set of questions;

comparing the set of answers received back from the question answering system to answers in the question-answer set;

creating at least one new field in the database;

automatically collecting trace data about the value of the new field; and

permitting querying on the new field;

wherein the trace data is generated during the generation of the set of answers.

39. The method of claim 32, wherein the steps are carried out in the order recited.