SYSTEM AND METHOD FOR DOMAIN ADAPTATION IN QUESTION ANSWERING
First Claim
Patent Images
1. A method for providing adaptation to a question answering system, wherein the question answering system has associated therewith a question-answer set, the method comprising the steps of:
- submitting a set of questions to the question answering system;
receiving back from the question answering system a set of answers generated in response to the set of questions;
comparing the set of answers received back from the question answering system to answers in the question-answer set;
identifying, based on the comparison, a plurality of answers from the set of answers received back from the question answering system that are not correct;
generating a plurality of groups by performing automated grouping on at least one of;
(a) a plurality of questions from the question-answer set that correspond to the identified answers that are not correct; and
(b) a plurality of answers from the question-answer set that correspond to the identified answers that are not correct;
creating a collection of related terms associated with the groups;
obtaining, from a corpus of data, textual information about each of the related terms; and
creating a plurality of textual resources from the obtained information.
1 Assignment
0 Petitions
Accused Products
Abstract
The present disclosure relates generally to question answering systems and methods and, particularly, to systems and methods for domain adaptation in question answering.
-
Citations
39 Claims
-
1. A method for providing adaptation to a question answering system, wherein the question answering system has associated therewith a question-answer set, the method comprising the steps of:
-
submitting a set of questions to the question answering system; receiving back from the question answering system a set of answers generated in response to the set of questions; comparing the set of answers received back from the question answering system to answers in the question-answer set; identifying, based on the comparison, a plurality of answers from the set of answers received back from the question answering system that are not correct; generating a plurality of groups by performing automated grouping on at least one of;
(a) a plurality of questions from the question-answer set that correspond to the identified answers that are not correct; and
(b) a plurality of answers from the question-answer set that correspond to the identified answers that are not correct;creating a collection of related terms associated with the groups; obtaining, from a corpus of data, textual information about each of the related terms; and creating a plurality of textual resources from the obtained information. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 33)
-
-
10. A method for providing adaptation to a question answering system, wherein the question answering system has associated therewith a question-answer set, the method comprising the steps of:
-
submitting a set of questions to the question answering system; receiving back from the question answering system a set of answers generated in response to the set of questions; comparing the set of answers received back from the question answering system to answers in the question-answer set; identifying, based on the comparison, a plurality of answers from the set of answers received back from the question answering system that are not correct; determining for each of the answers from the set of answers received back from the question answering system that are not correct a missing semantic data type, wherein the determining is performed by using a plurality of ontologies; and automatically creating a software program to run against a corpus of data to create at least one textual resource about each missing semantic data type; wherein the automatic creation comprises; (a) for each question with an answer that is not correct, obtaining a corresponding correct answer and computing for each corresponding correct answer an associated semantic data type; (b) for each semantic data type associated with a corresponding correct answer, finding a collection of words or expressions that are related to the semantic data type; and (c) for each the words or expressions that are in the collection that is found, obtaining additional related information from the world wide web and creating a new textual resource with the additional information. - View Dependent Claims (11, 12, 13, 14, 15, 16, 17, 18, 34)
-
-
19. A method for providing adaptation to a question answering system, wherein the question answering system has associated therewith a plurality of trace data and a question-answer set, the method comprising the steps of:
-
submitting a set of questions to the question answering system; receiving back from the question answering system a set of answers generated in response to the set of questions; comparing the set of answers received back from the question answering system to answers in the question-answer set; and generating, based on the comparison and the plurality of trace data, an estimate of how much more training data is needed for each of a plurality of question types or answer types; wherein the training data comprises a plurality of training questions along with a corresponding plurality of correct training answers; wherein the trace data is generated during the generation of the set of answers; and wherein the step of generating further comprises the following steps; (a) for each question type, successively sample an increasing number of question-answer pairs from the question-answer set; (b) automatically train the question answering system using the sampled question-answer pairs; (c) automatically compute a functional dependence of the question answering system performance on the remaining questions relative to the size of the sample; and (d) extrapolate from the functional dependence a required number of training question-answer pairs of each question type required. - View Dependent Claims (20, 35)
-
-
21. A method for providing adaptation to a question answering system, wherein the question answering system has associated therewith a plurality of trace data and a question-answer set, the method comprising the steps of:
-
submitting a set of questions to the question answering system; receiving back from the question answering system a set of answers generated in response to the set of questions; comparing the set of answers received back from the question answering system to answers in the question-answer set; and generating, based on the comparison and the plurality of trace data, a plurality of proposals, wherein each of the proposals identifies at least one available feature and at least one available scorer that should be used in answering each of a plurality of question types; wherein the trace data is generated during the generation of the set of answers; and wherein the step of generating further comprises the following steps; (a) automatically computing a correlation between the use of at least one of;
(i) one or more features and system performance on the set of questions of this type; and
(ii) one or more scorers and system performance on the set of questions of this type; and(b) automatically identifying at least one of;
(i) one or more features having a correlation with system performance; and
(b) one or more scorers having a correlation with system performance. - View Dependent Claims (22, 36)
-
-
23. A method for providing adaptation to a question answering system, wherein the question answering system has associated therewith a plurality of trace data and a question-answer set, the method comprising the steps of:
-
submitting a set of questions to the question answering system; receiving back from the question answering system a set of answers generated in response to the set of questions; comparing the set of answers received back from the question answering system to answers in the question-answer set; and generating, based on the comparison and the plurality of trace data, an estimate of at least one of;
(a) headroom per feature;
(b) headroom per feature set; and
(c) headroom per content type;wherein the trace data is generated during the generation of the set of answers. - View Dependent Claims (24, 25, 26, 37)
-
-
27. A method for providing adaptation to a question answering system, wherein the question answering system has associated therewith a plurality of trace data and a question-answer set, the method comprising the steps of:
-
submitting a set of questions to the question answering system; receiving back from the question answering system a set of answers generated in response to the set of questions; comparing the set of answers received back from the question answering system to answers in the question-answer set; identifying, based on the comparison, a plurality of answers from the set of answers received back from the question answering system that are not correct; identifying, based on the answers that are not correct and the plurality of trace data, at least one type of functional adaptation required; wherein the trace data is generated during the generation of the set of answers; and wherein the step of identifying at least one functional adaptation further comprises the following steps; (a) identifying at least one of;
(i) at least one question feature in the answers that are not correct; and
(ii) at least one answer feature in the answers that are not correct;(b) comparing each identified feature to at least one feature in an annotator; and (c) based on the comparison made in step b, identifying at least one of;
(i) a question feature that is not covered by at least one annotator; and
(ii) an answer feature that is not covered by at least one annotator. - View Dependent Claims (28, 29, 30, 31, 38)
-
-
32. A method for providing adaptation to a question answering system, wherein the question answering system has associated therewith at least one tool comprising a database containing trace data and a database query system, the method comprising the steps of:
-
submitting a set of questions to the question answering system; receiving back from the question answering system a set of answers generated in response to the set of questions; comparing the set of answers received back from the question answering system to answers in the question-answer set; creating at least one new field in the database; automatically collecting trace data about the value of the new field; and permitting querying on the new field; wherein the trace data is generated during the generation of the set of answers. - View Dependent Claims (39)
-
Specification