Utilizing failures in question and answer system responses to enhance the accuracy of question and answer systems
First Claim
1. A device, comprising:
- a question-answer system comprising software for performing question answering processes;
a receiver receiving a question into said question-answer system;
a processor connected to said question-answer system, said processor;
generating candidate answers to said question,determining a confidence score for each of said candidate answers,evaluating sources of evidence used to generate said candidate answers,identifying missing information from a corpus of data, said missing information comprising any information that improves the ability of said question-answer system to understand and evaluate said sources of evidence associated with said candidate answers, said identifying said missing information comprising;
evaluating a piece of evidence and producing an evidence score according to a scoring process that reflects a degree to which said piece of evidence supports or refutes said candidate answer,classifying said piece of evidence as marginal evidence if said score is below a threshold value, andevaluating said marginal evidence to identify said missing information, andgenerating a follow-on inquiry based on said missing information; and
a network interface outputting said follow-on inquiry to external sources separate from said question-answer system to obtain responses to said follow-on inquiry and receiving from said external sources a response to said follow-on inquiry,said processor inputting said response into said corpus of data.
1 Assignment
0 Petitions
Accused Products
Abstract
A computerized device for enhancing the accuracy of a question-answer system is disclosed. The computerized device comprises a question-answer system comprising software for performing a plurality of question answering processes. A receiver receives a question into the question-answer system. A processor that generates a plurality of candidate answers to the question is connected to the question-answer system. The processor determines a confidence score for each of the plurality of candidate answers. The processor evaluates sources of evidence used to generate the plurality of candidate answers. The processor identifies missing information from a corpus of data. The missing information comprises any information that improves a confidence score for a candidate answer. The processor generates at least one follow-on inquiry based on the missing information. A network interface outputs the at least one follow-on inquiry to external sources separate from the question-answer system.
-
Citations
20 Claims
-
1. A device, comprising:
-
a question-answer system comprising software for performing question answering processes; a receiver receiving a question into said question-answer system; a processor connected to said question-answer system, said processor; generating candidate answers to said question, determining a confidence score for each of said candidate answers, evaluating sources of evidence used to generate said candidate answers, identifying missing information from a corpus of data, said missing information comprising any information that improves the ability of said question-answer system to understand and evaluate said sources of evidence associated with said candidate answers, said identifying said missing information comprising; evaluating a piece of evidence and producing an evidence score according to a scoring process that reflects a degree to which said piece of evidence supports or refutes said candidate answer, classifying said piece of evidence as marginal evidence if said score is below a threshold value, and evaluating said marginal evidence to identify said missing information, and generating a follow-on inquiry based on said missing information; and a network interface outputting said follow-on inquiry to external sources separate from said question-answer system to obtain responses to said follow-on inquiry and receiving from said external sources a response to said follow-on inquiry, said processor inputting said response into said corpus of data. - View Dependent Claims (2, 3, 4, 5, 6)
-
-
7. A question answering (QA) system comprising:
-
a processor; an evidence analysis module connected to said processor; a first interface connected to said processor; a second interface connected to said processor; and a corpus of data connected to said evidence analysis module, said corpus of data comprising a total amount of evidence, said evidence analysis module classifying said total amount of evidence as good evidence or marginal evidence, said first interface receiving a first question to be answered by said QA system, said processor creating a collection of candidate answers to said first question from said corpus of data, each said candidate answer having supporting evidence and a confidence score generated by said processor, said good evidence having an evidence score above a previously established evidence threshold value, and enabling said QA system to provide a candidate answer to said first question with a confidence score above a previously established confidence threshold value, and said marginal evidence having an evidence score below said previously established evidence threshold value, enabling said QA system to provide a candidate answer to said first question with a confidence score below said previously established confidence threshold value, and requiring said QA system to obtain additional information to improve said confidence score for said candidate answer to said first question, said processor identifying missing information from said corpus of data, said missing information comprising any information that improves the ability of said QA system to understand and evaluate said supporting evidence associated with said collection of candidate answers, said evidence analysis module producing a second question based on said missing information, said processor presenting said second question through said second interface to external sources separate from said QA system to obtain responses to said second question, said processor receiving a response or knowledge item from said external sources through said second interface, said processor inputting said response or knowledge item into said corpus of data, and said processor automatically developing additional logical rules and additional evidence for said QA system based on said response or knowledge item. - View Dependent Claims (8, 9, 10, 11)
-
-
12. A method, comprising:
-
generating candidate answer to a question determining a confidence score for each of said candidate answers; evaluating sources of evidence used to generate said candidates answers; identifying missing information from a corpus of data, said missing information comprising any information that improves the ability of a question-answer system to understand and evaluate sources of evidence associated with a candidate answer; generating a follow-on inquiry, said follow-on inquiry prompting for said missing information to be provided; outputting said follow-on inquiry to an external source; receiving a response to said follow-on inquiry from said external source; and adding said response to said corpus of data. - View Dependent Claims (13, 14, 15, 16, 17, 18, 19, 20)
-
Specification