Utilizing failures in question and answer system responses to enhance the accuracy of question and answer systems

• US 9,047,567 B2
• Filed: 07/12/2012
• Issued: 06/02/2015
• Est. Priority Date: 07/15/2011
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Barborak, Michael A., Chu-Carroll, Jennifer, Ferrucci, David A., Murdock, James W. IV, Zadrozny, Wlodek W.
• Primary Examiner(s): Chang, Li-Wu
• Assistant Examiner(s): Chubb, Mikayla
• Application Number:
  US13/547,492
• Publication Number:
  US 20130017523A1
• Time in Patent Office: 1,055 Days
• US Class Current: 1/1
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 5/041 Abduction
• CPC Class Codes: G09B 7/04 characterised by modifying the teaching programme in response to a wrong answer, e.g. repeating the question, supplying a further explanation

Abstract

A method of enhancing the accuracy of a question-answer system. Missing information from a corpus of data is identified. The missing information is any information that improves a confidence for a candidate answer to a question. A follow-on inquiry is generated. The follow-on inquiry prompts for the missing information to be provided. The follow-on inquiry is output to an external source. A response to the follow-on inquiry is received from the external source. The response is added to the corpus of data.

Claims

1. A method of enhancing the accuracy of a question-answer system, said method comprising:

identifying missing information from a corpus of data, said missing information comprising any information that improves a confidence for a candidate answer to a question, said identifying said missing information comprising;

evaluating a piece of evidence and producing an evidence score according to a scoring process that reflects a degree to which said piece of evidence supports or refutes said candidate answer,

classifying said piece of evidence as marginal evidence if said score is below a threshold value, and

evaluating said marginal evidence to identify said missing information;

generating a follow-on inquiry that prompts for said missing information to be provided;

outputting said follow-on inquiry to an external source;

receiving a response to said follow-on inquiry from said external source; and

adding said response to said corpus of data.

2. The method of claim 1, said missing information comprising a data, a fact, a syntactical relationship, a grammatical relationship, a logical rule, or a taxonomy rule.

3. The method of claim 1, said identifying said missing information comprising evaluating a piece of evidence relevant to said candidate answer to identify said missing information.

4. The method of claim 3, said evaluating a piece of evidence comprising:

parsing said question into a first collection of elements;

parsing said piece of evidence into a second collection of elements;

analyzing said first collection of elements for said question and said piece of evidence in order to determine a relationship between a first element of said first collection of elements for said question and a second element of said second collection of elements for said piece of evidence; and

locating a missing relationship between said first element and said second element, said missing relationship comprising said missing information.

5. The method of claim 4, said missing relationship between said first element and said second element comprising a lexical relationship, a grammatical relationship, or a semantic relationship.

6. The method of claim 1, further comprising:

receiving said question into a question-answer system; and

generating said candidate answer to said question using said question-answer system and said corpus of data.

7. The method of claim 1, wherein said step of evaluating said marginal evidence comprises:

parsing said question into a first collection of elements;

parsing said piece of evidence into a second collection of elements;

analyzing said first collection of elements for said question and said piece of evidence in order to determine a relationship between a first element of said first collection of elements for said question and a second element of said second collection of elements for said piece of evidence; and

locating a missing relationship between said first element and said second element, said missing relationship comprising said missing information.

8. The method of claim 1, further comprising

generating a confidence score for said candidate answer;

evaluating whether said confidence score is below a threshold value; and

if said confidence score is below a threshold value, triggering said step of identifying missing information.

9. The method of claim 1, further comprising re-evaluating said confidence score after said response is added to said corpus of data.

10. The method of claim 1, said external source comprising human sources, electronic sources, or question-answer systems.

11. The method of claim 1, further comprising validating said response is accurate prior to adding said response to said corpus of data.

12. The method of claim 11, said validating said response is accurate comprising comparing said response to a most common response received from other external sources, said response being validated as accurate if said response matches said most common response.

13. A method of enhancing the accuracy of a question-answer system, said method comprising:

receiving a question into a computerized question-answer system operating on a computerized device;

automatically generating, by said computerized device, a plurality of candidate answers to said question;

evaluating, by said computerized device, sources of evidence used to generate said plurality of candidate answers to identify marginal evidence, said marginal evidence contributes only partially to a candidate answer;

determining, by said computerized device, a confidence score for each of said plurality of candidate answers;

automatically identifying, by said computerized device, information not provided by said marginal evidence that could further develop said confidence score;

automatically generating, by said computerized device, at least one follow-on inquiry based on said information;

outputting, by said computerized device, said at least one follow-on inquiry to external sources separate from said question-answer system to obtain responses to said at least one follow-on inquiry;

receiving, by said computerized device, from said external sources at least one response to said at least one follow-on inquiry;

inputting, using said computerized device, said at least one response into said question-answer system; and

automatically developing, by said computerized device, additional logical rules and additional evidence for said computerized question-answer system based on said at least one response to said at least one follow-on inquiry.

14. The method of claim 13, further comprising:

after inputting said at least one response into said question-answer system, re-evaluating said confidence score for said plurality of candidate answers, using said computerized device.

15. The method of claim 13, said question-answer system further comprising a corpus of data containing a plurality of passages, said method further comprising comparing said question to said plurality of passages to provide said plurality of candidate answers to said question, using said computerized device.

16. The method of claim 13, said question-answer system further comprising a corpus of data, said method further comprising extracting evidence related to said question from said corpus of data and classifying said evidence as one of good evidence and marginal evidence, using said computerized device,

said good evidence having an evidence score above a previously established evidence threshold value, and enabling said question-answer system to provide a candidate answer to said question with a confidence score above a previously established confidence threshold value, and

said marginal evidence having an evidence score below said previously established evidence threshold value, enabling said question-answer system to provide a candidate answer to said question with a confidence score below said previously established confidence threshold value, and requiring said question-answer system to obtain additional information to improve said confidence score.

17. The method of claim 13, further comprising evaluating said at least one response received from said external sources, using said computerized device, said evaluating said at least one response comprising at least one of:

identifying a most common response received from said external sources and selecting said most common response;

selecting responses from first outside sources, said first outside sources having a previously established level of accuracy in providing previous responses to previous inquiries;

when said question-answer system comprises a corpus of data, said corpus of data including evidence having been classified by said question-answer system as one of good evidence and bad evidence, said good evidence enabling said question-answer system to provide a candidate answer to said question with a high confidence score and said bad evidence being irrelevant to said question, selecting responses from second outside sources, said second outside sources having a previously established level of accuracy in classifying said evidence as good evidence or bad evidence that matches classification of said evidence by said question-answer system as good evidence and bad evidence; and

selecting responses from third outside sources, said third outside sources having demonstrated a previously established level of accuracy in providing responses that match responses from a known source.

18. A method comprising:

providing a first question to be answered by a Question Answering (QA) system to a processor;

creating, by said processor, a collection of candidate answers to said first question, said collection of candidate answers being created from a corpus of data;

generating, by said processor, supporting evidence and a confidence score for each said candidate answer, said supporting evidence comprising good evidence or marginal evidence,

said good evidence having an evidence score above a previously established evidence threshold value, and enabling said QA system to provide a candidate answer to said first question with a confidence score above a previously established confidence threshold value, and

said marginal evidence having an evidence score below said previously established evidence threshold value, enabling said QA system to provide a candidate answer to said first question with a confidence score below said previously established confidence threshold value, and requiring said QA system to obtain additional information to improve said confidence score for said candidate answer to said first question;

producing, by an evidence analysis module, a second question based on said supporting evidence;

presenting, by said processor, said second question to one or more external sources separate from said QA system to obtain responses to said second question;

receiving, by said processor, at least one response or knowledge item from said one or more external sources;

inputting, by said processor, said at least one response or knowledge item into said corpus of data; and

automatically developing additional logical rules and additional evidence for said QA system based on said at least one response or knowledge item.

19. The method of claim 18, said corpus of data further comprising a plurality of passages, said processor comparing said first question to said plurality of passages to provide said collection of candidate answers.

20. The method of claim 18, further comprising evaluating said at least one response received from said external sources, said evaluating said at least one response comprising at least one of:

identifying a most common response received from said external sources and selecting said most common response;

selecting responses from first outside sources, said first outside sources having a previously established level of accuracy in providing previous responses to previous inquiries;

when said QA system comprises a total amount of evidence, said total amount of evidence having been classified by said QA system as one of good evidence and bad evidence, said good evidence enabling said QA system to provide a correct answer to said question and said bad evidence being irrelevant to said question, selecting responses from second outside sources, said second outside sources having a previously established level of accuracy in classifying said total amount of evidence as good evidence or bad evidence that matches classification of said total amount of evidence by said QA system as good evidence and bad evidence; and

selecting responses from third outside sources, said third outside sources having demonstrated a previously established level of accuracy in providing responses that match responses from a known source.

21. A method comprising:

receiving a question into a Question Answering (QA) system;

said QA system comparing said question to a corpus of data;

said QA system generating hypotheses about relationships between linguistic and semantic entities of said question and said corpus of data;

said QA system generating a plurality of candidate answers to said question using said hypotheses;

said QA system evaluating sources of evidence used to generate said plurality of candidate answers to identify marginal evidence, said marginal evidence contributing only partially to a candidate answer;

said QA system determining a confidence score for each of said plurality of candidate answers;

determining a missing piece of knowledge, said missing piece of knowledge comprising information that would enable said QA system to further develop said confidence score;

formulating a follow-on inquiry to obtain said missing piece of knowledge;

outputting said follow-on inquiry to an external expert community source separate from said QA system to obtain responses to said follow-on inquiry;

receiving from said external expert community source responses to said follow-on inquiry comprising said missing piece of knowledge;

inputting said missing piece of knowledge into said QA system; and

said QA system automatically developing additional logical rules and additional evidence for said QA system based on said obtained missing piece of information.

22. The method of claim 21, further comprising:

evaluating said missing piece of knowledge by at least one of;

identifying a most common response received from said external sources and selecting said most common response;

selecting responses from first outside sources, said first outside sources having a previously established level of accuracy in providing previous responses to previous inquiries;

when said QA system comprises a total amount of evidence, said total amount of evidence having been classified by said QA system as one of good evidence and bad evidence, said good evidence enabling said QA system to provide a correct answer to said question, and said bad evidence being irrelevant to said question, selecting responses from second outside sources, said second outside sources having a previously established level of accuracy in classifying said total amount of evidence as good evidence or bad evidence that matches classification of said total amount of evidence by said QA system as good evidence and bad evidence; and

selecting responses from third outside sources, said third outside sources having demonstrated a previously established level of accuracy in providing responses that match responses from a known source.

23. The method of claim 21, said QA system further comprising a plurality of passages, said method further comprising comparing said question to said plurality of passages to provide said plurality of candidate answers to said question.

24. The method of claim 21, said method further comprising:

said QA system extracting evidence related to said question from said corpus of data; and

said QA system classifying said total amount of evidence as one of good evidence and marginal evidence,

said good evidence having an evidence score above a previously established evidence threshold value, and enabling said QA system to provide a candidate answer to said question with a confidence score above a previously established confidence threshold value, and

said marginal evidence having an evidence score below said previously established evidence threshold value, enabling said QA system to provide a candidate answer to said question with a confidence score below said previously established confidence threshold value, and requiring said QA system to obtain additional information to improve said confidence score.