Utilizing failures in question and answer system responses to enhance the accuracy of question and answer systems

• US 9,053,425 B2
• Filed: 08/01/2012
• Issued: 06/09/2015
• Est. Priority Date: 07/15/2011
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Barborak, Michael A., Chu-Carroll, Jennifer, Ferrucci, David A., Murdock, James W. IV, Zadrozny, Wlodek W.
• Primary Examiner(s): Chang, Li-Wu
• Assistant Examiner(s): Chubb, Mikayla
• Application Number:
  US13/563,838
• Publication Number:
  US 20130017524A1
• Time in Patent Office: 1,042 Days
• US Class Current: 1/1
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 5/041 Abduction
• CPC Class Codes: G09B 7/04 characterised by modifying the teaching programme in response to a wrong answer, e.g. repeating the question, supplying a further explanation

Abstract

A computerized device for enhancing the accuracy of a question-answer system. The computerized device comprises a question-answer system comprising software for performing a plurality of question answering processes. A receiver receives a question into the question-answer system. A processor that generates a plurality of candidate answers to the question is connected to the question-answer system. The processor determines a confidence score for each of the plurality of candidate answers. The processor evaluates sources of evidence used to generate the plurality of candidate answers. The processor identifies missing information from a corpus of data. The missing information comprises any information that improves a confidence score for a candidate answer. The processor generates at least one follow-on inquiry based on the missing information. A network interface outputs the at least one follow-on inquiry to external sources separate from the question-answer system.

Claims

1. A computerized device for enhancing the accuracy of a question-answer system, comprising:

a question-answer system comprising software for performing a plurality of question answering processes;

a receiver receiving a question into said question-answer system;

a processor connected to said question-answer system, said processor;

generating a plurality of candidate answers to said question,

determining a confidence score for each of said plurality of candidate answers,

evaluating sources of evidence used to generate said plurality of candidate answers,

identifying missing information from a corpus of data, said missing information comprising any information that improves a confidence score for a candidate answer of said plurality of candidate answers, said identifying said missing information comprising;

evaluating a piece of evidence and producing an evidence score according to a scoring process that reflects a degree to which said piece of evidence supports or refutes said candidate answer,

classifying said piece of evidence as marginal evidence if said score is below a threshold value, and

evaluating said marginal evidence to identify said missing information, and

generating at least one follow-on inquiry based on said missing information; and

a network interface outputting said at least one follow-on inquiry to external sources separate from said question-answer system to obtain responses to said at least one follow-on inquiry and receiving from said external sources at least one response to said at least one follow-on inquiry,

said processor inputting said at least one response into said corpus of data.

2. The computerized device of claim 1, said processor developing additional logical rules and additional evidence for said question-answer system based on said at least one response to said at least one follow-on inquiry.

3. The computerized device of claim 1, in response to said processor inputting said at least one response into said question-answer system, said processor re-evaluating said confidence score for said plurality of candidate answers.

4. The computerized device of claim 1, further comprising:

a parser, said parser;

parsing said question into a first collection of elements; and

parsing said piece of evidence into a second collection of elements; and

said processor further analyzing said first collection of elements for said question and said piece of evidence in order to determine a relationship between a first element of said first collection of elements for said question and a second element of said second collection of elements for said piece of evidence; and

locating a missing relationship between said first element and said second element, said missing relationship comprising said missing information.

5. The computerized device of claim 1, said processor further:

generating a confidence score for said candidate answer;

evaluating whether said confidence score is below a confidence threshold value; and

if said confidence score is below said confidence threshold value, triggering said step of identifying missing information.

6. The computerized device of claim 1, said processor validating that said response is accurate prior to adding said response to said corpus of data, said validating comprising one of:

identifying a most common response received from external sources and selecting said most common response;

selecting responses from first outside sources, said first outside sources having a previously established level of accuracy in providing previous responses to previous inquiries;

when said question-answer system comprises a total amount of evidence, said total amount of evidence having been classified by said question-answer system as one of good evidence and bad evidence, said good evidence enabling said question-answer system to provide a correct answer to said question, and said bad evidence being irrelevant to said question, selecting responses from second outside sources, said second outside sources having a previously established level of accuracy in classifying said total amount of evidence as good evidence or bad evidence that matches classification of said total amount of evidence by said question-answer system as good evidence and bad evidence; and

selecting answers from third outside sources, said third outside sources having demonstrated a previously established level of accuracy in providing responses that match responses from a known source.

7. A computer system for enhancing the accuracy of a question answering system, comprising:

an automated question answering (QA) system comprising;

a corpus of data,

a processor, said processor having software for performing a plurality of question answering processes;

a receiver; and

a network interface,

said receiver receiving a question into said automated QA system,

said processor generating a plurality of candidate answers to said question,

said processor evaluating sources of evidence used to generate said plurality of candidate answers to identify marginal evidence, said marginal evidence contributing only partially to a candidate answer,

said processor determining a confidence score for each of said plurality of candidate answers,

said processor identifying missing information not provided by said marginal evidence that would enable said automated QA system to further develop said confidence score,

said processor automatically generating at least one follow-on inquiry based on said missing information,

said network interface outputting said at least one follow-on inquiry to external sources separate from said automated QA system and receiving from said external sources at least one response to said at least one follow-on inquiry,

said processor inputting at least one response into said corpus of data, and

said processor automatically developing additional logical rules and additional evidence for said automated QA system based on said at least one response to said at least one follow-on inquiry.

8. The computer system of claim 7, in response to said processor inputting said at least one response into said corpus of data, said processor re-evaluating said confidence score for said plurality of candidate answers.

9. The computer system of claim 7, said processor further evaluating said at least one response prior to inputting said at least one response into said corpus of data by at least one of:

identifying a most common response received from said external sources and selecting said most common response;

selecting responses from first outside sources, said first outside sources having a previously established level of accuracy in providing previous responses to previous inquiries;

when said QA system comprises a total amount of evidence, said total amount of evidence having been classified by said QA system as one of good evidence and bad evidence, said good evidence enabling said QA system to provide a correct answer to said question, and said bad evidence being irrelevant to said question, selecting responses from second outside sources, said second outside sources having a previously established level of accuracy in classifying said total amount of evidence as good evidence or bad evidence that matches classification of said total amount of evidence by said QA system as good evidence and bad evidence; and

selecting responses from third outside sources, said third outside sources having demonstrated a previously established level of accuracy in providing responses that match responses from a known source.

10. The computer system of claim 7, said processor identifying said missing information further comprising:

evaluating a piece of evidence to produce an evidence score according to a scoring process that reflects a degree to which said piece of evidence supports or refutes said candidate answer;

classifying said piece of evidence as marginal evidence if said score is below a threshold value; and

evaluating said marginal evidence to identify said missing information.

11. The computer system of claim 10, further comprising:

a parser, said parser;

parsing said question into a first collection of elements; and

parsing said piece of evidence into a second collection of elements; and

said processor further analyzing said first collection of elements for said question and said piece of evidence in order to determine a relationship between a first element of said first collection of elements for said question and a second element of said second collection of elements for said piece of evidence; and

locating a missing relationship between said first element and said second element, said missing relationship comprising said missing information.

12. The computer system of claim 7, said processor further:

generating a confidence score for said candidate answer;

evaluating whether said confidence score is below a confidence threshold value; and

if said confidence score is below said confidence threshold value, triggering said step of identifying missing information.

13. A question answering (QA) system comprising:

a processor;

an evidence analysis module, said evidence analysis module being connected to said processor;

a first interface connected to said processor;

a second interface connected to said processor; and

a corpus of data connected to said evidence analysis module comprising a total amount of evidence, said evidence analysis module classifying said total amount of evidence as good evidence or marginal evidence,

said first interface receiving a first question to be answered by said QA system,

said processor creating a collection of candidate answers to said first question from said corpus of data, each said candidate answer having supporting evidence and a confidence score generated by said processor,

said good evidence having an evidence score above a previously established evidence threshold value, and enabling said QA system to provide a candidate answer to said first question with a confidence score above a previously established confidence threshold value, and

said marginal evidence having an evidence score below said previously established evidence threshold value, enabling said QA system to provide a candidate answer to said first question with a confidence score below said previously established confidence threshold value, and requiring said QA system to obtain additional information to improve said confidence score for said candidate answer to said first question,

said evidence analysis module producing a second question based on said supporting evidence,

said processor presenting said second question through said second interface to one or more external sources separate from said QA system to obtain responses to said second question,

said processor receiving at least one response or knowledge item from said one or more external sources through said second interface,

said processor inputting said at least one response or knowledge item into said corpus of data, and

said processor automatically developing additional logical rules and additional evidence for said QA system based on said at least one response or knowledge item.

14. The question answering system of claim 13, said corpus of data further comprising a plurality of passages, said processor comparing said first question to said plurality of passages to provide said collection of candidate answers.

15. The question answering system of claim 13, said processor further evaluating said at least one response by at least one of:

identifying a most common response received from said external sources and selecting said most common response;

selecting responses from first outside sources, said first outside sources having a previously established level of accuracy in providing previous responses to previous inquiries;

when said QA system comprises a total amount of evidence, said total amount of evidence having been classified by said QA system as one of good evidence and bad evidence, said good evidence enabling said QA system to provide a correct answer to said question, and said bad evidence being irrelevant to said question, selecting responses from second outside sources, said second outside sources having a previously established level of accuracy in classifying said total amount of evidence as good evidence or bad evidence that matches classification of said total amount of evidence by said QA system as good evidence and bad evidence; and

selecting answers from third outside sources, said third outside sources having demonstrated a previously established level of accuracy in providing responses that match responses from a known source.

16. The question answering system of claim 13, in response to inputting said at least one response or knowledge item into said corpus of data, said processor re-evaluating said confidence score for said collection of candidate answers.

17. A non-transitory computer readable storage medium readable by a computerized device, said computerized device comprising a question-answer system comprising question answering processes, said non-transitory computer readable storage medium storing instructions executable by said computerized device to perform a method comprising:

identifying missing information from a corpus of data, said missing information comprising any information that improves a confidence for a candidate answer to a question, said identifying said missing information comprising;

evaluating a piece of evidence and producing an evidence score according to a scoring process that reflects a degree to which said piece of evidence supports or refutes said candidate answer,

classifying said piece of evidence as marginal evidence if said score is below a threshold value, and

evaluating said marginal evidence to identify said missing information;

generating a follow-on inquiry, said follow-on inquiry prompting for said missing information to be provided;

outputting said follow-on inquiry to an external source;

receiving a response to said follow-on inquiry from said external source; and

adding said response to said corpus of data.

18. The non-transitory computer readable storage medium of claim 17, said identifying missing information further comprising:

evaluating a piece of evidence relevant to said candidate answer to identify said missing information, said missing information comprising a data, a fact, a syntactical relationship, a grammatical relationship, a logical rule, or a taxonomy rule.

19. The non-transitory computer readable storage medium of claim 18, said evaluating a piece of evidence further comprising:

parsing said question into a first collection of elements;

parsing said piece of evidence into a second collection of elements;

analyzing said first collection of elements for said question and said piece of evidence in order to determine a relationship between an element of said first collection of elements for said question and an element of said second collection of element for said piece of evidence, said relationship between elements comprising a lexical relationship, a grammatical relationship, or a semantic relationship; and

locating a missing relationship between said elements, said missing relationship comprising said missing information.

20. The non-transitory computer readable storage medium of claim 17, said method further comprising:

receiving said question into said question-answer system; and

generating said candidate answer to said question using said question-answer system and said corpus of data.

21. The non-transitory computer readable storage medium of claim 17, said method further comprising:

generating a confidence score for said candidate answer;

evaluating whether said confidence score is below a confidence threshold value; and

if said confidence score is below said confidence threshold value, triggering said step of identifying missing information.

22. The non-transitory computer readable storage medium of claim 17, said method further comprising validating said response is accurate prior to adding said response to said corpus of data, said validating comprising one of:

identifying a most common response received from external sources and selecting said most common response;

selecting responses from first outside sources, said first outside sources having a previously established level of accuracy in providing previous responses to previous inquiries;

when said question-answer system comprises a total amount of evidence, said total amount of evidence having been classified by said question-answer system as one of good evidence and bad evidence, said good evidence enabling said question-answer system to provide a correct answer to said question, and said bad evidence being irrelevant to said question, selecting responses from second outside sources, said second outside sources having a previously established level of accuracy in classifying said total amount of evidence as good evidence or bad evidence that matches classification of said total amount of evidence by said question-answer system as good evidence and bad evidence; and

selecting answers from third outside sources, said third outside sources having demonstrated a previously established level of accuracy in providing responses that match responses from a known source.