Generating secondary questions in an introspective question answering system

• US 10,614,725 B2
• Filed: 09/11/2012
• Issued: 04/07/2020
• Est. Priority Date: 09/11/2012
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Boguraev, Branimir K., Buchanan, David W., Chu-Carroll, Jennifer, Ferrucci, David A., Kalyanpur, Aditya A., Murdock, IV, James W., Patwardhan, Siddharth A.
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Seck, Ababacar
• Application Number:
  US13/610,355
• Publication Number:
  US 20140072948A1
• Time in Patent Office: 2,765 Days
• Field of Search: 706 45
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F

Abstract

A method of generating secondary questions in a question-answer system. Missing information is identified from a corpus of data using a computerized device. The missing information comprises any information that improves confidence scores for candidate answers to a question. The computerized device automatically generates a plurality of hypotheses concerning the missing information. The computerized device automatically generates at least one secondary question based on each of the plurality of hypotheses. The hypotheses are ranked based on relative utility to determine an order in which the computerized device outputs the at least one secondary question to external sources to obtain responses.

Claims

1. A computerized device, comprising:

a question-answer system comprising a processor running software for performing a plurality of question answering processes and a corpus of data;

a receiver receiving a first question into the question-answer system; and

a network interface connected to external sources comprising a community of human respondents;

said processor;

comparing the first question to the corpus of data;

generating candidate answers for the first question posed to the question-answer system, the candidate answers for the first question being generated from the corpus of data;

determining a confidence score for each of the candidate answers based on evidence from the corpus of data used to generate the candidate answers, the determining a confidence score further comprises assigning an evidence score to the evidence based on how well the evidence matches the first question, wherein the evidence comprises good evidence, marginal evidence, and bad evidence;

identifying information to supplement the marginal evidence, the information improves the confidence scores for the candidate answers to the first question;

generating a plurality of hypotheses concerning the information that supplements the marginal evidence and improves the confidence scores for the candidate answers to the first question;

generating the at least one secondary question based on each hypothesis of the plurality of hypotheses concerning the information that supplements the marginal evidence and improves the confidence scores for the candidate answers to the first question, an answer to the at least one secondary question improving the ability of the question-answer system to understand and evaluate evidence associated with the candidate answers to the first question;

ranking the hypotheses based on relative utility to determine an order in which to output the at least one secondary question to the external sources;

outputting the at least one secondary question to the external sources using the network interface;

receiving responses to the at least one secondary question from the external sources using the network interface;

validating the responses to the at least one secondary question to extract a piece of data, fact, syntactical relationship, grammatical relationship, logical rule, or taxonomy rule that improves the confidence scores for the candidate answers to the first question, the validating comprises validating that the responses are supported by a threshold number of external sources; and

adding the piece of data, fact, syntactical relationship, grammatical relationship, logical rule, or taxonomy rule extracted from responses to the at least one secondary question to the corpus of data.

2. The computerized device of claim 1, the processor generating a plurality of hypotheses further comprising:

analyzing elements of the first question;

for each of the candidate answers, forming a hypothesis based on considering each candidate answer in context of the first question;

spawning an independent thread for each hypothesis that attempts to prove the candidate answer;

extracting evidence related to each hypothesis from the corpus of data; and

for each evidence-hypothesis pair, analyzing elements of the first question and the evidence.

3. The computerized device of claim 2, the analyzing elements of the first question and the evidence being along dimensions selected from the group consisting of:

type classification,

time,

geography,

popularity,

passage support,

source reliability, and

semantic relatedness.

4. The computerized device of claim 1, wherein factors affecting the ranking the hypotheses further comprise at least one of:

cumulative impact on all candidate answers based on the responses to one or more secondary questions,

estimated impact on future questions in the corpus of data based on determining a likelihood that a hypothesis will appear in relevant content for future questions,

frequency of observation of terms in the question in the corpus of data, and

likelihood that an intuitive and answerable secondary question may be formulated for a human expert.

5. The computerized device of claim 1, processor generating at least one secondary question comprising forming a natural-language inquiry;

and

the processor generating at least one secondary question comprising at least one of;

using the question and the evidence to formulate the at least one secondary question,

using semantic concepts to formulate the at least one secondary question,

using relations and data to formulate the at least one secondary question, and

using background knowledge to formulate the at least one secondary question.

6. The computerized device of claim 1, the processor generating at least one secondary question selected from the group consisting of:

simple Yes/No questions;

questions requiring responses on a qualitative scale; and

questions requiring a quantitative response.

7. A computer system, comprising:

an automated question answering (QA) system comprising;

a corpus of data;

a processor operatively connected to the corpus of data, the processor having software for performing a plurality of question answering processes;

a receiver operatively connected to the processor; and

a network interface operatively connected to the processor and to external expert community sources;

the receiver receiving a question into the automated QA system,

the processor comparing the question to the corpus of data and generating a plurality of candidate answers to the question from the corpus of data,

the processor determining a confidence score for each candidate answer of the plurality of candidate answers based on evidence used to generate the each candidate answer of the plurality of candidate answers, wherein the evidence comprises good evidence, marginal evidence, and bad evidence,

the processor identifying information to supplement the marginal evidence, the information improves the confidence scores for at least one candidate answer in the plurality of candidate answers,

the processor generating a plurality of hypotheses concerning the information that supplements the marginal evidence and improves the confidence scores for the at least one candidate answer,

the processor generating the at least one follow-on inquiry based on each hypothesis of the plurality of hypotheses,

the processor ranking the hypotheses based on relative utility, the ranking determining an order for the automated QA system to output the at least one follow-on inquiry to the external expert community sources,

the processor outputting the at least one follow-on inquiry to the external expert community sources using the network interface,

the processor receiving responses to the at least one follow-on inquiry from the external expert community sources using the network interface,

the processor validating the responses to the at least one follow-on inquiry and extracting a piece of data, fact, syntactical relationship, grammatical relationship, logical rule, or taxonomy rule that improves the confidence scores for the at least one candidate answer, the validating comprises validating that the responses are supported by a threshold number of external expert community sources, and

the processor adding the piece of data, fact, syntactical relationship, grammatical relationship, logical rule, or taxonomy rule extracted from the responses to the at least one follow-on inquiry to the corpus of data.

8. The computer system of claim 7, wherein factors affecting the ranking the hypotheses further comprise at least one of:

cumulative impact on all candidate answers based on responses to one or more follow-on inquiries,

estimated impact on future questions in the corpus of data based on determining a likelihood that a hypothesis will appear in relevant content for future questions,

frequency of observation of terms in the question in the corpus of data, and

likelihood that an intuitive and answerable follow-on inquiry may be formulated for a human expert.

9. The computer system of claim 7, the processor generating a plurality of hypotheses further comprising:

analyzing the question;

for each candidate answer of the plurality of candidate answers, forming a hypothesis based on considering each the candidate answer in context of the question;

spawning an independent thread for each hypothesis that attempts to prove the candidate answer;

extracting evidence related to each hypothesis from the corpus of data; and

for each evidence-hypothesis pair, analyzing elements of the question and the evidence.

10. The computer system of claim 9, the analyzing the elements of the question and the extracting the evidence being along dimensions selected from the group consisting of:

type classification,

time,

geography,

popularity,

passage support,

source reliability, and semantic relatedness.

11. The computer system of claim 7, the processor generating at least one follow-on inquiry comprising forming a natural language inquiry, and

the processor generating at least one follow-on inquiry comprising at least one of:

using the question and evidence to generate the at least one follow-on inquiry,

using semantic concepts to generate the at least one follow-on inquiry,

using relations and data to generate the at least one follow-on inquiry, and

using background knowledge to generate the at least one follow-on inquiry.

12. The computer system of claim 7, the at least one follow-on inquiry being selected from the group consisting of:

simple Yes/No questions;

questions requiring responses on a qualitative scale; and

questions requiring a quantitative response.

13. A question answering (QA) system comprising:

a processor;

an evidence analysis module, the evidence analysis module being operatively connected to the processor;

a first interface operatively connected to the processor;

a second interface operatively connected to the processor and to one or more external sources separate from the QA system, the one or more external sources comprising a community of human respondents; and

a corpus of data operatively connected to the evidence analysis module,

the first interface receiving a first question to be answered by the QA system,

the processor comparing the first question to the corpus of data and creating a collection of candidate answers to the first question from the corpus of data, each candidate answer in the collection of candidate answers to the first question having supporting evidence and a confidence score generated by the processor based on the evidence from corpus of data used to generate the candidate answer, wherein the evidence comprises good evidence, marginal evidence, and bad evidence;

the evidence analysis module identifying information to supplement the marginal evidence, the information that improves the confidence scores for the candidate answers to the first question,

the processor generating a plurality of hypotheses concerning the information to supplement the marginal evidence, the information that improves the confidence scores for the candidate answers to the first question,

the evidence analysis module producing the secondary question based on each hypothesis of the plurality of hypotheses, an answer to the secondary question improving the ability of the QA system to understand and evaluate evidence associated with candidate answers to the first question,

the processor ranking the hypotheses based on relative utility, the ranking determining an order in which the QA system outputs a secondary question to the one or more external sources,

the processor presenting the secondary question through the second interface to the one or more external sources separate from the QA system to obtain responses to the secondary question,

the processor receiving at least one response to the secondary question from the one or more external sources through the second interface,

the evidence analysis module validating the at least one response to the secondary question and extracting a piece of data, fact, syntactical relationship, grammatical relationship, logical rule, or taxonomy rule that improves the confidence scores for the candidate answer to the first question, the validating comprises validating that the responses are supported by a threshold number of external sources,

the processor adding the piece of data, fact, syntactical relationship, grammatical relationship, logical rule, or taxonomy rule extracted from the at least one response to the corpus of data.

14. The question answering system of claim 13, the processor generating a plurality of hypotheses concerning the information that improves the confidence scores for the candidate answers further comprising:

analyzing the first question;

for each candidate answer or the collection of candidate answers, forming a hypothesis based on considering the each candidate answer in context of the first question;

spawning an independent thread for each hypothesis that attempts to prove the candidate answer;

extracting evidence related to each hypothesis from the corpus of data; and

for each evidence-hypothesis pair, analyzing elements of the first question and the evidence along dimensions selected from the group consisting of;

type classification,

time,

geography,

popularity,

passage support,

source reliability, and semantic relatedness.

15. The question answering system of claim 13, factors affecting the ranking the ranking of the hypotheses further comprising at least one of:

cumulative impact on all candidate answers based on responses to one or more secondary questions,

estimated impact on future questions in the corpus of data based on determining a likelihood that a hypothesis will appear in relevant content for further questions,

frequency of observations of terms in the first question in background knowledge base, and

likelihood that an intuitive and answerable question may be formulated for a human expert.

16. The question answering system of claim 13, the evidence analysis module producing a secondary question based on the hypotheses comprising forming a natural-language inquiry;

and

the evidence analysis module producing a secondary question comprising at least one of;

using question and evidence to generate the secondary question,

using semantic concepts to generate the secondary question,

using relations and data to generate the secondary question, and

using background knowledge to generate the secondary question.

17. The question answering system of claim 13, the secondary question being selected from the group consisting of:

simple Yes/No questions;

questions requiring responses on a qualitative scale; and

questions requiring a quantitative response.

18. A non-transitory computer readable storage medium readable by a computerized device,

the computerized device comprising a question-answer system, the non-transitory computer readable storage medium storing instructions executable by the computerized device to perform a method comprising:

receiving a first question into the question-answer system;

comparing the question to the corpus of data;

generating candidate answers for the first question from the corpus of data;

determining a confidence score for each of the candidate answers based on evidence from the corpus of data used to generate the candidate answers, wherein the evidence comprises good evidence, marginal evidence, and bad evidence;

identifying information from the corpus of data to supplement the marginal evidence, the information that improves confidence scores for candidate answers to a first question posed to the question-answer system;

automatically generating a plurality of hypotheses concerning the information that supplements the marginal evidence and improves the confidence scores for the candidate answers to the first question posed to the question-answer system, the automatically generating the plurality of hypotheses further comprising;

analyzing the first question,

for each candidate answer of the candidate answers, forming a hypothesis based on considering each the candidate answer in context of the first question,

spawning an independent thread for each hypothesis that attempts to prove the candidate answer,

extracting evidence related to each hypothesis from the corpus of data, and

for each evidence-hypothesis pair, analyzing elements of the first question and the evidence along dimensions selected form the group consisting of;

type classification,

time,

geography,

popularity,

passage support,

source reliability, and semantic relatedness;

automatically generating at least one secondary question based on each of the plurality of hypotheses, an answer to the at least one secondary question improving the ability of the question-answer system to understand and evaluate evidence associated with the candidate answers to the first question;

ranking the hypotheses based on relative utility, the ranking determining an order in which to output the at least one secondary question to external sources comprising a community of human respondents, wherein the community of human respondents are capable of answering the at least one secondary question;

outputting the at least one secondary question to the external sources using a network interface;

receiving responses to the at least one secondary question from the external sources using the network interface;

validating the responses to the at least one secondary question to extract a piece of data, fact, syntactical relationship, grammatical relationship, logical rule, or taxonomy rule that improves confidence scores for the candidate answers to the first question, the validating comprises validating that the responses are supported by a threshold number of external sources; and

adding the piece of data, fact, syntactical relationship, grammatical relationship, logical rule, or taxonomy rule extracted from responses to the at least one secondary question to the corpus of data.

19. The non-transitory computer readable storage medium of claim 18, factors affecting the ranking the hypotheses further comprising at least one of:

cumulative impact on all candidate answers based on responses to one or more follow-on inquiries,

estimated impact on future questions in the corpus of data based on determining a likelihood that a hypothesis will appear in relevant content for future questions,

frequency of observation of terms in the question in the corpus of data, and

likelihood that an intuitive and answerable secondary question may be formulated for a human expert.

20. The non-transitory computer readable storage medium of claim 18, the generating at least one secondary question comprising formulating a natural language inquiry, and

the generating at least one secondary question comprising at least one of:

using the first question and evidence related to at least one hypothesis to formulate the at least one secondary question,

using semantic concepts to generate the at least one secondary question,

using relations and data to generate the at least one secondary question, and

using background knowledge to generate the at least one secondary question.

21. The non-transitory computer readable storage medium of claim 18, the at least one secondary question selected from the group consisting of:

simple Yes/No questions;

questions requiring responses on a qualitative scale; and

questions requiring a quantitative response.