GENERATING SECONDARY QUESTIONS IN AN INTROSPECTIVE QUESTION ANSWERING SYSTEM

• US 20140072947A1
• Filed: 09/11/2012
• Published: 03/13/2014
• Est. Priority Date: 09/11/2012
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Boguraev, Branimir K., Buchanan, David W., Chu-Carroll, Jennifer, Ferrucci, David A., Kalyanpur, Aditya A., Murdock, James W. IV, Patwardhan, Siddharth A.
• Granted Patent: US 10,621,880 B2
• Time in Patent Office: Days
• US Class Current: 434/362
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F

Abstract

A method of generating secondary questions in a question-answer system. Missing information is identified from a corpus of data using a computerized device. The missing information comprises any information that improves confidence scores for candidate answers to a question. The computerized device automatically generates a plurality of hypotheses concerning the missing information. The computerized device automatically generates at least one secondary question based on each of the plurality of hypotheses. The hypotheses are ranked based on relative utility to determine an order in which the computerized device outputs the at least one secondary question to external sources to obtain responses.

Claims

1. A method of generating secondary questions in a question-answer system, said method comprising:

identifying missing information from a corpus of data, using a computerized device, said missing information comprising any information that improves confidence scores for candidate answers to a question;

automatically generating, by said computerized device, a plurality of hypotheses concerning said missing information;

automatically generating, by said computerized device, at least one secondary question based on each of said plurality of hypotheses; and

ranking said hypotheses, using said computerized device, based on relative utility to determine an order in which said computerized device outputs said at least one secondary question to external sources to obtain responses.

2. The method of claim 1, said automatically generating a plurality of hypotheses further comprising:

analyzing said question;

for each of said candidate answers, forming a hypothesis based on considering each candidate answer in context of said question;

spawning an independent thread that attempts to prove said candidate answer;

extracting evidence related to each hypothesis from said corpus of data; and

for each evidence-hypothesis pair, analyzing elements of said question and said evidence.

3. The method of claim 2, said analyzing elements of said question and said evidence being along dimensions selected from the group consisting of:

type classification,

time,

geography,

popularity,

passage support,

source reliability, and

semantic relatedness.

4. The method of claim 1, said missing information comprising a data, a fact, a syntactical relationship, a grammatical relationship, a logical rule, or a taxonomy rule.

5. The method of claim 4, factors affecting said ranking said hypotheses further comprising at least one of:

cumulative impact on all candidate answers based on said responses to one or more secondary questions,

estimated impact on future questions in said corpus of data based on determining a likelihood that a hypothesis will appear in relevant content for future questions,

frequency of observation of terms in said question in said corpus of data, and

likelihood that an intuitive and answerable secondary question may be formulated for a human expert.

6. The method of claim 1, said generating at least one secondary question comprising formulating a natural language inquiry, and

said generating at least one secondary question comprising at least one of:

using said question and said evidence to formulate said at least one secondary question,

using semantic concepts to formulate said at least one secondary question,

using relations and data to formulate said at least one secondary question, and

using background knowledge to formulate said at least one secondary question.

7. The method of claim 1, said at least one secondary question being selected from the group consisting of:

simple Yes/No questions;

questions requiring responses on a qualitative scale; and

questions requiring a quantitative response.

8. A computer implemented method of generating follow-on inquiries in a question answering system, said method comprising the steps of:

attempting to answer a question by an automated question answering system operating on a computerized device;

generating a plurality of candidate answers to said question, using said computerized device;

determining a confidence score for each of said plurality of candidate answers based on evidence used to generate said candidate answers, using said computerized device;

identifying missing information from a corpus of data, using said computerized device, said missing information comprising any information that improves said confidence score for said candidate answers;

automatically generating, by said computerized device, a plurality of hypotheses concerning said missing information;

automatically generating, by said computerized device, at least one follow-on inquiry based on each of said plurality of hypotheses; and

ranking said hypotheses, using said computerized device, based on relative utility to determine an order in which said computerized device outputs said at least one follow-on inquiry to external sources to obtain responses.

9. The computer implemented method of claim 8, said automatically generating a plurality of hypotheses further comprising:

analyzing said question;

for each of said candidate answers, forming a hypothesis based on considering each candidate answer in context of said question;

spawning an independent thread that attempts to prove said candidate answer;

extracting evidence related to each hypothesis from said corpus of data; and

for each evidence-hypothesis pair, analyzing elements of said question and said evidence.

10. The computer implemented method of claim 9, said analyzing elements of said question and said evidence being along dimensions selected from the group consisting of:

type classification,

time,

geography,

popularity,

passage support,

source reliability, and

semantic relatedness.

11. The computer implemented method of claim 8, said missing information comprising a data, a fact, a syntactical relationship, a grammatical relationship, a logical rule, or a taxonomy rule.

12. The computer implemented method of claim 8, factors affecting said ranking said hypotheses further comprising at least one of:

cumulative impact on all candidate answers based on said responses to one or more follow-on inquiries,

estimated impact on future questions in said corpus of data based on determining a likelihood that a hypothesis will appear in relevant content for future questions,

frequency of observation of terms in said question in said corpus of data, and

likelihood that an intuitive and answerable follow-on inquiry may be formulated for a human expert.

13. The computer implemented method of claim 8, said generating at least one follow-on inquiry comprising forming a natural language inquiry, and

said generating at least one follow-on inquiry comprising at least one of:

using said question and evidence to generate said at least one follow-on inquiry,

using semantic concepts to generate said at least one follow-on inquiry,

using relations and data to generate said at least one follow-on inquiry, and

using background knowledge to generate said at least one follow-on inquiry.

14. The computer implemented method of claim 8, said at least one follow-on inquiry being selected from the group consisting of:

simple Yes/No questions;

questions requiring responses on a qualitative scale; and

questions requiring a quantitative response.

15. A method comprising:

providing a first question to be answered by a Question Answering (QA) system to a processor;

creating, by said processor, a collection of candidate answers to said first question, said collection of candidate answers being created from a corpus of data;

generating, by said processor, supporting evidence for each said candidate answer;

identifying, by an evidence analysis module, missing information from said corpus of data, said missing information comprising any information that improves confidence scores for said candidate answers;

generating, by said processor, a plurality of hypotheses concerning said missing information;

producing, by said evidence analysis module, a secondary question based on each hypothesis; and

ranking, by said processor, said hypotheses based on relative utility to determine an order in which said QA system outputs said secondary question to external sources.

16. The method of claim 15, said generating a plurality of hypotheses concerning said missing information further comprising:

analyzing said first question;

for each of said candidate answers, forming a hypothesis based on considering each candidate answer in context of said first question;

spawning an independent thread that attempts to prove said candidate answer;

extracting evidence related to each hypothesis from said corpus of data; and

for each evidence-hypothesis pair, analyzing elements of said first question and said evidence along dimensions selected from the group consisting of;

type classification,

time,

geography,

popularity,

passage support,

source reliability, and

semantic relatedness.

17. The method of claim 15, factors affecting said processor ranking said hypotheses further comprising at least one of:

cumulative impact on all candidate answers based on responses to one or more secondary questions,

estimated impact on future questions in said corpus of data based on determining a likelihood that a hypothesis will appear in relevant content for future questions,

frequency of observation of terms in said first question in said corpus of data, and

likelihood that an intuitive and answerable secondary question may be formulated for a human expert.

18. The method of claim 15, said producing a secondary question comprising formulating a natural language inquiry, and

said producing a secondary question comprising at least one of:

using said question and evidence to generate said secondary question,

using semantic concepts to generate said secondary question,

using relations and data to generate said secondary question, and

using background knowledge to generate said secondary question.

19. The method of claim 15, said secondary question being selected from the group consisting of:

simple Yes/No questions;

questions requiring responses on a qualitative scale; and

questions requiring a quantitative response.

20. A method comprising:

receiving a question into a Question Answering (QA) system, said QA system comprising a plurality of question answering processes;

said QA system comparing said question to a corpus of data;

said QA system creating a collection of candidate answers to said question from said corpus of data;

said QA system identifying missing information from said corpus of data, said missing information comprising any information that improves a confidence score for at least one candidate answer in said collection of candidate answers;

said QA system generating a plurality of hypotheses concerning said question and said collection of candidate answers;

said QA system generating at least one follow-on inquiry based on each of said plurality of hypotheses; and

said QA system ranking said hypotheses based on relative utility to determine an order in which to output said at least one follow-on inquiry to external sources.

21. The method of claim 20, said generating a plurality of hypotheses further comprising:

analyzing said question;

for each said candidate answer in said collection of candidate answers, forming a hypothesis based on considering each candidate answer in context of said question;

spawning an independent thread that attempts to prove said candidate answer;

extracting evidence related to each hypothesis from said corpus of data; and

for each evidence-hypothesis pair, analyzing elements of said question and said evidence along dimensions selected from the group consisting of;

type classification,

time,

geography,

popularity,

passage support,

source reliability, and

semantic relatedness.

22. The method of claim 20, said missing information comprising a data, a fact, a syntactical relationship, a grammatical relationship, a logical rule, or a taxonomy rule.

23. The method of claim 20, factors affecting said ranking said hypotheses further comprising at least one of:

cumulative impact on all candidate answers based on responses to one or more follow-on inquiries,

estimated impact on future questions in said corpus of data based on determining a likelihood that a hypothesis will appear in relevant content for future questions,

frequency of observation of terms in said question in said corpus of data, and

likelihood that an intuitive and answerable follow-on inquiry may be formulated for a human expert.

24. The method of claim 20, said generating at least one follow-on inquiry comprising formulating a natural language inquiry, and

said generating at least one follow-on inquiry comprising at least one of:

using said question and evidence to generate said at least one follow-on inquiry,

using semantic concepts to generate said at least one follow-on inquiry,

using relations and data to generate said at least one follow-on inquiry, and

using background knowledge to generate said at least one follow-on inquiry.

25. The method of claim 20, said at least one follow-on inquiry being selected from the group consisting of:

simple Yes/No questions;

questions requiring responses on a qualitative scale; and

questions requiring a quantitative response.