GENERATING SECONDARY QUESTIONS IN AN INTROSPECTIVE QUESTION ANSWERING SYSTEM

• US 20140072948A1
• Filed: 09/11/2012
• Published: 03/13/2014
• Est. Priority Date: 09/11/2012
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Boguraev, Branimir K., Buchanan, David W., Chu-Carroll, Jennifer, Ferrucci, David A., Murdock, James W. IV, Kalyanpur, Aditya A., Patwardhan, Siddharth A.
• Granted Patent: US 10,614,725 B2
• Time in Patent Office: Days
• US Class Current: 434/362
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F

Abstract

A method of generating secondary questions in a question-answer system. Missing information is identified from a corpus of data using a computerized device. The missing information comprises any information that improves confidence scores for candidate answers to a question. The computerized device automatically generates a plurality of hypotheses concerning the missing information. The computerized device automatically generates at least one secondary question based on each of the plurality of hypotheses. The hypotheses are ranked based on relative utility to determine an order in which the computerized device outputs the at least one secondary question to external sources to obtain responses.

Claims

1. A computerized device for generating secondary questions in a question-answer system, comprising:

a question-answer system comprising software for performing a plurality of question answering processes and a corpus of data;

a receiver receiving a question into said question-answer system;

a processor operatively connected to said question-answer system, said processor;

identifying missing information from said corpus of data, said missing information comprising any information that improves confidence scores for candidate answers to said question;

generating a plurality of hypotheses concerning said missing information;

generating at least one secondary question based on each of said plurality of hypotheses; and

ranking said hypotheses based on relative utility to determine an order in which to output said at least one secondary question to external sources.

2. The computerized device of claim 1, said processor generating a plurality of hypotheses further comprising:

analyzing said question;

for each of said candidate answers, forming a hypothesis based on considering each candidate answer in context of said question;

spawning an independent thread that attempts to prove said candidate answer;

extracting evidence related to each hypothesis from a corpus of data; and

for each evidence-hypothesis pair, analyzing elements of said question and said evidence.

3. The computerized device of claim 2, said analyzing elements of said question and said evidence being along dimensions selected from the group consisting of:

type classification,

time,

geography,

popularity,

passage support,

source reliability, and

semantic relatedness.

4. The computerized device of claim 1, said missing information comprising a data, a fact, a syntactical relationship, a grammatical relationship, a logical rule, or a taxonomy rule.

5. The computerized device of claim 4, further comprising a network interface outputting said at least one secondary question to said external sources to obtain responses,

factors affecting said ranking said hypotheses further comprising at least one of:

cumulative impact on all candidate answers based on said responses to one or more secondary questions,

estimated impact on future questions in said corpus of data based on determining a likelihood that a hypothesis will appear in relevant content for future questions,

frequency of observation of terms in said question in said corpus of data, and

likelihood that an intuitive and answerable secondary question may be formulated for a human expert.

6. The computerized device of claim 1, said processor generating at least one secondary question comprising forming a natural-language inquiry;

and

said processor generating at least one secondary question comprising at least one of;

using said question and said evidence to formulate said at least one secondary question,

using semantic concepts to formulate said at least one secondary question,

using relations and data to formulate said at least one secondary question, and

using background knowledge to formulate said at least one secondary question.

7. The computerized device of claim 1, said processor generating at least one secondary question selected from the group consisting of:

simple Yes/No questions;

questions requiring responses on a qualitative scale; and

questions requiring a quantitative response.

8. A computer system for generating follow-on inquiries in a question-answer system, comprising:

an automated question answering (QA) system comprising;

a corpus of data;

a processor operatively connected to said corpus of data, said processor having software for performing a plurality of question answering processes; and

a receiver operatively connected to said processor;

said receiver receiving a question into said automated QA system,

said processor comparing said question to said corpus of data and generating a plurality of candidate answers to said question from said corpus of data,

said processor identifying missing information from said corpus of data, said missing information comprising any information that improves a confidence score for at least one candidate answer in said plurality of candidate answers,

said processor generating a plurality of hypotheses concerning said question and said plurality of candidate answers,

said processor generating at least one follow-on inquiry based on each of said plurality of hypotheses, and said processor ranking said hypotheses based on relative utility to determine an order in which said QA system outputs said at least one follow-on inquiry to external sources.

9. The computer system of claim 8, further comprising:

a network interface outputting said at least one follow-on inquiry to an external expert community source separate from said QA system and receiving from said external expert community source at least one response to said at least one follow-on inquiry,

factors affecting said ranking said hypotheses further comprising at least one of;

cumulative impact on all candidate answers based on responses to one or more follow-on inquiries,

estimated impact on future questions in said corpus of data based on determining a likelihood that a hypothesis will appear in relevant content for future questions,

frequency of observation of terms in said question in said corpus of data, and

likelihood that an intuitive and answerable follow-on inquiry may be formulated for a human expert.

10. The computer system of claim 8, said processor generating a plurality of hypotheses further comprising:

analyzing said question;

for each of said candidate answers, forming a hypothesis based on considering each candidate answer in context of said question;

spawning an independent thread that attempts to prove said candidate answer;

extracting evidence related to each hypothesis from said corpus of data; and

for each evidence-hypothesis pair, analyzing elements of said question and said evidence.

11. The computer system of claim 10, said analyzing elements of said question and said evidence being along dimensions selected from the group consisting of:

type classification,

time,

geography,

popularity,

passage support,

source reliability, and

semantic relatedness.

12. The computer system of claim 8, said processor generating at least one follow-on inquiry comprising forming a natural language inquiry, and

said generating at least one follow-on inquiry comprising at least one of:

using said question and evidence to generate said at least one follow-on inquiry,

using semantic concepts to generate said at least one follow-on inquiry,

using relations and data to generate said at least one follow-on inquiry, and

using background knowledge to generate said at least one follow-on inquiry.

13. The computer system of claim 8, said at least one secondary question being selected from the group consisting of:

simple Yes/No questions;

questions requiring responses on a qualitative scale; and

questions requiring a quantitative response.

14. A question answering (QA) system comprising:

a processor;

an evidence analysis module, said evidence analysis module being operatively connected to said processor;

a first interface operatively connected to said processor; and

a corpus of data operatively connected to said evidence analysis module,

said first interface receiving a first question to be answered by said QA system,

said processor creating a collection of candidate answers to said first question from said corpus of data, each said candidate answer having supporting evidence and a confidence score generated by said processor,

said evidence analysis module identifying missing information from said corpus of data, said missing information comprising any information that improves confidence scores for said candidate answers,

said processor generating a plurality of hypotheses concerning said missing information,

said evidence analysis module producing a secondary question based on each hypothesis, and

said processor ranking said hypotheses based on relative utility to determine an order in which said QA system outputs said secondary question to external sources.

15. The question answering system of claim 14, further comprising:

a second interface connected to said processor, said processor presenting said secondary question through said second interface to one or more external sources separate from said QA system to obtain responses to said secondary question, and

said processor receiving at least one response from said one or more external sources through said second interface.

16. The question answering system of claim 14, said processor generating a plurality of hypotheses concerning said missing information further comprising:

analyzing said first question;

for each candidate answer in said collection of candidate answers, forming a hypothesis based on considering each candidate answer in context of said first question;

spawning an independent thread that attempts to prove said candidate answer;

extracting evidence related to each hypothesis from said corpus of data; and

for each evidence-hypothesis pair, analyzing elements of said first question and said evidence along dimensions selected from the group consisting of;

type classification,

time,

geography,

popularity,

passage support,

source reliability, and

semantic relatedness.

17. The question answering system of claim 14, factors affecting said ranking said hypotheses further comprising at least one of:

cumulative impact based on said responses to one or more secondary questions,

estimated impact on future questions based on how well said hypothesis generalizes,

frequency of observations in background knowledge base, and

likelihood that an intuitive and answerable question may be formulated for a human expert.

18. The question answering system of claim 14, said evidence analysis module producing a secondary question based on said hypotheses comprising forming a natural-language inquiry;

and

said evidence analysis module producing a secondary question comprising at least one of;

using said question and evidence to generate said secondary question,

using semantic concepts to generate said secondary question,

using relations and data to generate said secondary question, and

using background knowledge to generate said secondary question.

19. The question answering system of claim 14, said secondary question being selected from the group consisting of:

simple Yes/No questions;

questions requiring responses on a qualitative scale; and

questions requiring a quantitative response.

20. A non-transitory computer readable storage medium readable by a computerized device, said computerized device comprising a question-answer system, said non-transitory computer readable storage medium storing instructions executable by said computerized device to perform a method comprising:

identifying missing information from a corpus of data, said missing information comprising any information that improves confidence scores for candidate answers to a question;

automatically generating a plurality of hypotheses concerning said question and said candidate answers to said question;

automatically generating at least one secondary question based on each of said plurality of hypotheses; and

ranking said hypotheses based on relative utility to determine an order in which to output said at least one secondary question to external sources to obtain responses.

21. The non-transitory computer readable storage medium of claim 20, said automatically generating a plurality of hypotheses further comprising:

analyzing said question;

for each of said candidate answers, forming a hypothesis based on considering each candidate answer in context of said question;

spawning an independent thread that attempts to prove said candidate answer;

extracting evidence related to each hypothesis from said corpus of data; and

for each evidence-hypothesis pair, analyzing elements of said question and said evidence along dimensions selected from the group consisting of;

type classification,

time,

geography,

popularity,

passage support,

source reliability, and

semantic relatedness.

23. The non-transitory computer readable storage medium of claim 20, factors affecting said ranking said hypotheses further comprising at least one of:

cumulative impact on all candidate answers based on said responses to one or more secondary questions,

estimated impact on future questions in said corpus of data based on determining a likelihood that a hypothesis will appear in relevant content for future questions,

frequency of observation of terms in said question in said corpus of data, and

likelihood that an intuitive and answerable secondary question may be formulated for a human expert.

24. The non-transitory computer readable storage medium of claim 20, said generating at least one secondary question comprising formulating a natural language inquiry, and

said generating at least one secondary question comprising at least one of:

using said question and said evidence to formulate said at least one secondary question,

using semantic concepts to formulate said at least one secondary question,

using relations and data to formulate said at least one secondary question, and

using background knowledge to formulate said at least one secondary question.

25. The non-transitory computer readable storage medium of claim 20, said at least one secondary question selected from the group consisting of:

simple Yes/No questions;

questions requiring responses on a qualitative scale; and

questions requiring a quantitative response.