QUESTIONS AND ANSWERS GENERATION

• US 20110125734A1
• Filed: 03/15/2010
• Published: 05/26/2011
• Est. Priority Date: 11/23/2009
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Ferrucci, David A., Zadrozny, Wlodek W., Gondek, David C., Murdock, James W. IV, Duboue, Pablo A.
• Application Number:
  US12/724,156
• Publication Number:
  US 20110125734A1
• Time in Patent Office: Days
• US Class Current: 707/723
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F

Abstract

A system, method and/or computer program product for automatically generating questions and answers based on any corpus of data. The computer system, given a collection of textual documents, automatically generates collections of questions about the documents together with answers to those questions. In particular, such a process can be applied to the so called ‘open’ domain, where the type of the corpus is not given in advance, and neither is the ontology of the corpus. The system improves the exploring of large bodies of textual information. Applications implementing the system and method include new types of tutoring systems, educational question-answering games, national security and business analysis systems, etc.

Claims

1. A computer-implemented method for generating questions and answers pairs based on any corpus of data, said method comprising:

generating, from a corpus of text data and a set of criteria, one ore more data structures;

generating, based on said set of criteria and one or more data structures, an initial set of questions;

retrieving a set of documents based on said initial set of questions;

generating from said documents, candidate question and answers;

conforming said set of candidate questions and answers to satisfy said set of criteria;

analyzing a quality of answers of said conformed set of questions and answers;

generating further one or more answers based on said analyzing; and

,

outputting, based on said further one or more answers and said criteria, a final list question-answer (QA) pairs, wherein a program using a processor unit executes one or more of said generating, retrieving, generating, conforming, analyzing, generating and outputting.

2. The computer-implemented method of claim 1, wherein said generating candidate question and answers from said documents comprises:

generating, from said set of documents, predicate-argument sets; and

converting said predicate/argument sets of into a set of candidate questions and answers, an answer being one or more arguments of a predicate and the question being a transformation of the predicate and remaining arguments.

3. The computer-implemented method of claim 1, where said criteria includes one of:

a number of questions or prominence of answers, said conforming comprises pruning said set of answers and questions to satisfy the criteria.

4. The computer-implemented method of claim 1, where said conforming comprises modifying said set of answers and questions to satisfy the criteria.

5. The computer-implemented method of claim 1, wherein said retrieving a set of documents from said initial set of questions comprises:

generating an input query to retrieve documents matching said query;

performing a query context analysis upon said input query to generate query terms;

conducting a search in said corpus, utilizing one of more searchable components of said query terms, to obtain passages potentially including candidate answers, all passages potentially including candidate answers being stored in a data storage device;

analyzing all retrieved passages and that passage'"'"'s metadata, in a candidate answer generation module, to generate an output plurality of data structures including candidate answers based upon the analyzing;

performing, by each of a plurality of parallel operating modules, supporting passage retrieval operation upon the set of candidate answers, and for each candidate answer, traversing the said data corpus and the said data storage device to find those passages having candidate answer in addition to query terms.

6. The computer-implemented method of claim 5, further comprising:

automatically scoring all candidate answers using supporting passages by a plurality of scoring modules, each producing a module score;

applying a candidate answer ranking function to the said modules scores to determine one or more query answers; and

,

generating a query response based on said one or more query answers for delivery to a user.

7. The computer-implemented method of claim 1, wherein said generating from a document, a candidate question and answer comprises:

receiving an input query, said input query comprising a string, a string with context, or a string with context wherein the context includes another string or data structure;

performing an automated query analysis including determining a lexical answer type; and

,

automatically computing candidate answers to the input query using said corpus.

8. The computer-implemented method of claim 7, wherein said analyzing a quality of answers of said conformed set of questions and answers comprises:

computing one or more lexical answer types (LAT) for each candidate answer;

utilizing an automated scoring function to compare candidate answer lexical types to the query LAT and producing a score for each candidate answer; and

,

returning one or more answers pared with a respective one or more questions based on the produced scores for delivery to a user.

9. The computer-implemented method of claim 1, wherein an initial question formed of a predicate has no corresponding uniquely determined answer, said method further comprising:

successively adding one or more additional predicates to ensure that the answer entity is uniquely determined by the predicate and the additional predicates.

10. The computer-implemented method of claim 1, further comprising:

automatically generating, for receipt by a first user via a first interface, questions and answers, a first user presenting a generated answer or question to a second user, to provide a respective responsive questions or answer, via a second interface.

11. A system for question-answer list generation comprising:

a memory device; and

a processor connected to the memory device, wherein the processor performs step of;

generating, from a corpus of text data and a set of criteria, one or more data structures;

generating, based on said set of criteria and one or more data structures, an initial set of questions;

retrieving a set of documents based on said initial set of questions;

generating from said documents, candidate question and answers;

conforming said set of candidate questions and answers to satisfy said set of criteria;

analyzing a quality of answers of said conformed set of questions and answers;

generating further one or more answers based on said analyzing; and

, outputting, based on said further one or more answers and said criteria, a final list question-answer (QA) pairs.

12. The system of claim 11, wherein said generating a candidate question and answer from said documents comprises:

generating, from said set of documents, predicate-argument sets; and

converting said predicate/argument sets of into a set of candidate questions and answers, an answer being one or more arguments of a predicate and the question being a transformation of the predicate and remaining arguments.

13. The system of claim 11, where said criteria includes one of:

a number of questions or prominence of answers, said conforming comprises pruning or modifying said set of answers and questions to satisfy the criteria.

14. The system of claim 11, wherein said retrieving a set of documents from said initial set of questions comprises:

generating an input query to retrieve documents matching said query;

performing a query context analysis upon said input query to generate query terms;

conducting a search in said corpus, utilizing one of more searchable components of said query terms, to obtain passages potentially including candidate answers, all passages potentially including candidate answers being stored in a data storage device;

analyzing all retrieved passages and that passage'"'"'s metadata, in a candidate answer generation module, to generate an output plurality of data structures including candidate answers based upon the analyzing;

performing, by each of a plurality of parallel operating modules, supporting passage retrieval operation upon the set of candidate answers, and for each candidate answer, traversing the said data corpus and the said data storage device to find those passages having candidate answer in addition to query terms.

15. The system of claim 14, further comprising:

automatically scoring all candidate answers using supporting passages by a plurality of scoring modules, each producing a module score;

applying a candidate answer ranking function to the said modules scores to determine one or more query answers; and

,

generating a query response based on said one or more query answers for delivery to a user.

16. The system of claim 11, wherein said generating from each document, a candidate question and answer comprises:

receiving an input query, said input query comprising a string, a string with context, or a string with context wherein the context includes another string or data structure;

performing an automated query analysis including determining a lexical answer type; and

,

automatically computing candidate answers to the input query using said corpus.

17. The system of claim 16, wherein said analyzing a quality of answers of said conformed set of questions and answers comprises:

computing one or more lexical answer types (LAT) for each candidate answer;

utilizing an automated scoring function to compare candidate answer lexical types to the query LAT and producing a score for each candidate answer; and

,

returning one or more answers pared with a respective one or more questions based on the produced scores for delivery to a user.

18. The system of claim 11, wherein an initial question formed of a predicate has no corresponding uniquely determined answer, said processor further performing:

successively adding one or more additional predicates to ensure that the answer entity is uniquely determined by the predicate and the additional predicates.

19. The system of claim 11, further comprising:

automatically generating, for receipt by a first user via a first interface, questions and answers, a first user presenting a generated answer or question to a second user, to provide a respective responsive questions or answer, via a second interface.

22. The computer program product of claim 11, where said criteria includes one of:

a number of questions or prominence of answers, said conforming comprises pruning or modifying said set of answers and questions to satisfy the criteria.

23. The computer program product of claim 11, wherein said retrieving a set of documents from said initial set of questions comprises:

generating an input query to retrieve documents matching said query;

performing a query context analysis upon said input query to generate query terms;

conducting a search in said corpus, utilizing one of more searchable components of said query terms, to obtain passages potentially including candidate answers, all passages potentially including candidate answers being stored in a data storage device;

analyzing all retrieved passages and that passage'"'"'s metadata, in a candidate answer generation module, to generate an output plurality of data structures including candidate answers based upon the analyzing;

performing, by each of a plurality of parallel operating modules, supporting passage retrieval operation upon the set of candidate answers, and for each candidate answer, traversing the said data corpus and the said data storage device to find those passages having candidate answer in addition to query terms.

24. The computer program product of claim 23, further comprising:

automatically scoring all candidate answers using supporting passages by a plurality of scoring modules, each producing a module score;

applying a candidate answer ranking function to the said modules scores to determine one or more query answers; and

,

generating a query response based on said one or more query answers for delivery to a user.

20. A computer program product for question-answer list generation, the computer program product comprising:

a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising;

computer readable program code configured to generate, from a corpus of text data and a set of criteria, one or more data structures;

computer readable program code configured to generate, based on said set of criteria and one or more data structures, an initial set of questions;

computer readable program code configured to retrieve a set of documents based on said initial set of questions;

computer readable program code configured to generate from said documents, candidate question and answers;

computer readable program code configured to conform said set of candidate questions and answers to satisfy said set of criteria;

computer readable program code configured to analyze a quality of answers of said conformed set of questions and answers;

computer readable program code configured to generate further one or more answers based on said analyzing; and

,

computer readable program code configured to output, based on said further one or more answers and said criteria, a final list question-answer (QA) pairs.

21. The computer program product of claim 20, wherein said generating a candidate question and answer from said documents comprises:

generating, from said set of documents, predicate-argument sets; and

converting said predicate/argument sets of into a set of candidate questions and answers, an answer being one or more arguments of a predicate and the question being a transformation of the predicate and remaining arguments.

25. A question answering (QA) system comprising:

a memory device; and

a processor connected to the memory device, wherein the processor performs step of;

automatically preparing a list of question/answer pairs, each consisting of a question and an answer, said preparing comprising;

providing a plurality of word or phrases based on a criteria;

selecting, an entity from among said plurality of entity word or phrases;

retrieving one or more documents including said entity;

automatically creating a question by selecting a predicate in a document within which the entity appears, and successively adding additional predicates to ensure that the entity is uniquely determined by the predicate and any additional predicates; and

,

setting the question to the predicate and the additional list of predicates retrieved, and setting the answer to the entity.

26. The question answering (QA) system according to claim 25, further comprising:

repeating, for each entity, said selecting a predicate and adding additional predicates, and formulating a respective answer for each said entity.

27. The question answering (QA) system according to claim 25, further comprising:

ordering a resulting list of question/answer pairs based on an additional criteria.