Question answering system-based generation of distractors using machine learning

• US 9,684,876 B2
• Filed: 03/30/2015
• Issued: 06/20/2017
• Est. Priority Date: 03/30/2015
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Parikh, Ankur, Agarwalla, Lalit, Mungi, Ashish, Mustafi, Joy
• Primary Examiner(s): Yao, Sam
• Assistant Examiner(s): Siebert, Steven
• Application Number:
  US14/672,286
• Publication Number:
  US 20160293034A1
• Time in Patent Office: 813 Days
• Field of Search: 434322
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G09B 7/06 of the multiple-choice answer-type, i.e. where a given question is provided with a series of answers and a choice has to be made from the answers

Abstract

Generating distractors for text-based MCT items. An MCT item stem is received. The stem is transmitted to a QA system and a plurality of candidate answers related to the stem is received from the QA system. Incorrect answers in the plurality of candidate answers are identified. Textual features are extracted from the stem. A set of semantic criteria associated with the extracted textual features is generated. Based on the generated semantic criteria, a subset of the incorrect candidate answers is selected.

Claims

1. A computer program product for generating distractors for text-based multiple choice test (MCT) items, the computer program product comprising:

one or more computer-readable storage media and program instructions stored on the one or more computer-readable storage media, the program instructions, when executed by a first computer, cause the first computer to perform a method comprising;

receiving, by the first computer from a second computer over a network, in response to information entered into a user interface on the second computer, an MCT item stem and key;

submitting, by the first computer, the stem to a Question Answering (QA) system, wherein a QA system is a computer system that, in response to a query, automatically generates a list of candidate answers to the query;

in response to submitting the stem to the QA system, receiving, by the first computer, from the QA system a plurality of candidate answers;

identifying, by the first computer, one or more incorrect answers in the plurality of candidate answers;

extracting, by the first computer, textual features from the stem wherein a textual feature is a term in the stem or a concept semantically related to a term in the stem;

applying, by the first computer, a machine learning model to generate a set of semantic criteria associated with the extracted textual features;

selecting, by the first computer, as distractors one or more of the incorrect candidate answers that satisfy the generated semantic criteria;

creating, by the first computer, an MCT item that includes the stem, the key, and the distractors; and

transmitting, by the first computer, the created MCT item via the network to the second computer.

2. A computer program product in accordance to claim 1, wherein the stem is one of:

a question to be answered, a problem to be solved, or an incomplete statement to be completed.

3. A computer program product in accordance to claim 1, wherein the MCT item includes a predefined number of alternative choices, and the plurality of candidate answers is at least the predefined number of alternative choices and less than a predefined maximum number.

4. A computer program product in accordance to claim 1,

wherein identifying, by the first computer, incorrect answers comprises:

identifying, by the first computer, a correct answer by comparing the plurality of candidate answers with the key; and

deleting, by the first computer, the correct answer from the plurality of candidate answers.

5. A computer program product in accordance to claim 1,

wherein semantic criteria associated with the extracted textual features are attributes, categories, or facts relevant to the extracted textual features, and wherein

applying, by the first computer, a machine learning model to generate the set of semantic criteria associated with the extracted textual features further comprises:

training, by the first computer, the machine learning model using a predefined set of semantic criteria; and

refining, by the first computer, the generated set of semantic criteria by applying a predefined set of rules.

6. A computer program product in accordance to claim 5, wherein applying, by the first computer, a trained machine learning model further comprises using, by the first computer, multi-label classification.

7. A computer program product in accordance to claim 1, wherein selecting a subset of the incorrect candidate answers, based on the generated semantic criteria, comprises:

generating, by the first computer, hypotheses concerning the semantic criteria and the incorrect candidate answers;

collecting, by the first computer, evidence supporting or refuting each hypothesis;

assigning, by the first computer, to each hypothesis a numerical value representing a degree to which the evidence justifies or refutes the hypothesis; and

selecting, by the first computer, the subset of the incorrect candidate answers based on the numerical value.

8. A computer system for generating distractors for text-based multiple choice test (MCT) items, the computer system comprising:

one or more computer processors, one or more computer-readable storage media, and program instructions stored on one or more of the computer-readable storage media, which when executed cause at least one of the one or more processors to perform a method comprising;

receiving, by the first computer from a second computer over a network, in response to information entered into a user interface on the second computer, an MCT item stem and key;

submitting, by the first computer, the stem to a Question Answering (QA) system, wherein a QA system is a computer system that, in response to a query, automatically generates a list of candidate answers to the query;

in response to submitting the stem to the QA system, receiving, by the first computer, from the QA system a plurality of candidate answers;

identifying, by the first computer, one or more incorrect answers in the plurality of candidate answers;

extracting, by the first computer, textual features from the stem, wherein a textual feature is a term in the stem or a concept semantically related to a term in the stem;

applying, by the first computer, a machine learning model to generate a set of semantic criteria associated with the extracted textual features;

selecting, by the first computer, as distractors one or more of the incorrect candidate answers that satisfy the generated semantic criteria;

creating, by the first computer, an MCT item that includes the stem, the key, and the distractors; and

transmitting, by the first computer, the created MCT item via the network to the second computer.

9. A computer system in accordance to claim 8, wherein the stem is one of:

a question to be answered, a problem to be solved, or an incomplete statement to be completed.

10. A computer system in accordance to claim 8, wherein the MCT item includes a predefined number of alternative choices, and the plurality of candidate answers is at least the predefined number of alternative choices and less than a predefined maximum number.

11. A computer system in accordance to claim 8,

wherein identifying, by the first computer, incorrect answers comprises:

identifying, by the first computer, a correct answer by comparing the plurality of candidate answers with the key; and

deleting, by the first computer, the correct answer from the plurality of candidate answers.

12. A computer system in accordance to claim 8,

wherein semantic criteria associated with the extracted textual features are attributes, categories, or facts relevant to the extracted textual features, and wherein

applying, by the first computer, a machine learning model to generate the set of semantic criteria associated with the extracted textual features further comprises:

training, by the first computer, the machine learning model using a predefined set of semantic criteria; and

refining, by the first computer, the generated set of semantic criteria by applying a predefined set of rules.

13. A computer system in accordance to claim 8, wherein selecting a subset of the incorrect candidate answers, based on the generated semantic criteria, comprises:

generating, by the first computer, hypotheses concerning the semantic criteria and the incorrect candidate answers;

collecting, by the first computer, evidence supporting or refuting each hypothesis;

assigning, by the first computer, to each hypothesis a numerical value representing a degree to which the evidence justifies or refutes the hypothesis; and

selecting, by the first computer, the subset of the incorrect candidate answers based on the numerical value.

14. A computer system in accordance to claim 12, wherein applying, by the first computer, a machine learning model further comprises using, by the first computer, multi-label classification.