Training data update

• US 10,387,572 B2
• Filed: 09/15/2017
• Issued: 08/20/2019
• Est. Priority Date: 09/15/2017
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Komine, Hiroaki, Maruyama, Kaori, Yanagawa, Takumi
• Primary Examiner(s): Yen, Eric
• Application Number:
  US15/705,596
• Publication Number:
  US 20190087408A1
• Time in Patent Office: 704 Days
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/268 Morphological analysis
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 5/048 Fuzzy inferencing

Abstract

Training data including a first and second group of questions each associated with an answer is read into memory by a computer. A class of answers including the answer to a question for each question is determined, where each class of answers has a class label that is associated with each of the questions, and each of the questions are classified into a respective class of answers, accordingly. First and second training data is generated including the first and second groups of questions and corresponding classes of answers for use in first and second classifiers, respectively. Each question of the first and second group of questions is classified by the second and first classifiers, respectively, where the classifying generates corresponding classification results. The first or second training data is updated based on the classification results to generate corresponding updated first or second training data, respectively.

Claims

1. A computer system comprising:

one or more computer processors, one or more computer-readable storage media, and program instructions stored on the one or more of the computer-readable storage media for execution by at least one of the one or more computer processors, the program instructions, when executed by the at least one of the one or more computer processors, causing the computer system to perform a method for updating a classifier, the method comprising;

reading data including a plurality of questions into memory, the data including a first and a second group of questions, wherein each question is associated with an answer;

determining a class of answers for each question of the first and second groups of questions, the class of answers having a class label being associated with each of the questions, and each of the questions being classified into a respective class of answers;

generating first and second training data, the first training data including the first group of questions and corresponding classes of answers for use in a first classifier, and the second training data including the second groups of questions and corresponding classes of answers for use in a second classifier;

classifying by the second classifier each question of the first group of questions, and classifying by the first classifier each question of the second group of questions, each question of the first and second groups of questions being classified into a corresponding class of answers, wherein the classifying by the first classifier generates second classification results and the classifying by the second classifier generates first classification results;

updating the first training data based on the first classification results and updating the second training data based on the second classification results; and

updating the first classifier using the updated first training data and updating the second classifier using the updated second training data.

2. A computer program product comprising:

one or more computer-readable storage devices and program instructions stored on at least one of the one or more computer-readable storage devices for execution by at least one or more computer processors of a computer system, the program instructions, when executed by the at least one of the one or more computer processors, causing the computer system to perform a method for updating a classifier, the method comprising;

reading data into memory, the data including a first and a second group of questions, wherein each question is associated with an answer;

determining a respective class of answers for each question of the first and second groups of questions, the class of answers having a class label being associated with each of the questions, and each of the questions being classified into a respective class of answers;

generating first and second training data, the first training data including the first group of questions and corresponding classes of answers for use in a first classifier, and the second training data including the second groups of questions and corresponding classes of answers for use in a second classifier;

classifying by the second classifier each question of the first group of questions, and classifying by the first classifier each question of the second group of questions, each question of the first and second groups of questions being classified into a corresponding class of answers, wherein the classifying by the first classifier generates second classification results and the classifying by the second classifier generates first classification results;

updating the first training data based on the first classification results and updating the second training data based on the second classification results;

and

updating the first classifier using the updated first training data and updating the second classifier using the updated second training data.

3. The computer program product of claim 2, wherein each of:

the classifying, by the second classifier, each question of the first group of questions, and classifying, by the first classifier, each question of the second group of questions,

the updating the first and second training data, and

the updating of the first and second classifiers are repeated until;

a degree of matching between questions in a class of the first training data before and after the classifying, by the second classifier, exceeds a predetermined threshold value, and

a degree of matching between questions in a class of the second training data before and after the classifying, by the first classifier, exceeds the predetermined threshold value.

4. The computer program product of claim 2, wherein each:

the classifying, by the second classifier, each question of the first group of questions, and classifying, by the first classifier each question of the second group of questions,

the updating the first and second training data, and

the updating of the first and second classifiers

are repeated until;

a difference between consecutive degrees of matching between questions in a class of the first training data before and after the classifying, by the second classifier, does not exceed a predetermined threshold value, and

a difference between consecutive degrees of matching between questions in a class of the second training data before and after the classifying, by the first classifier, does not exceed the predetermined threshold value.

5. The computer program product of claim 2, wherein each:

the classifying by the second classifier each question of the first group of questions, and classifying by the first classifier each question of the second group of questions,

the updating the first and second training data, and

the updating of the first and second classifiers

are repeated until;

a moving average value of degrees of matching between questions in a class of the first training data before and after the classifying, by the second classifier, exceeds a predetermined threshold value, and

a moving average value of degrees of matching between questions in a class of the second training data before and after the classifying, by the first classifier, exceeds the predetermined threshold value.

6. The computer program product of claim 2, wherein the first and second classification results include degree of confidence values corresponding to the classifying of each question with respect to corresponding classes of answers by the first and second classifiers, respectively, and wherein updating the first and second training data occurs when a degree of confidence value exceeds a predetermined threshold value.

7. The computer program product of claim 2, wherein the determining a respective class of answers for each question of the first and second groups of questions includes:

determining, for each question of the first and second groups of questions, a degree of confidence value for each class of answers comprising the answer to the question; and

determining, for each question of the first and second groups of questions, the class of answers based on one of;

the class of answers having a highest degree of confidence value relative to a degree of confidence of the question with respect to another class, and the class of answers having a corresponding degree of confidence value that exceeds a predetermined threshold value.

8. The computer program product of claim 2, wherein the first and second classification results comprise degree of confidence values corresponding to the classifying of each question with respect to corresponding classes of answers by the first and second classifiers, respectively, and wherein updating the first and second training data comprises:

reclassifying a question from one class of answers into another class of answers where a corresponding degree of confidence value exceeds a predetermined threshold value.

9. The computer program product of claim 2, the method further comprising:

dividing the data into the first and the second groups of questions; and

associating each question with an answer.

10. The computer program product of claim 9,

wherein the first and second classification results include degree of confidence values corresponding to the classifying of each question with respect to corresponding classes of answers by the first and second classifiers, respectively, and

wherein the associating each question with an answer is iterated until a classifying of a question with respect to a corresponding class of answers having a lowest degree of confidence value relative to a degree of confidence of the question with respect to another class of answers does not exceed a predetermined threshold value.

11. The computer program product of claim 9, wherein the first and second classification results comprise degree of confidence values corresponding to the classifying of each question with respect to corresponding classes of answers by the first and second classifiers, respectively, and wherein the associating each question with an answer comprises:

using a search engine and an answer corpus.

12. The computer program product of claim 11, wherein each question is classified into a class of answers having a highest degree of confidence value relative to a degree of confidence value of the question with respect to other classes of answers.

13. The computer program product of claim 10, wherein the associating each question with an answer is iterated until a classifying of a question with respect to a corresponding class of answers having a degree of confidence value relative to a degree of confidence of the question with respect to another class of answers exceeds the lowest relative degree of confidence value.