Training data update

• US 10,372,826 B2
• Filed: 12/18/2017
• Issued: 08/06/2019
• Est. Priority Date: 09/15/2017
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Komine, Hiroaki, Maruyama, Kaori, Yanagawa, Takumi
• Primary Examiner(s): Yen, Eric
• Application Number:
  US15/845,031
• Publication Number:
  US 20190087411A1
• Time in Patent Office: 596 Days
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/268 Morphological analysis
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 5/048 Fuzzy inferencing

Abstract

Training data including a first and second group of questions each associated with an answer is read into memory by a computer. A class of answers including the answer to a question for each question is determined, where each class of answers has a class label that is associated with each of the questions, and each of the questions are classified into a respective class of answers, accordingly. First and second training data is generated including the first and second groups of questions and corresponding classes of answers for use in first and second classifiers, respectively. Each question of the first and second group of questions is classified by the second and first classifiers, respectively, where the classifying generates corresponding classification results. The first or second training data is updated based on the classification results to generate corresponding updated first or second training data, respectively.

Claims

1. A computer-implemented method for updating classifiers, the method comprising:

reading data including a plurality of questions into memory, the data including a first and a second group of questions;

generating first and second training data by;

identifying an answer for each question of the first and second groups of questions, each of the answers having a class label, and

associating each question of the first and second groups of questions with the answer identified for the question and a class corresponding with the class label of the identified answer,

wherein the first training data includes the first group of questions, and respective associated answers and classes, and the second training data includes the second group of questions, and respective associated answers and classes;

generating a first classifier based on the first training data and generating a second classifier based on the second training data;

classifying by the second classifier each question of the first group of questions, and classifying by the first classifier each question of the second group of questions, each question of the first and second groups of questions being classified into a corresponding class of answers, wherein the classifying by the first classifier generates second classification results and the classifying by the second classifier generates first classification results;

updating the first training data based on the first classification results and updating the second training data based on the second classification results; and

updating the first classifier using the updated first training data and updating the second classifier using the updated second training data.

2. The computer-implemented method of claim 1, wherein each of:

the classifying by the second classifier each question of the first group of questions, and the classifying by the first classifier each question of the second group of questions,

the updating the first training data based on the first classification results and the updating the second training data based on the second classification results, and

the updating the first classifier using the updated first training data and the updating the second classifier using the updated second training data are repeated until a degree of matching each question of the first group of questions exceeds a predetermined threshold value, both before a classifying by the first classifier and after a classifying by the second classifier, and a degree of matching each question of the second group of questions exceeds the predetermined threshold value, both before a classifying by the second classifier and after a classifying by the first classifier.

3. The computer-implemented method of claim 1 wherein each of:

the classifying by the second classifier each question of the first group of questions, and the classifying by the first classifier each question of the second group of questions,

the updating the first training data based on the first classification results and the updating the second training data based on the second classification results, and

the updating the first classifier using the updated first training data and the updating the second classifier using the updated second training data are repeated until a difference between consecutive degrees of matching each question of the first group of questions does not exceed a predetermined threshold value, both before a classifying by the first classifier and after a classifying by the second classifier, and a difference between consecutive degrees of matching each question of the second group of questions does not exceed the predetermined threshold value, both before a classifying by the second classifier and after a classifying by the first classifier.

4. The computer-implemented method of claim 1, wherein each of:

the classifying by the second classifier each question of the first group of questions, and the classifying by the first classifier each question of the second group of questions,

the updating the first training data based on the first classification results and the updating the second training data based on the second classification results, and

the updating the first classifier using the updated first training data and the updating the second classifier using the updated second training data are repeated until a moving average value of the degrees of matching each question of the first group of questions exceeds a predetermined threshold value, both before a classifying by the first classifier and after a classifying by the second classifier, and a moving average value of the degrees of matching each question of the second group of questions exceeds the predetermined threshold value, both before a classifying by the second classifier and after a classifying by the first classifier.

5. The computer-implemented method of claim 1, wherein the first and second classification results include degree of confidence values corresponding to the classifying of each question with respect to corresponding classifications by the first and second classifiers, respectively, and wherein updating of the first and second training data occurs when a degree of confidence value exceeds a predetermined threshold value.

6. The computer-implemented method of claim 1, wherein the generating of first and second training data further comprises:

determining a degree of confidence value for each answer identified for a question of the first and second groups of questions; and

identifying each answer based on one of;

a highest relative degree of confidence value for the answer, and a degree of confidence value for the answer that exceeds a predetermined threshold value.

7. The computer-implemented method of claim 1, wherein the first and second classification results include degree of confidence values corresponding to the classifying of each question with respect to corresponding classifications by the first and second classifiers, respectively, and wherein updating the first and second training data comprises:

reclassifying a question from one class into another class when a degree of confidence value for the question exceeds a predetermined threshold value.

8. The computer-implemented method of claim 1, the method further comprising:

dividing the data into the first and second group of questions.

9. The computer-implemented method of claim 8,

wherein the first and second classification results include degree of confidence values corresponding to the classifying of each question with respect to corresponding classifications by the first and second classifiers, respectively, and

wherein the associating each question with an answer is repeated until a lowest relative degree of confidence value that does not exceed a predetermined threshold value occurs.

10. The computer-implemented method of claim 1, wherein the first and second classification results include degree of confidence values corresponding to the classifying of each question with respect to corresponding classifications by the first and second classifiers, respectively, and wherein the associating each question of the first and second groups of questions with the answer identified for the question further comprises:

searching an answer corpus using a search engine.

11. The computer-implemented method of claim 10, wherein each associated answer is of a class of answers having a highest relative degree of confidence value.

12. The computer-implemented method of claim 1, wherein the associating each question of the first and second groups of questions with the answer identified for the question is repeated until a relative degree of confidence value for an answer exceeds a lowest relative degree of confidence value.

13. A computer-implemented method for updating classifiers, the method comprising:

dividing a group of questions into a first and a second group of questions;

associating each question with an answer;

reading the first and second groups of questions into memory, wherein each question is associated with an answer;

determining a class of answers for each question of the first and second groups of questions, the class of answers having a class label being associated with each of the questions, and each of the questions being classified into a respective class of answers;

generating first and second training data comprising the first and second groups of questions and corresponding classes of answers for use in first and second classifiers, respectively;

classifying by the second classifier each question of the first group of questions, and classifying by the first classifier each question of the second group of questions, each question of the first and second groups being classified into a corresponding class of answers, wherein the classifying by the first classifier generates second classification results and the classifying by the second classifier generates first classification results;

updating the first training data based on the first classification results and updating the second training data based on the second classification results; and

updating the first classifier using the updated first training data and updating the second classifier using the updated second training data.