Training data update

• US 10,614,269 B2
• Filed: 06/28/2019
• Issued: 04/07/2020
• Est. Priority Date: 09/15/2017
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Komine, Hiroaki, Maruyama, Kaori, Yanagawa, Takumi
• Primary Examiner(s): Yen, Eric
• Application Number:
  US16/455,959
• Publication Number:
  US 20190317998A1
• Time in Patent Office: 284 Days
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/268 Morphological analysis
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 5/048 Fuzzy inferencing

Abstract

Training data including a first and second group of questions each associated with an answer is read into memory by a computer. A class of answers including the answer to a question for each question is determined, where each class of answers has a class label that is associated with each of the questions, and each of the questions are classified into a respective class of answers, accordingly. First and second training data is generated including the first and second groups of questions and corresponding classes of answers for use in first and second classifiers, respectively. Each question of the first and second group of questions is classified by the second and first classifiers, respectively, where the classifying generates corresponding classification results. The first or second training data is updated based on the classification results to generate corresponding updated first or second training data, respectively.

Claims

1. A computer system comprising:

one or more computer processors, one or more computer-readable storage media, and program instructions stored on the one or more of the computer-readable storage media for execution by at least one of the one or more computer processors, the program instructions, when executed by the at least one of the one or more computer processors, causing the computer system to perform a method for updating first and second classifiers, the method comprising;

reading data including a plurality of questions into memory, the data including a first and a second group of questions, wherein each question is associated with an answer;

determining a class of answers for each question of the first and second groups of questions, each of the questions being classified into a respective class of answers;

generating first classification results by classifying, using the second classifier, each question of the first group of questions into a corresponding class of answers;

generating second classification results by classifying, using the first classifier, each question of the second group of questions into a corresponding class of answers;

updating first training data based on the first classification results;

updating second training data based on the second classification results;

updating the first classifier using the updated first training data; and

updating the second classifier using the updated second training data.

2. A computer program product comprising:

one or more computer-readable storage devices and program instructions stored on at least one of the one or more computer-readable storage devices for execution by at least one or more computer processors of a computer system, the program instructions, when executed by the at least one of the one or more computer processors, causing the computer system to perform a method for updating first and second classifiers, the method comprising;

determining a respective class of answers for each question of first and second groups of questions, each of the questions being classified into a respective class of answers;

generating first classification results by classifying, using the second classifier, each question of the first group of questions into a corresponding class of answers;

generating second classification results by classifying, using the first classifier, each question of the second group of questions into a corresponding class of answers;

updating first training data based on the first classification results;

updating second training data based on the second classification results;

updating the first classifier using the updated first training data; and

updating the second classifier using the updated second training data.

3. The computer program product of claim 2, further comprising:

generating first training data, the first training data including the first group of questions and corresponding classes of answers for use in generating the first classifier; and

generating second training data, the second training data including the second group of questions and corresponding classes of answers for use in generating the second classifier.

4. The computer program product of claim 2, wherein each of:

the generating first classification results by classifying, using the second classifier, each question of the first group of questions into a corresponding class of answers,

the generating second classification results by classifying, using the first classifier, each question of the second group of questions into a corresponding class of answers;

the updating the first and second training data, and

the updating of the first and second classifiers

are repeated until;

a degree of matching between questions in a class of the first training data before and after the classifying, by the second classifier, exceeds a predetermined threshold value, and

a degree of matching between questions in a class of the second training data before and after the classifying, by the first classifier, exceeds the predetermined threshold value.

5. The computer program product of claim 2, wherein each:

the generating first classification results by classifying, using the second classifier, each question of the first group of questions into a corresponding class of answers,

the generating second classification results by classifying, using the first classifier, each question of the second group of questions into a corresponding class of answers;

the updating the first and second training data, and

the updating of the first and second classifiers

are repeated until;

a difference between consecutive degrees of matching between questions in a class of the first training data before and after the classifying, by the second classifier, does not exceed a predetermined threshold value, and

a difference between consecutive degrees of matching between questions in a class of the second training data before and after the classifying, by the first classifier, does not exceed the predetermined threshold value.

6. The computer program product of claim 2, wherein each:

the generating first classification results by classifying, using the second classifier, each question of the first group of questions into a corresponding class of answers,

the generating second classification results by classifying, using the first classifier, each question of the second group of questions into a corresponding class of answers;

the updating the first and second training data, and

the updating of the first and second classifiers

are repeated until;

a moving average value of degrees of matching between questions in a class of the first training data before and after the classifying, by the second classifier, exceeds a predetermined threshold value, and

a moving average value of degrees of matching between questions in a class of the second training data before and after the classifying, by the first classifier, exceeds the predetermined threshold value.

7. The computer program product of claim 2, wherein the first and second classification results include degree of confidence values corresponding to the classifying of each question with respect to corresponding classes of answers by the first and second classifiers, respectively, and wherein updating the first and second training data occurs when a degree of confidence value exceeds a predetermined threshold value.

8. The computer program product of claim 2, wherein the determining a respective class of answers for each question of the first and second groups of questions includes:

determining, for each question of the first and second groups of questions, a degree of confidence value for the respective class of answers; and

determining, for each question of the first and second groups of questions, the class of answers based on one of;

the class of answers having a highest degree of confidence value relative to a degree of confidence of the question with respect to another class, and the class of answers having a corresponding degree of confidence value that exceeds a predetermined threshold value.

9. The computer program product of claim 2, wherein the first and second classification results comprise degree of confidence values corresponding to the classifying of each question with respect to corresponding classes of answers by the first and second classifiers, respectively, and wherein updating the first and second training data comprises:

reclassifying a question from one class of answers into another class of answers where a corresponding degree of confidence value exceeds a predetermined threshold value.

10. The computer program product of claim 2, further comprising:

reading data into memory, the data including the first and second groups of questions, wherein each question is associated with an answer.

11. The computer program product of claim 2, the method further comprising:

associating each question of the first and the second groups of questions with an answer.

12. The computer program product of claim 11,

wherein the first and second classification results include degree of confidence values corresponding to the classifying of each question with respect to corresponding classes of answers by the first and second classifiers, respectively, and

wherein the associating each question with an answer is iterated until a classifying of a question with respect to a corresponding class of answers having a lowest degree of confidence value relative to a degree of confidence of the question with respect to another class of answers does not exceed a predetermined threshold value.

13. The computer program product of claim 11, wherein the first and second classification results comprise degree of confidence values corresponding to the classifying of each question with respect to corresponding classes of answers by the first and second classifiers, respectively, and wherein the associating each question with an answer comprises:

using a search engine and an answer corpus.

14. The computer program product of claim 13, wherein each question is classified into a class of answers having a highest degree of confidence value relative to a degree of confidence value of the question with respect to other classes of answers.

15. The computer program product of claim 11, wherein the associating each question with an answer is iterated until a classifying of a question with respect to a corresponding class of answers having a degree of confidence value relative to a degree of confidence of the question with respect to another class of answers exceeds a lowest relative degree of confidence value.