Multistage machine learning process

• US 6,513,025 B1
• Filed: 12/09/1999
• Issued: 01/28/2003
• Est. Priority Date: 12/09/1999
• Status: Expired due to Fees

• Current Assignee: Tollgrade Communications Incorporated (Enghouse Systems Limited)
• Original Assignee: Teradyne Incorporated
• Inventors: Rosen, Joseph S.
• Primary Examiner(s): STARKS, WILBERT L
• Application Number:
  US09/457,989
• Time in Patent Office: 1,146 Days
• Field of Search: 706/45, 706/12, 706/13, 706/47
• US Class Current: 706/45
• CPC Class Codes: G06N 5/025 Extracting rules from data

Abstract

The present invention includes a mechanism for applying expert knowledge and machine-learning routines to a continuous stream of information. The present method comprises learning a set of dependability models, one for each classification model, that characterize the situations in which each of the classification models is able to make correct predictions. At appropriate intervals the method produces new fault localization knowledge in the form of decision-tree based classification and dependability models. Such knowledge is used to enhance the existing classification knowledge already available. Each of these classification models has a particular sub-domain where it is the most reliable, and hence the best choice to use. For future unlabeled examples, these dependability models are consulted to select the most appropriate classification model, and the prediction of that classification model is then accepted.

Claims

1. A method of operating an expert system to diagnose service affecting conditions on lines in a telecommunications network comprising:

providing a set of training data, the data set made up of records with each record containing parameters indicative of measurements made on a line and a state of the line at the time the measurements were made;

selecting a plurality of subsets of the training data, wherein at least a first one of the subsets represents training data gathered during a first interval and at least a second one of the subsets representing training data gathered during a second interval after the first interval;

building a plurality of classification models, one from each of said plurality of subsets of the training data;

building a plurality of dependability models, one for each of said plurality of classification models;

providing test data derived from a line in the telecommunication network to each of said dependability models;

selecting a dependability model indicating the highest dependability from said plurality of dependability models; and

classifying the test data with the classification model associated with the dependability model indicating the highest dependability

providing additional training data during a third interval after the second interval;

using the additional training data to create additional classification models and additional dependability models;

replacing the classification models and dependability models created with a first of the subset of data with the additional classification models and additional dependability models; and

thereafter applying test data on a line in the telecommunication network gathered after the third interval to the dependability models to select a classification model.

2. The method of claim 1 wherein said step of building a plurality of classification models comprises building classification models of the type selected from the group consisting of a decision tree model, an expert system model and a case-based model.

4. The method of claim 1 wherein said step of building a plurality of classification models comprises building a plurality of classification model having an output which is an estimated class probability vector.

5. The method of claim 1 wherein said step of providing a set of training data includes the step of periodically providing a new set of training data.

6. The method of claim 5 wherein said step of periodically providing a new set of data comprises providing loop failure information data acquired since a previous cycle.

7. The method of claim 1 wherein said step of providing a set of training data further comprises the step of applying feature mapping rules to an initial set of data.

8. The method of claim 7 wherein said step of providing a set of training data further comprises the step of augmenting the initial set of data with derived features to promote more accurate classification model construction.

9. The method of claim 1 wherein said step of providing a set of training data further comprises the step of performing a plausibility analysis which removes or reclassifies data having implausible class labels.

10. The method of claim 1 wherein providing additional training data comprises providing the new data at a predetermined cycle time.

11. The method of claim 10 wherein said step of providing a new set of data at a predetermined cycle time is adjustable.

12. The method of claim 1 wherein said step of providing a set of training data further comprises the step of eliminating misclassified training data.

13. The method of claim 1 wherein said step of selecting a plurality of subsets of the training data comprises the step of utilizing a randomization technique to select a feature to partition the training data by.

14. The method of claim 13 wherein said step of utilizing a randomization technique comprises randomly selecting from among all the features having information gain ratio values within a feature-selection-percent of each other.

15. The method of claim 1, after said step of building a plurality of classification models, including the step of keeping up to a predetermined maximum number of classification models.

16. The method of claim 1 wherein said step of providing a set of training data comprises providing a set of subscriber loop test records.

17. The method of claim 16 wherein said step of building a plurality of dependability models comprises building dependability models which predict whether the class of an unclassified loop test example will be predicted correctly by a corresponding classification model.

18. The method of claim 1 wherein said step of building a plurality of dependability models includes the step of forming an initial training set.

21. The method of claim 1 wherein said step of providing a set of training data comprises providing a set of fault localization records.

3. A method of operating an expert system comprising:

providing a set of training data;

selecting a plurality of subsets of the training data;

building a plurality of classification models, one from each of said plurality of subsets of the training data;

building a plurality of dependability models, one for each of said plurality of classification models;

providing test data to each of said dependability models;

selecting a dependability model indicating the highest dependability from said plurality of dependability models; and

classifying the test data with the classification model associated with the dependability model indicating the highest dependability;

wherein said step of building a plurality of dependability models comprises building decision tree based dependability models.

19. A method of operating an expert system comprising:

providing a set of training data;

selecting a plurality of subsets of the training data;

building a plurality of classification models, one from each of said plurality of subsets of the training data;

building a plurality of dependability models, one for each of said plurality of classification models;

providing test data to each of said dependability models;

selecting a dependability model indicating the highest dependability from said plurality of dependability models; and

classifying the test data with the classification model associated with the dependability model indicating the highest dependability;

wherein said step of building a plurality of dependability models includes the step of computing a dependability training set from said initial training set.

20. The method of claim 19 wherein said step building a plurality of dependability models includes learning a dependability model based on said dependability training set.