Enhancing knowledge discovery from multiple data sets using multiple support vector machines

• US 6,658,395 B1
• Filed: 05/24/2000
• Issued: 12/02/2003
• Est. Priority Date: 05/01/1998
• Status: Expired due to Term

• Current Assignee: Curtis Anderson, Health Discovery Corporation, James Roberts, Joe Mckenzie, Jules B. Paderewski, Julian N. Stern, Memorial Health Systems Incorporated, Timothy P. O'Hayer
• Original Assignee: BIOwulf Technologies, LLC
• Inventors: Barnhill, Stephen D.
• Primary Examiner(s): Davis, George B.
• Application Number:
  US09/578,011
• Time in Patent Office: 1,287 Days
• Field of Search: 706/16, 706/12, 706/45, 706/25, 706/20
• US Class Current: 706/16
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]

Abstract

A system and method for enhancing knowledge discovery from data using multiple learning machines in general and multiple support vector machines in particular. Training data for a learning machine is pre-processed in order to add meaning thereto. Pre-processing data may involve transforming the data points and/or expanding the data points. By adding meaning to the data, the learning machine is provided with a greater amount of information for processing. With regard to support vector machines in particular, the greater the amount of information that is processed, the better generalizations about the data that may be derived. Multiple support vector machines, each comprising distinct kernels, are trained with the pre-processed training data and are tested with test data that is pre-processed in the same manner. The test outputs from multiple support vector machines are compared in order to determine which of the test outputs if any represents a optimal solution. Selection of one or more kernels may be adjusted and one or more support vector machines may be retrained and retested. Optimal solutions based on distinct input data sets may be combined to form a new input data set to be input into one or more additional support vector machine.

Claims

1. A computer-implemented method for extracting information from large data sets using multiple support vector machines comprising:

(a) receiving a training input comprising a plurality of training data sets containing a plurality of training data points of different data types;

(b) pre-processing each of a first training data set comprising a first data type and a second training data set comprising a second data type to add dimensionality to each of the training data points within the first and second data sets;

(c) training a first plurality of first-level support vector machines using the first pre-processed training data set, each first-level support vector machine of the first plurality comprising a plurality of different kernels selected from a first set of kernels;

(d) training a second plurality of first-level support vector machines using the second pre-processed training data set, each first-level support vector machine of the second plurality comprising a second plurality of different kernels selected from a second set of kernels;

(e) receiving test input comprising a plurality of test data sets containing a plurality of test data points of the different data types;

(f) pre-processing each of a first test data set comprising the first data type and a second test data set comprising the second data type to add dimensionality to each of the test data points within the first and second test data sets;

(g) testing each of the first plurality of trained first-level support vector machines using the first pre-processed test data set to generate a first plurality of test outputs;

(h) testing each of the second plurality of trained first-level support vector machines using the second pre-processed test data set to generate a second plurality of test outputs;

(i) identifying a first optimal solution, if any, from the first plurality of test outputs;

(j) identifying a second optimal solution, if any, from the second plurality of test outputs;

(k) combining the first optimal solution and the second optimal solution to create a second-level input data set to be input into each of a plurality of second-level support vector machines;

(l) generating a second-level output for each second-level support vector machine; and

(m) comparing the second-level outputs to identify and optimal second-level solution.

2. The method of claim 1, wherein step (b) further comprises:

determining that at least one of the training data points is dirty; and

in response to determining that the training data point is dirty, cleaning the dirty training data point.

3. The method of claim 2, wherein cleaning the dirty training data point comprises deleting, repairing or replacing the data point.

4. The method of claim 1, wherein each training data point comprises

a vector having one or more original coordinates;

and

wherein pre-processing the training data set comprises adding one or more new coordinates to the vector.

5. The method of claim 4, wherein the one or more new coordinates added to the vector are derived by applying a transformation to one or more of the original coordinates.

6. The method of claim 5, wherein the transformation is based on expert knowledge.

7. The method of claim 5, wherein the transformation is computationally derived.

8. The method of claim 7, wherein the training data set comprises a continuous variable;

and

wherein the transformation comprises optimally categorizing the continuous variable of the training data set.

9. The method of claim 1, wherein step (i) comprises:

post-processing each of the first test outputs by interpreting each of the test outputs into a common format; and

comparing each of the post-processed first test outputs with each other to determine which of the first test outputs represents a first lowest global minimum error.

10. The method of claim 1, wherein the information to be extracted from the data relates to a regression or density estimation;

wherein each support vector machine produces a training output comprising a continuous variable; and

wherein the method further comprises the step of post-processing each of the training outputs by optimally categorizing the training output to derive cutoff points in the continuous variable.

11. The method of claim 1, further comprising the steps of:

if step (i) identifies no optimal solution;

selecting different kernels for each first-level support vector machines; and

repeating steps (d), (e), (h) and (j).

12. The method of claim 11, wherein the step of selecting different kernels is performed based on prior performance or historical data and is dependant on the nature of the data.

13. The method of claim 1, wherein step (j) comprises:

post-processing each of the second test outputs by interpreting each of the second test outputs into a common format; and

comparing each of the post-processed second test outputs with each other to determine which of the second test outputs represents a second lowest global minimum error.