Enhancing knowledge discovery using multiple support vector machines

• US 6,427,141 B1
• Filed: 05/09/2000
• Issued: 07/30/2002
• Est. Priority Date: 05/01/1998
• Status: Expired due to Fees

• Current Assignee: Curtis Anderson, Health Discovery Corporation, James Roberts, Joe Mckenzie, Jules B. Paderewski, Julian N. Stern, Memorial Health Systems Incorporated, Timothy P. O'Hayer
• Original Assignee: BIOwulf Technologies, LLC
• Inventors: Barnhill, Stephen D.
• Primary Examiner(s): Davis, George B.
• Application Number:
  US09/568,301
• Time in Patent Office: 812 Days
• Field of Search: 706/16, 706/12, 706/45, 706/25, 706/20
• US Class Current: 706/16
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]

Abstract

A system and method for enhancing knowledge discovery from data using multiple learning machines in general and multiple support vector machines in particular. Training data for a learning machine is pre-processed in order to add meaning thereto. Pre-processing data involves transforming the data points and/or expanding the data points. By adding meaning to the data, the learning machine is provided with a greater amount of information for processing. With regard to support vector machines in particular, the greater the amount of information that is processed, the better generalizations about the derived data. Multiple support vector machines, each comprising distinct kernels, are trained with the pre-processed training data and are tested with test data that is pre-processed in the same manner. The test outputs from multiple support vector machines are compared in order to determine which of the test outputs if any represents a optimal solution. Selection of one or more kernels is to be adjusted and one or more support vector machines is to be retrained and retested. When it is determined that an optimal solution has been achieved, live data is pre-processed and input into the support vector machine comprising the kernel that produced the optimal solution. The live output from the learning machine is post-processed into a computationally derived alphanumerical classifier for interpretation by a human or computer automated process.

Claims

1. A method for enhancing knowledge discovery using multiple support vector machines comprising:

(a) pre-processing a training data set to add dimensionality to each of a plurality of training data points;

(b) training each of a plurality of support vector machines using the pre-processed training data set, each support vector machine comprising a different kernel;

(c) pre-processing a test data set in the same manner as was the training data set;

(d) testing each of the plurality of trained support vector machines using the pre-processed test data set; and

(e) in response to receiving a test output from each of the plurality of trained support vector machines, comparing each of the test outputs with each other to determine which if any of the test outputs is an optimal solution.

2. The method of claim 1, further comprising programming a computer with computer-executable instructions corresponding to steps (a) through (e) and storing the computer-executable instructions on a computer-readable medium.

3. The method of claim 1,

wherein pre-processing the training data set further comprises:

determining that at least one of the training data points is dirty; and

in response to determining that one of the training data points is dirty, cleaning the dirty training data point.

4. The method of claim 3, wherein each training data point comprises a vector having one or more coordinates;

and

wherein cleaning the training data point comprises deleting, repairing or replacing one or more of the coordinates of the data point.

5. The method of claim 4, further comprising programming a computer with computer-executable instructions corresponding to steps (a) through (e) and storing the computer-executable instructions on a computer-readable medium.

6. The method of claim 1, wherein each training data point comprises a vector having one or more original coordinates;

and

wherein adding dimensionality to each of the plurality of training data points comprises adding one or more new coordinates to each of the vectors, the new coordinates being derived by applying a transformation to one or more of the original coordinates.

7. The method of claim 6, wherein the transformation is based on expert knowledge.

8. The method of claim 6, wherein the transformation is computationally derived.

9. The method of claim 6, wherein the training data set comprises a continuous variable;

and

wherein the transformation comprises optimally categorizing the continuous variable of the training data set.

10. The method of claim 9, further comprising programming a computer with computer-executable instructions corresponding to steps (a) through (e) and storing the computer-executable instructions on a computer-readable medium.

11. The method of claim 1, wherein comparing each of the test outputs with each other comprises:

post-processing each of the test outputs by interpreting each of the test outputs into a common format; and

comparing each of the post-processed test outputs with each other to determine which of the test outputs represents a lowest global minimum error.

12. The method of claim 11, further comprising programming a computer with computer-executable instructions corresponding to steps (a) through (e) and storing the computer-executable instructions on a computer-readable medium.

13. The method of claim 1, wherein the knowledge to be discovered from the data relates to a regression or density estimation;

wherein each support vector machine produces a training output comprising a continuous variable; and

wherein the method further comprises the step of post-processing each of the training outputs by optimally categorizing the training output to derive cutoff points in the continuous variable.

14. The method of claim 1, further comprising the steps of:

(f) in response to comparing each of the test outputs with each other, determining that none of the test outputs is the optimal solution;

(g) adjusting the different kernels of one or more of the plurality of support vector machines; and

(h) in response to adjusting the different kernels, retraining and retesting each of the plurality of support vector machines.

15. The method of claim 14, further comprising programming a computer with computer-executable instructions corresponding to steps (a) through (h) and storing the computer-executable instructions on a computer-readable medium.

16. The method of claim 14, wherein adjusting the different kernels is performed based on prior performance or historical data and is dependant on the nature of the knowledge to be discovered from the data or the nature of the data.

17. The method of claim 1, further comprising the steps of:

(f) in response to comparing each of the test outputs with each other, determining that a selected one of the test outputs is the optimal solution, the selected one of the test outputs produced by a selected one of the plurality of trained support vector machines comprising a selected kernel;

(g) collecting a live data set;

(h) pre-processing the live data set in the same manner as was the training data set;

(i) inputting the pre-processed live data set into the selected trained support vector machine comprising the selected kernel; and

(j) receiving a live output from the selected trained support vector machine.

18. The method of claim 17, further comprising programming a computer with computer-executable instructions corresponding to steps (a) through (i) and storing the computer-executable instructions on a computer-readable medium.

19. The method of claim 17, further comprising the step of post-processing the live output by interpreting the live output into a computationally derived alphanumerical classifier.

20. The method of claim 1, further comprising the steps of:

(f) in response to comparing each of the test outputs with each other, determining that a selected one of the test outputs is the optimal solution, the selected one of the test outputs produced by a selected one of the plurality of trained support vector machines comprising a selected kernel;

(g) collecting a live data set;

(h) pre-processing the live data set in the same manner as was the training data set;

(i) configuring two or more of the plurality of support vector machines for parallel processing based on the selected kernel;

(j) inputting the pre-processed live data set into the support vector machines configured for parallel processing; and

(k) receiving a live output from the trained support vector machine.

21. The method of claim 20, further comprising programming a computer with computer-executable instructions corresponding to steps (a) through (k) and storing the computer-executable instructions on a computer-readable medium.

22. A method for enhancing knowledge discovery relating to a regression or density estimation using multiple support vector machines, each support vector machine comprising a different kernel, the method comprising:

(a) pre-processing a training data set to add dimensionality to each of a plurality of training data points;

(b) training each of a plurality of support vector machines using the pre-processed training data set such that each support vector machine produces a training output comprising a continuous variable;

(c) post-processing each of the training outputs by optimally categorizing each of the training outputs to derive cutoff points in the continuous variable;

(d) pre-processing a test data set in the same manner as was the training data set;

(e) testing each of the plurality of trained support vector machines using the pre-processed test data set;

(f) in response to receiving a test output from each of the plurality of trained support vector machines, post-processing each of the test outputs in the same manner as were the training outputs; and

(g) comparing each of the post-processed test outputs with each other to determine which if any of the post-processed test output is an optimal solution.

23. The method of claim 22, further comprising programming a computer with computer-executable instructions corresponding to steps (a) through (g) and storing the computer-executable instructions on a computer-readable medium.