Automatic algorithm generation

• US 20020164070A1
• Filed: 03/13/2002
• Published: 11/07/2002
• Est. Priority Date: 03/14/2001
• Status: Abandoned Application

• Current Assignee: Battelle Memorial Institute
• Original Assignee: Battelle Memorial Institute
• Inventors: Taylor, Kevin M., Stark, Gregory V., Kuhner, Mark B., Keller, Paul E., Whitney, Paul D., Schelhorn, Jean E., Sinnott, Loraine T., Rust, Steven W., Burgoon, David A.
• Application Number:
  US10/097,198
• Publication Number:
  US 20020164070A1
• Time in Patent Office: Days
• US Class Current: 382/159
• CPC Class Codes: G06F 18/211 Selection of the most significant subset of features
• CPC Class Codes: G06F 18/41 Interactive pattern learning with a human teacher

Abstract

Several approaches are provided for designing algorithms that allow for fast retrieval, classification, analysis or other processing of data, with minimal expert knowledge of the data being analyzed, and further, with minimal expert knowledge of the math and science involved in building classifications and performing other statistical data analysis. Further, methods of analyzing data are provided where the information being analyzed is not easily susceptible to quantitative description.

Claims

1. A pattern recognition construction system comprising:

a feature module arranged to interact with a plurality of data objects and generate feature vectors therefrom, said feature vectors defined by a candidate feature set comprising a plurality of candidate features;

a training module arranged to select and train at least one candidate classifier based upon said feature vectors generated by said feature module; and

,

an effectiveness module arranged to determine at least one performance measure for each candidate classifier and enable refinement thereof, wherein feedback is provided from said effectiveness module to at least one of said feature module to modify said feature vectors, and said training module to modify at least one candidate classifier.

2. The pattern recognition construction system according to claim 1, wherein said feature module is arranged to add new candidate features, remove select ones of said plurality of candidate features, and modify select ones of said plurality of candidate features in any combination thereof to modify said feature vectors in response to said feedback from said effectiveness module.

3. The pattern recognition construction system according to claim 1, wherein said feature module is arranged to extract additional feature vectors from said plurality of data objects to modify said feature vectors in response to said feedback from said effectiveness module.

4. The pattern recognition construction system according to claim 1, wherein said feature module is arranged to utilize new data objects and extract feature vectors therefrom in response to said feedback from said effectiveness module.

5. The pattern recognition construction system according to claim 1, wherein said training module is arranged to add a candidate classifier, remove a select one of said at least one candidate classifier, retrain said at least one candidate classifier based upon modified parameters of said at least one classifier, and retrain said at least one candidate classifier based upon modified feature vectors in any combination thereof in response to said feedback from said effectiveness module.

6. The pattern recognition construction system according to claim 1, wherein said feature module comprises a feature select module and a feature extract module, said feature select module configured to define said candidate feature set and said feature extract module configured to extract said feature vectors from said data objects based upon said candidate feature set.

7. The pattern recognition construction system according to claim 6, wherein said feature select module allows user guided selection of at least one candidate feature.

8. The pattern recognition construction system according to claim 1, wherein at least one candidate feature is derived from a feature library.

9. The pattern recognition construction system according to claim 1, wherein at least one candidate classifier is derived from a classifier library.

10. The pattern recognition construction system according to claim 1, wherein said feedback repeats iteratively until a predetermined stopping criterion is met, whereafter said feature set defines a final feature set, and a select one of said at least one candidate classifier defines a final classifier.

11. The pattern recognition construction system according to claim 10, wherein said effectiveness module is configured to report said at least one performance measure.

12. The pattern recognition construction system according to claim 10, further comprising:

a feature extract module arranged to extract a first feature vector from an unknown data object based upon said final feature set, and,

a classifier module arranged to classify said first feature vector using said final classifier.

13. The pattern recognition construction system according to claim 12, further comprising a feedback path arranged to route said unknown data object to a determine classification module, wherein said unknown data object is independently classified, then routed to said feature module to refine said final feature set and said final classifier.

14. The pattern recognition construction system according to claim 1, wherein said data set defines a training data set and a testing data set, and further comprising:

a feature extract module arranged to extract testing feature vectors from a testing data set using said final feature set; and

,

a classifier module arranged to classify said testing feature vectors using at least one candidate classifier, wherein said effectiveness module is arranged to determine said at least one performance measure for each candidate classifier used by said classifier module.

15. The pattern recognition construction system according to claim 14, further comprising:

a second feature extract module arranged to extract a first feature vector from an unknown data object based upon said final feature set, and,

a second classifier module arranged to classify said first feature vector using said final classifier.

16. The pattern recognition construction system according to claim 15, further comprising:

a feedback path arranged to route said unknown data object to a determine classification module, wherein said unknown data object is independently classified, then routed to said feature module to refine said final feature set and said final classifier.

18. The computer based pattern recognition construction system according to claim 17, wherein said feature vectors are modified in any combination of adding new candidate features, removing select ones of said plurality of candidate features, modifying select ones of said plurality of candidate features, and extracting additional feature vectors.

19. The computer based pattern recognition construction system according to claim 17, wherein said feature module is configured to selectively modify said feature vectors to add new candidate features, remove select ones of said plurality of candidate features, modify select ones of said plurality of candidate features, and extract additional feature vectors from said plurality of data objects in any combination thereof.

20. The computer based pattern recognition construction system according to claim 17, wherein said training module is configured to selectively modify said at least one candidate classifier to add a candidate classifier, remove a select one of said at least one candidate classifier, retrain said at least one candidate classifier based upon modified classifier parameters, and retrain said at least one candidate classifier based upon modified feature vectors in any combination thereof.

21. The computer based pattern recognition construction system according to claim 17, wherein said feedback repeats iteratively until a predetermined stopping criterion is met, said candidate feature set at the time said stopping criterion is met defining a final feature set, and a select one of said at least one candidate classifier defining a final classifier.

22. The computer based pattern recognition construction system according to claim 21, further comprising:

a feature extract module arranged to extract a first feature vector from an unknown data object based upon said final feature set, and,

a classifier module arranged to classify said first feature vector using said final classifier.

23. The computer based pattern recognition construction system according to claim 22, further comprising a feedback path arranged to route said unknown data object to a determine classification module, wherein said unknown data object is independently classified, then routed to said feature process to refine said final feature set and said final classifier.

24. The computer based pattern recognition construction system according to claim 17, wherein said data set defines a training data set and a testing data set, and further comprising:

a feature extract module arranged to extract testing feature vectors from a testing data set using said final feature set; and

,

a classifier module arranged to classify said testing feature vectors using at least one candidate classifier, wherein said effectiveness module is arranged to determine said at least one performance measure for each candidate classifier used by said classifier module.

25. The computer based pattern recognition construction system according to claim 24, further comprising a second feature extract module arranged to extract a first feature vector from an unknown data object based upon said final feature set, and,

a second classifier module arranged to classify said first feature vector using said final classifier.

26. The computer based pattern recognition construction system according to claim 25, further comprising a feedback path arranged to route said unknown data object to a determine classification module, wherein said unknown data object is independently classified, then routed to said feature process to refine said final feature set and said final classifier.

28. The pattern recognition construction system according to claim 27, wherein said feature module is configured to selectively modify said feature vectors to add new candidate features, remove select ones of said plurality of candidate features, modify select ones of said plurality of candidate features, and extract additional feature vectors from said plurality of data objects in any combination thereof.

29. The pattern recognition construction system according to claim 27, wherein said training module is configured to selectively modify said at least one candidate classifier to add a candidate classifier, remove a select one of said at least one candidate classifier, retrain said at least one candidate classifier based upon modified parameters of said at least one classifier, and retrain said at least one candidate classifier based upon modified feature vectors in any combination thereof.

30. The pattern recognition construction system according to claim 27, wherein said feedback repeats iteratively until a predetermined stopping criterion is met, said candidate feature set at the time said stopping criterion is met defining a final feature set, and a select one of said at least one candidate classifier defining a final classifier.

31. The pattern recognition construction system according to claim 30, wherein said effectiveness process is configured to report said at least one performance measure.

17. A computer based pattern recognition construction system comprising:

a feature module having;

a feature selection module arranged to derive a candidate feature set having a plurality of candidate features; and

,

a feature extract module arranged to interact with a plurality of digitally stored data objects to extract feature vectors therefrom, wherein said feature vectors are derived from said candidate feature set;

a classifier training module having;

a classifier selection module arranged to select a candidate classifier set having at least one candidate classifier; and

,

a training module arranged to train said candidate classifier set based upon said feature vectors generated by said feature extract module;

a classifier effectiveness module arranged to evaluate said candidate classifier set and generate at least one performance measure;

a first feedback path from said classifier effectiveness module to said feature module; and

,

a second feedback path from said classifier effectiveness module to said classifier training module, wherein said at least one performance measure generated by said classifier effectiveness module determines whether feedback is required to said feature module via said first feedback path to modify said feature vectors, to said classifier training module via said second feedback path to modify said candidate classifier set, or to both.

27. A pattern recognition construction system comprising:

a feature module arranged to interact with a plurality of pre-classified training data objects and generate training feature vectors therefrom, said training feature vectors defined by a candidate feature set comprising a plurality of candidate features;

a training module arranged to select and train at least one candidate classifier based upon said training feature vectors generated by said feature module;

a feature extract module arranged to interact with a plurality of pre-classified testing data objects and generate testing feature vectors therefrom, said testing feature vectors defined by said candidate feature set;

a classifier module arranged to classify said testing feature vectors using said at least one candidate classifier, and,

an effectiveness module arranged to determine at least one performance measure for each candidate classifier trained by said training module, or used by said classifier module, said at least one performance measure arranged to enable refinement of said at least one classifier through iterative feedback from said effectiveness module to at least one of said feature module to modify said training feature vectors, and said training module to modify said at least one candidate classifier, until a predetermined stopping criterion is met.

32. A pattern recognition construction system comprising:

a feature module comprising;

a feature selection module arranged to generate a candidate feature set comprising a plurality of candidate features; and

,

a first feature extract module arranged to extract training feature vectors from a pre-classified training data set based upon said candidate feature set;

a training module comprising;

a classifier selection module arranged to select a classifier set comprising at least one candidate classifier defined by a classifier algorithm; and

,

a classifier training module arranged to train said at least one candidate classifier based upon said training feature vectors;

a second feature extract module arranged to extract testing feature vectors from a pre-classified testing data set based upon said candidate feature set;

a first classifier module arranged to classify said testing feature vectors using said at least one candidate classifier;

a classifier effectiveness module arranged to evaluate said candidate classifier set either trained by said classifier training module, or used by said first classifier module to classify said testing feature vectors, and generate at least one performance measure;

a first feedback path from said classifier effectiveness module to said feature module, wherein said feature module is arranged to add new candidate features, remove select ones of said plurality of candidate features, modify select ones of said plurality of candidate features, and extract additional feature vectors from said plurality of data objects in any combination to modify said feature vectors; and

,

a second feedback path from said classifier effectiveness module to said training is module, wherein said training module is arranged to add a candidate classifier, remove a select one of said at least one candidate classifier, retrain said at least one candidate classifier based upon modified parameters of said at least one classifier, and retrain said at least one candidate classifier based upon modified feature vectors in any combination to modify said at least one candidate classifier, wherein said feedback repeats iteratively until a predetermined stopping criterion is met, said candidate feature set at the time said stopping criterion is met defining a final feature set, and a select one of said at least one candidate classifier defining a final classifier.

33. The pattern recognition construction system according to claim 32, further comprising:

a third feature extract module arranged to extract a first feature vector from an unknown data object based upon said final feature set, and,

a classifier module arranged to classify said first feature vector using said final classifier.

34. The pattern recognition construction system according to claim 33, further comprising a feedback path arranged to route said unknown data object to a determine classification module, wherein said unknown data object is independently classified, then routed to said feature process to refine said final feature set and said final classifier.

36. The pattern recognition construction system according to claim 35, wherein said candidate feature set is automatically generated by said processor.

37. The pattern recognition construction system according to claim 35, wherein at least one candidate feature in said candidate feature set is user selected.

38. The pattern recognition construction system according to claim 35, wherein said processor comprises a general purpose computer.

39. The pattern recognition construction system according to claim 35, wherein said software is further executable to derive said performance measure for each classifier in said classifier set by:

performing a first bootstrap operation to identify the performance of each candidate classifier in said classifier set;

performing a second bootstrap to identify the performance of each candidate classifier in said classifier set;

examining the bias evident in the results of said second bootstrap;

applying a bias correction to the first bootstrap results; and

,

obtaining at least one of an estimate and a confidence interval of the performance of each candidate classifier based upon said bias correction to said bootstrap results.

40. The pattern recognition construction system according to claim 39, wherein said bias comprises the difference between the estimates of said first and second bootstraps.

41. The pattern recognition construction system according to claim 39, wherein said software is further executable to compare said candidate classifiers based upon the obtained one of said estimate and said confidence interval.

42. The pattern recognition construction system according to claim 41, wherein said software is further executable to compute estimates for each candidate classifier, and a lower confidence bound is determined as a measure of each candidate classifier performance.

43. The pattern recognition construction system according to claim 39, wherein said software is further executable to output data to enable a visual clustering based upon the obtained one of said estimate and said confidence interval.

44. The pattern recognition construction system according to claim 35, wherein said processor comprises a network of computers.

45. The pattern recognition construction system according to claim 35, wherein said software comprises a plurality of individually executable software modules.

46. The pattern recognition construction system according to claim 35, wherein said data objects comprise a training set of pre-classified data objects, and further comprising a testing set of pre-classified data objects, wherein said software is further executable for:

extracting testing feature vectors from said testing set based upon said candidate feature set; and

,

classifying said testing feature vectors using said candidate classifier set.

48. The pattern recognition construction system according to claim 47, wherein said processor further refines said classifier set by the execution of at least one program to:

extract additional feature vectors and train said at least one candidate classifier thereon;

modify said candidate feature set, wherein feature vectors are extracted from said digitally stored representations of data objects based upon the modified candidate feature set, and said at least one candidate classifier is retrained thereon;

modify said candidate feature set by either the addition of at least one new candidate classifier or the removal of at least one candidate classifier from said candidate classifier set, wherein said classifier set is retrained on said feature vectors; and

,

modify at least one parameter of at least one candidate classifier, wherein the candidate classifier is retrained.

49. The pattern recognition construction system according to claim 47, wherein said output device is adapted to output at least one candidate classifier in said classifier set and said candidate feature set after a predetermined stopping criterion is met.

50. The pattern recognition construction system according to claim 47, wherein said candidate feature set is automatically generated by said processor.

51. The pattern recognition construction system according to claim 47, wherein at least one candidate feature in said candidate feature set is user selected.

52. The pattern recognition construction system according to claim 47, wherein said processor is programmed to derive said performance measure for each classifier in said classifier set by executing code to:

perform a first bootstrap operation to identify the performance of each candidate classifier in said classifier set;

perform a second bootstrap to identify the performance of each candidate classifier in said classifier set;

examine the bias evident in the results of said second bootstrap;

apply a bias correction to the first bootstrap results; and

,

obtain at least one of an estimate and a confidence interval of the performance of each candidate classifier based upon said bias correction to said bootstrap results.

53. The pattern recognition construction system according to claim 52, wherein said bias comprises the difference between the estimates of said first and second bootstraps.

54. The pattern recognition construction system according to claim 52, wherein said processor is further programmed to compare said candidate classifiers based upon the obtained one of said estimate and said confidence interval.

55. The pattern recognition construction system according to claim 54, wherein said processor is programmed to compute estimates for each candidate classifier, and a lower confidence bound is determined as a measure of each candidate classifier performance.

56. The pattern recognition construction system according to claim 52, wherein said processor is programmed to output data to enable a visual clustering based upon the obtained one of said estimate and said confidence interval.

57. The pattern recognition construction system according to claim 47, wherein the refinement of said classifier set is further based upon a comparison of said performance measure to a predetermined benchmark.

58. The pattern recognition construction system according to claim 47, wherein said classifier set comprises at least two candidate classifiers, and the refinement of said classifier set is further based upon a comparison of at least two of said candidate classifiers.

59. The pattern recognition construction system according to claim 47, wherein said performance measure comprises the identification of complimentary, application specific features, and is derived by said processor without the input from a domain aware human user.

60. The pattern recognition construction system according to claim 47, wherein said processor outputs to said output device, information corresponding to select ones of said classifier set explored by said processor.

61. The pattern recognition construction system according to claim 60, wherein said output comprises at least one of said performance measure, an identification of which data objects are misclassified, commonalities in misclassified data objects, and an identification of which features influenced the development of said candidate classifiers.

62. The pattern recognition construction system according to claim 47, wherein said processor is programmed to reduce noise picked up by said candidate classifiers by an examination of the feature set of over-trained candidate classifier algorithms.

63. The pattern recognition construction system according to claim 47, wherein said processor is programmed to interact with a domain-aware human user to identify a correct classification for misclassified data objects.

64. The pattern recognition construction system according to claim 47, wherein said data objects comprise a training set of pre-classified data objects, and further comprising a testing set of pre-classified data objects, wherein said processor is programmed to:

extract testing feature vectors from said testing set based upon said candidate feature set; and

,

classify said testing feature vectors using said candidate classifier set.

35. A pattern recognition construction system comprising:

at least one processor;

a storage device;

an output device; and

,

software executable by said at least one processor for;

accessing in said storage device digitally stored representations of data objects;

generating a candidate feature set having a plurality of candidate features;

extracting feature vectors from said digitally stored representations of data objects based upon said candidate feature set;

selecting at least one candidate classifier defining a candidate classifier set;

training said at least one candidate classifier using said feature vectors; and

,

iteratively refining said at least one classifier until a predetermined stopping criterion is met, said at least one classifier refined by;

providing a performance measure for each of said at least one candidate classifier; and

, performing at least one of;

extracting additional feature vectors and training said at least one candidate classifier thereon;

modifying said candidate feature set, wherein feature vectors are extracted from said digitally stored representations of data objects based upon the modified candidate feature set, and said at least one candidate classifier is retrained thereon;

modifying said candidate feature set by either adding at least one new candidate classifier or removing at least one candidate classifier from said candidate classifier set, wherein said classifier set is retrained on said feature vectors; and

,

modifying at least one parameter of at least one candidate classifier, wherein said candidate classifier is retrained, wherein said output device is adapted to output at least one candidate classifier in said classifier set and said candidate feature set after said predetermined stopping criterion is met.

47. A pattern recognition construction system comprising:

a storage device;

an output device; and

,

a processor programmed to;

access from said storage device, digitally stored representations of data objects;

extract feature vectors from said digitally stored representations of data objects based upon a candidate feature set; and

,

train a classifier set comprising at least one candidate classifier using said feature vectors;

provide a performance measure for each of said at least one candidate classifier; and

,

refine said classifier set based upon said performance measure by at least one of a modification to said candidate feature set and a modification to said candidate feature set.

65. A pattern recognition construction system comprising:

a feature module arranged to interact with a plurality of data objects and generate feature vectors therefrom, said feature vectors defined by a candidate feature set comprising a plurality of candidate features;

a training module arranged to select and train at least one candidate classifier based upon said feature vectors generated by said feature module; and

,

an effectiveness module arranged to determine at least one performance measure for each candidate classifier and enable refinement thereof, wherein;

at least one of said feature module and said training module are arranged to accept feedback of said performance measure from said effectiveness module;

said feature module, where arranged to accept said feedback, is further arranged to modify said feature vectors in response to feedback of said performance measure to said feature module;

said training module, where arranged to accept said feedback, is further arranged to modify said candidate classifier in response to feedback of said performance measure to said training module.

66. A computer readable carrier including a computer program that causes a computer to automate the development of classifiers, the computer program configured to cause said computer to perform operations comprising:

accessing a data set comprising a plurality of data objects;

identifying a candidate feature set based upon at least one candidate feature;

using a feature extraction process to extract feature vectors from said data set based upon said candidate feature set;

using a training process to train at least one candidate classifier from said feature vectors;

using an effectiveness process to provide a performance measure of said at least one candidate classifier; and

,

iteratively refining said candidate classifier based upon said performance measure until a predetermined stopping criterion is met by performing for each iteration, at least one of;

extracting additional feature vectors from said data set based upon said candidate feature set, wherein said candidate classifier is trained by said training process using said additional feature vectors and a new performance measure of said candidate classifier is recomputed by said effectiveness process;

modifying said candidate feature set, wherein said feature extraction process extracts new feature vectors from said data objects based upon the modified version of said candidate feature set, said candidate classifier is retrained using said new feature vectors and a new performance measure of said candidate classifier is recomputed; and

,

modifying said candidate classifier, wherein the modified version of said candidate classifier is retrained using said feature vectors, and a new performance measure is recomputed.

67. A pattern recognition construction system comprising:

means for integrating into a feedback driven system that can iterate until a predetermined stopping criterion is met having;

means for extracting feature vectors from a training set of data objects based upon a candidate feature set;

means for training at least one candidate classifier based upon said feature set;

means for providing a performance measure of said at least one candidate classifier; and

,

means for refining said at least one candidate classifier by at least one of modifying said feature vectors and modifying said at least one classifier; and

,

means for outputting at least one candidate classifier.

68. A computer automated method for pattern recognition construction comprising:

identifying a candidate feature set based upon at least one candidate feature;

executing a feature extraction process computer code to extract feature vectors from a training set of digitally stored representations of data objects based upon said candidate feature set;

executing a training process computer code to train a candidate classifier set having at least one candidate classifier on said feature vectors;

executing an effectiveness process computer code to provide a performance measure of said at least one candidate classifier; and

,

iteratively developing said candidate classifier set based upon said performance measure until a predetermined stopping criterion is met by performing for each iteration, at least one of;

executing said feature extraction process computer code to extract additional feature vectors from said training set based upon said candidate feature set;

modifying said candidate feature set, wherein said feature extraction process computer code is executed to extract new feature vectors from said data objects based upon the modified version of said candidate feature set;

modifying said candidate classifier set;

retraining said candidate classifier set; and

,

providing a new performance measure of said at least one candidate classifier, wherein a final feature set is defined by the candidate feature set at the time said predetermined stopping criterion is met, and a final classifier is defined by a select one of said at least one candidate classifier when said predetermined stopping criterion is met.

69. A method for automating pattern recognition comprising:

accessing a data set comprising a plurality of data objects;

identifying a candidate feature set based upon at least one candidate feature;

using a feature extraction process to extract feature vectors from said data set based upon said candidate feature set;

using a training process to train a candidate classifier from said feature vectors;

using an effectiveness process to provide a performance measure of said at least one candidate classifier; and

,

iteratively refining said candidate classifier based upon said performance measure until a predetermined stopping criterion is met by performing for each iteration, at least one of;

extracting additional feature vectors from said data set based upon said candidate feature set, wherein said candidate classifier is trained by said training process using said additional feature vectors and a new performance measure of said candidate classifier is recomputed by said effectiveness process;

modifying said candidate feature set, wherein said feature extraction process extracts new feature vectors from said data objects based upon the modified version of said candidate feature set, said candidate classifier is retrained using said new feature vectors and a new performance measure of said candidate classifier is recomputed; and

,

modifying said candidate classifier, wherein the modified version of said candidate classifier is retrained using said feature vectors, and a new performance measure is recomputed.

70. A method of performing automated pattern recognition comprising:

integrating into a computer environment;

a feature selection module arranged select features to define a feature set;

a feature extraction module arranged to extract feature vectors from data objects based upon said feature set;

a classifier selection module arranged to select at least one classifier;

a classifier training module arranged to train said at least one classifier selected by said classifier selection module based upon feature vectors extracted from said feature extraction module; and

,

a classifier performance evaluation module arranged to report at least one performance measure for each classifier trained by said classifier training module;

providing a training data set comprising a plurality of digitally stored representations of pre-classified data objects;

using said feature selection module to define a candidate feature set;

using said feature extraction module to extract training feature vectors from said training data set based upon said candidate feature set;

using said classifier selection module to select at least one candidate classifier;

using said classifier training module to train said at least one candidate classifier using said training feature vectors extracted by said feature extraction module;

using said classifier performance evaluation module to report at least one performance measure for each candidate classifier; and

,

using said report of said at least one performance measure to direct change to at least one of said training feature vectors and said at least one candidate classifier.

71. The method of performing automated pattern recognition according to claim 70, wherein said computer environment further includes a classify module;

and further comprising;

providing a testing data set comprising a plurality of digitally stored representations of pre-classified data objects;

using said feature extraction module to extract testing feature vectors from said testing data set based upon said candidate classifier;

using said classify module to classify said testing feature vectors using said at least one candidate classifier;

using said classifier performance evaluation module to report at least one performance measure for each candidate classifier; and

,

using said report of said at least one performance measure to direct change to at least one of said training feature vectors, said testing feature vectors, and said at least one candidate classifier.

72. A method of refining a classifier comprising:

obtaining a data set;

sampling from said data set, a training set of data, and an evaluation set of data;

developing a plurality of candidate classifiers using said training data;

evaluating said plurality of candidate classifiers using said evaluation data;

performing a first bootstrap operation to determine the performance of each of said candidate classifiers;

performing a second bootstrap operation to determine the performance of each of said candidate classifiers;

examining a bias evident in the results of said second bootstrap;

applying a bias correction to the first bootstrap results based upon said bias in said second bootstrap;

obtaining at least one of an estimate and a confidence interval of the bias corrected performance of each of said plurality of candidate classifiers to derive at least one performance measure; and

,

using said at least one performance measure as feedback to improve at least one of said plurality of candidate classifiers.

73. The method of refining a classifier according to claim 72, wherein said bias comprises the difference between the estimates of said first and second bootstraps.

74. The method of refining a classifier according to claim 72, further comprising comparing said plurality of classifiers after obtaining at least one of said estimates and confidence intervals for each of said plurality of classifiers.

75. The method of refining a classifier according to claim 74, wherein estimates are computed for each classifier, and a lower confidence bound is determined for classifier performance.

76. The method of refining a classifier according to claim 72, comprising visually clustering said at least one of said estimate and confidence interval.