Methods and devices relating to estimating classifier performance

• US 20030187584A1
• Filed: 03/28/2002
• Published: 10/02/2003
• Est. Priority Date: 03/28/2002
• Status: Active Grant

• Current Assignee: Exagen Diagnostics, Inc.
• Original Assignee: Exagen Diagnostics, Inc.
• Inventors: Harris, Cole Coryell
• Granted Patent: US 7,561,971 B2
• Time in Patent Office: Days
• US Class Current: 702/19
• CPC Class Codes: G06F 18/211 Selection of the most significant subset of features
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G16H 10/40 for data related to laboratory analysis, e.g. patient specimen analysis
• CPC Class Codes: G16H 50/70 for mining of medical data, e.g. analysing previous cases of other patients

Abstract

Methods and devices, including methods and devices for estimating classifier performance such as generalization performance, are disclosed. One method includes providing multiple samples. Each sample is characterized by one or more features. This method also includes associating a feature variability with at least one of the one or more features; and computing a first probability of misclassification by a first classifier using the feature variability. Devices, including integrated circuits (ICs) and field programmable gate arrays (FPGAs), that are configured for use in carrying out the present methods are also disclosed.

Claims

1. A method for estimating performance of a classifier comprising:

(a) providing multiple samples, each sample characterized by one or more features;

(b) associating a feature variability with at least one of the one or more features;

(c) using the classifier, computing one or more scores for one or more samples in a subset of samples from the multiple samples;

(d) associating a score variability with at least one of the one or more scores using the feature variability; and

(e) computing a probability of misclassification by the classifier using one or more of the one or more scores and the score variability.

2. The method of claim 1, where (e) comprises:

(e) computing and storing a probability of misclassification by the classifier using one or more of the one or more scores and the score variability;

and where the method further comprises;

(f) using a second classifier, computing one or more scores for one or more samples in a subset of samples from the multiple samples;

(g) using the feature variability, associating a second score variability with at least one of the one or more scores computed using the second classifier; and

(h) computing a second probability of misclassification by the second classifier using;

(i) one or more of the one or more scores computed using the second classifier and

(ii) the second score variability.

3. The method of claim 1, where the multiple samples are multiple tissue samples.

4. The method of claim 1, where the classifier is a linear discriminant function classifier.

5. The method of claim 1, where the classifier is a quadratic discriminant function classifier.

6. The method of claim 1, where the classifier is a neural network classifier.

7. The method of claim 1, where the variability in (b) is estimated using multiple measurements of one of the one or more features.

8. The method of claim 1, where the associating in (d) includes using a linear approximation of the classifier.

9. A program comprising machine-readable instructions for implementing the method of claim 1.

10. A device comprising a field programmable gate array (FPGA) configured for use in carrying out the method of claim 1.

11. A device comprising an integrated circuit configured for use in carrying out the method of claim 1.

12. A method comprising:

(a) providing multiple samples, each sample characterized by one or more features;

(b) associating a feature variability with at least one of the one or more features; and

(c) computing a first probability of misclassification by a first classifier using the feature variability.

13. The method of claim 12, further comprising:

(d) computing a second probability of misclassification by a second classifier using the feature variability.

14. The method of claim 12, where (c) comprises:

(c) computing multiple probabilities of misclassification by multiple classifiers using the feature variability; and

(d) averaging the multiple probabilities of misclassification.

15. The method of claim 12, where the multiple samples are multiple tissue samples.

16. The method of claim 12, where the first classifier is a linear discriminant function classifier.

17. The method of claim 12, where the first classifier is a quadratic discriminant function classifier.

18. The method of claim 12, where the first classifier is a neural network classifier.

19. The method of claim 12, where the variability in (b) is estimated using multiple measurements of one of the one or more features.

20. The method of claim 12, where (c) includes using a linear approximation of the first classifier prior to computing the first probability of misclassification.

21. A program comprising machine-readable instructions for implementing the method of claim 12.

22. A device comprising a field programmable gate array (FPGA) configured for use in carrying out the method of claim 12.

23. A device comprising an integrated circuit configured for use in carrying out the method of claim 12.

24. A method of comparing two or more classifiers comprising:

performing (a)-(d) of claim 14, where (d) comprises averaging the multiple probabilities of misclassification to estimate performance of a first final classifier;

performing (a)-(d) of claim 14, where (d) comprises averaging the probabilities of misclassification by the multiple classifiers to estimate performance of a second final classifier; and

comparing the estimated performance of the first final classifier with the estimated performance of the second final classifier.