Method of, and system for, classification count adjustment

• US 8,260,730 B2
• Filed: 03/14/2005
• Issued: 09/04/2012
• Est. Priority Date: 03/14/2005
• Status: Active Grant

• Current Assignee: Micro Focus LLC (Open Text Corporation)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Forman, George H., Suermondt, Henri Jacques
• Primary Examiner(s): Starks, Wilbert L
• Application Number:
  US11/080,098
• Publication Number:
  US 20060206443A1
• Time in Patent Office: 2,731 Days
• Field of Search: 706/20, 706/25, 706/45, 435/6
• US Class Current: 706/20
• CPC Class Codes: G06F 18/21 Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques

Abstract

A classification count adjustment system for adjusting a count estimate of items in a dataset D classified into a class is disclosed. The system includes a count estimate produced by a classifier of the number of items in the dataset D classified into the class. The system further comprises one or more measures of behavior of the classifier, indicating the ability of the classifier to classify items into the class. The system further comprises a processor for computing an adjusted estimate based on the count estimate by the classifier and the one or more measures of behavior.

Claims

1. A classification count adjustment system comprising:

a computer-readable storage medium to store;

a count estimate produced by a classifier of a number of items in a dataset that have been classified into a class;

one or more measures of behavior of the classifier indicating an ability of the classifier to classify the items into the class;

a processor for computing an adjusted estimate based on the count estimate by the classifier and the one or more measures of behavior; and

a reporting module to output the adjusted estimate for use in determining an amount of items that belong to the class.

2. A system as claimed in claim 1 wherein the one or more measures of behavior are any one of a precision, a recall, an accuracy, a true positive rate, a false positive rate, a true negative rate, a false negative rate, a confusion matrix, an information gain, a bi-normal separation, a lift, an F-measure or an area under a ROC curve.

3. A system as claimed in claim 1 wherein the processor is to compute the adjusted estimate by multiplying the count estimate by a measure of a precision of the classifier and a measure of a recall of the classifier.

4. A system as claimed in claim 1 wherein the processor is to compute the adjusted estimate by subtracting a product of a false positive rate of the classifier and the number of items in the dataset from the count estimate, and dividing a result of the subtraction by the difference between a true positive rate of the classifier and the false positive rate of the classifier.

5. A system as claimed in claim 1, further comprising a computer having the processor, wherein the classifier is executable in the computer, and wherein the classifier is to be applied to an item in the dataset to produce the count estimate.

6. A system as claimed in claim 1 wherein the processor is to further compute a complement adjusted estimate of the number of items in a complement of the class, and the reporting module is to further report the complement adjusted estimate.

7. A system as claimed in claim 1 further comprising a calibration dataset, a record containing an indication as to whether a particular item in the calibration dataset is classified into the class by the classifier, and an evaluation module for computing the one or more measures based on the calibration dataset and the record.

8. A system as claimed in claim 7 further comprising a recording module for computing the record by applying the classifier to the particular item.

9. A system as claimed in claim 7 wherein the evaluation module is to further compute a setting for a parameter in the classifier based on the record.

10. A system as claimed in claim 7 wherein the evaluation module is to perform cross-validation.

11. A system as claimed in claim 1 further comprising an estimator module for estimating a proportion of a population belonging to the class based on the adjusted estimate.

12. A system as claimed in claim 11 wherein the dataset is substantially a subset of the population and the proportion estimate is further based on the size of the population.

13. A system as claimed in claim 1 wherein the processor is to compute a bound on the adjusted estimate.

14. A system as claimed in claim 13 wherein the adjusted estimate has an upper bound determined by dividing the count estimate by a recall of the classifier, and wherein the adjusted estimate has a lower bound determined by multiplying the count estimate by a precision of the classifier.

15. A system as claimed in claim 13 wherein the processor is to compute the bound by assuming a change to the one or more measures of behavior.

16. A system as claimed in claim 1, wherein the computer-readable storage medium is to further store:

a second count estimate provided by a second classifier of a number of items in the dataset classified into a second class;

one or more measures of behavior of the second classifier indicating an ability of the second classifier to classify items into the second class;

wherein the processor is to further compute a second adjusted estimate based on the second count estimate and the one or more measures of behavior of the second classifier.

17. A system as claimed in claim 1, wherein the classifier comprises an automated classifier, and wherein the processor is to compute the adjusted estimate based on the count estimate by the automated classifier and the one or more measures of behavior of the automated classifier.

18. A system as claimed in claim 1, wherein the items of the dataset comprise one of (1) news articles and (2) representations of genes, the system further comprising the classifier being configured to be applied to the items to classify the items into the class, and wherein the output of the adjusted estimate is for use in determining an amount of the news articles or the representations of the genes in the class.

19. A method comprising:

providing a count estimate produced by a classifier of a number of items in a dataset classified into a class;

using one or more measures of behavior of the classifier to indicate an ability of the classifier to classify the items into the class;

computing, by a classification count adjustment computer system having a processor, an adjusted estimate based on the count estimate by the classifier and the one or more measures of behavior; and

outputting the adjusted estimate for use in determining an amount of items that belong to the class.

20. A method as claimed in claim 19 wherein the one or more measures of behavior are any one of a precision, a recall, an accuracy, a true positive rate, a false positive rate, a true negative rate, a false negative rate, a confusion matrix, an information gain, a bi-normal separation, a lift, an F-measure or an area under a ROC curve.

21. A method as claimed in claim 19 further comprising computing the adjusted estimate by multiplying the count estimate by a measure of a precision of the classifier and a measure of a recall of the classifier.

22. A method as claimed in claim 19 further comprising computing the adjusted estimate by subtracting a product of a false positive rate of the classifier and the number of items in the dataset from the count estimate, and dividing a result of the subtraction by the difference between a true positive rate of the classifier and the false positive rate.

23. A method as claimed in claim 19 further comprising applying the classifier to an item in the dataset to produce the count estimate.

24. A method as claimed in claim 19 wherein reporting the adjusted estimate is performed using a reporting module.

25. A method as claimed in claim 24 further comprising computing a complement adjusted estimate of a number of items in the complement of the class and reporting the complement adjusted estimate using the reporting module.

26. A method as claimed in claim 19 further comprising providing a calibration dataset and a record containing an indication as to whether a particular item in the calibration dataset is classified into the class by the classifier, and computing the one or more measures of behavior based on the calibration dataset and the record.

27. A method as claimed in claim 26 further comprising computing the record by applying the classifier to the particular item.

28. A method as claimed in claim 26 further comprising performing cross-validation.

29. A method as claimed in claim 26 further comprising computing a setting for a parameter in the classifier based on the record.

30. A method as claimed in claim 19 further comprising estimating a proportion of a population belonging to the class based on the adjusted estimate.

31. A method as claimed in claim 19 further comprising computing a bound on the adjusted estimate.

32. A method as claimed in claim 19 further comprising;

providing a second count estimate by a second classifier of a number of items in the dataset classified into a second class;

using one or more measures of behavior of the second classifier indicating an ability of the second classifier to classify the items into the second class;

computing a second adjusted estimate based on the second count estimate and the one or more measures of behavior of the second classifier.

33. A method as claimed in claim 19, wherein the classifier comprises an automated classifier, wherein computing the adjusted estimate comprises computing the adjusted estimate based on the count estimate by the automated classifier and the one or more measures of behavior of the automated classifier, and wherein outputting the adjusted estimate is for use in determining an amount of the news articles or the representations of the genes in the class.

34. The method as claimed in claim 19, wherein the items of the dataset comprise one of (1) news articles and (2) representations of genes, the method further comprising applying the classifier to the items to classify the items into the class.

35. A computer program embodied in a computer-readable storage medium, the computer program when executed to control a computer to:

obtain a count estimate by a classifier of a number of items in a dataset classified into a class;

use one or more measures of behavior of the classifier to indicate an ability of the classifier to classify the items into the class;

compute an adjusted estimate based on the count estimate by the classifier and the one or more measures of behavior; and

output the adjusted estimate for use in determining an amount of items that belong to the class.