Method of, and system for, classification count adjustment

• US 20060206443A1
• Filed: 03/14/2005
• Published: 09/14/2006
• Est. Priority Date: 03/14/2005
• Status: Active Grant

• Current Assignee: Micro Focus LLC (Open Text Corporation)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Suermondt, Henri Jacques, Forman, George H.
• Granted Patent: US 8,260,730 B2
• Time in Patent Office: Days
• US Class Current: 706/20
• CPC Class Codes: G06F 18/21 Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques

Abstract

A classification count adjustment system for adjusting a count estimate of items in a dataset D classified into a class is disclosed. The system includes a count estimate produced by a classifier of the number of items in the dataset D classified into the class. The system further comprises one or more measures of behavior of the classifier, indicating the ability of the classifier to classify items into the class. The system further comprises a processor for computing an adjusted estimate based on the count estimate by the classifier and the one or more measures of behavior.

Claims

1. A classification count adjustment system for adjusting a count estimate of items in a dataset classified into a class, the system comprising:

a count estimate produced by a classifier of the number of items in the dataset classified into the class;

one or more measures of behavior of the classifier indicating the ability of the classifier to classify the items into the class; and

a processor for computing an adjusted estimate based on the count estimate by the classifier and the one or more measures of behavior.

2. A system as claimed in claim 1 wherein the one or more measures of behavior are any one of a precision, a recall, an accuracy, a true positive rate, a false positive rate, a true negative rate, a false negative rate, a confusion matrix, an information gain, a bi-normal separation, a lift, an F-measure or an area under a ROC curve.

3. A system as claimed in claim 1 wherein the processor computes the adjusted estimate by multiplying the count estimate by the quotient of a precision of the classifier and a recall of the classifier.

4. A system as claimed in claim 1 wherein the processor computes the adjusted estimate by subtracting the product of a false positive rate of the classifier and the number of items in the dataset from the count estimate, and dividing the result of the subtraction by the difference between a true positive rate of the classifier and the false positive rate of the classifier.

5. A system as claimed in claim 1, further comprising the classifier and wherein the classifier is applied to an item in the dataset in order to contribute to the count estimate.

6. A system as claimed in claim 1 further comprising a reporting module for reporting the adjusted estimate.

7. A system as claimed in claim 6 wherein the processor further computes a complement adjusted estimate of the number of items in a complement of the class and the reporting module reports the complement adjusted estimate.

8. A system as claimed in claim 1 further comprising a calibration dataset, a record containing an indication as to whether an item in the calibration dataset is classified into the class by the classifier, and an evaluation module for computing the one or more measures based on the calibration dataset and the record.

9. A system as claimed in claim 8 further comprising a recording module for computing the record by applying the classifier to the item.

10. A system as claimed in claim 1 further comprising an estimator module for estimating the proportion of a population belonging to the class based on the adjusted estimate.

11. A system as claimed in claim 10 wherein the dataset is substantially a subset of the population and the proportion estimate is further based on the the size of the population.

12. A system as claimed in claim 1 wherein a bound is computed on the adjusted estimate by the processor.

13. A system as claimed in claim 12 wherein the adjusted estimate has an upper bound determined by dividing the count estimate by a recall of the classifier and wherein the adjusted estimate has a lower bound determined by multiplying the count estimate by a precision of the classifier.

14. A system as claimed in claim 12 wherein the bound is computed by assuming a change to the one or more measures of behavior.

15. A system as claimed in claim 8 wherein the evaluation module further computes a setting for a parameter in the classifier based on the record.

16. A system as claimed in claim 1 further comprising:

a second count estimate provided by a second classifier of the number of items in the dataset classified into a second class;

one or more measures of behavior of the second classifier indicating the ability of the second classifier to classify items into the second class;

wherein the processor further computes a second adjusted estimate based on the second count estimate and the one or more measures of behavior of the second classifier.

17. A system as claimed in claim 8 wherein the evaluation module performs cross-validation.

18. A method of adjusting a count estimate of items in a dataset classified in a class comprising:

providing a count estimate produced by a classifier of the number of items in the dataset classified into the class;

using one or more measures of behavior of the classifier to indicate the ability of the classifier to classify the items into the class; and

computing an adjusted estimate based on the count estimate by the classifier and the one or more measures of behavior.

19. A method as claimed in claim 18 wherein the one or more measures of behavior are any one of a precision, a recall, an accuracy, a true positive rate, a false positive rate, a true negative rate, a false negative rate, a confusion matrix, an information gain, a bi-normal separation, a lift, an F-measure or an area under a ROC curve.

20. A method as claimed in claim 18 further comprising computing the adjusted estimate by multiplying the count estimate by the quotient of a precision of the classifier and a recall of the classifier.

21. A method as claimed in claim 18 further comprising computing the adjusted estimate by subtracting the product of a false positive rate of the classifier and the number of items in the dataset from the count estimate, and dividing the result of the subtraction by the difference between a true positive rate of the classifier and the false positive rate.

22. A method as claimed in claim 18 further comprising applying the classifier to an item in the dataset in order to contribute to the count estimate.

23. A method as claimed in claim 18 further comprising reporting the adjusted estimate using a reporting module.

24. A method as claimed in claim 23 further comprising computing a complement adjusted estimate of the number of items in the complement of the class and reporting the complement adjusted estimate using the reporting module.

25. A method as claimed in claim 18 further comprising providing a calibration dataset and a record containing an indication as to whether an item in the calibration dataset is classified into the class by the classifier, and computing the one or more measures of behavior based on the calibration dataset and the record.

26. A method as claimed in claim 25 further comprising computing the record by applying the classifier to the item.

27. A method as claimed in claim 25 further comprising performing cross-validation.

28. A method as claimed in claim 18 further comprising estimating the proportion of a population belonging to the class based on the adjusted estimate.

29. A method as claimed in claim 18 further comprising computing a bound on the adjusted estimate.

30. A method as claimed in claim 25 further comprising computing a setting for a parameter in the classifier based on the record.

31. A method as claimed in claim 18 further comprising;

providing a second count estimate by a second classifier of the number of items in the dataset classified into a second class;

using one or more measures of behavior of the second classifier indicating the ability of the second classifier to classify the items into the second class;

computing a second adjusted estimate based on the second count estimate and the one or more measures of behavior of the second classifier.

32. A computer program to control a computer to adjust a count estimate of items in a dataset classified in a class, the computer program being operable to control the computer to:

obtain a count estimate by a classifier of the number of items in the dataset classified into the class;

use one or more measures of behavior of the classifier to indicate the ability of the classifier to classify the items into the class; and

compute an adjusted estimate based on the count estimate by the classifier and the one or more measures of behavior.