Computing a count of cases in a class

• US 7,502,767 B1
• Filed: 07/21/2006
• Issued: 03/10/2009
• Est. Priority Date: 07/21/2006
• Status: Active Grant

• Current Assignee: Micro Focus LLC (Open Text Corporation)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Suermondt, Henri Jacques, Forman, George Henry
• Primary Examiner(s): STARKS, WILBERT L
• Application Number:
  US11/490,781
• Time in Patent Office: 963 Days
• Field of Search: 706/20, 702/19, 715/811, 435/4
• US Class Current: 706/20
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

A count of cases in a target set classified in a class is computed using a classifier. In the method, one or more measures of behavior of the classifier are determined for a plurality of classification thresholds and at least one classification threshold for the classifier is selected based on the one or more measures of behavior. In addition, a score for a plurality of the cases in the target set using the classifier is computed and the count of the cases is computed based on the scores, the selected at least one classification threshold, and the one or more measures of behavior.

Claims

1. A computer-readable medium on which is embedded one or more computer programs, said one or more computer programs implementing a method for computing a count of cases in a target set classified in a class, said one or more computer programs comprising a set of instructions for:

providing a classifier capable of producing a score based on a case and the class;

determining one or more measures of behavior of the classifier for a plurality of classification thresholds, wherein the one or more measures of behavior indicate an ability of the classifier to classify cases into the class;

selecting at least one classification threshold for the classifier based on the one or more measures of behavior;

computing a score for a plurality of the cases in the target set using the classifier; and

computing a count of the cases in the target set classified into the class based on the scores for the plurality of cases in the target set, the selected at least one classification threshold, and the one or more measures of behavior.

2. The computer-readable medium according to claim 1, wherein the one or more measures of behavior include:

a false negative rate comprising the percentage of true positive rates that are incorrectly predicted to be negative;

a false positive rate, wherein the false positive rate comprises the percentage of true false rates that are incorrectly predicted to be positive; and

a true positive rate, wherein the true positive rate comprises the percentage of true positive rates that are correctly predicted to be positive.

3. The computer-readable medium according to claim 2, wherein selecting the classification threshold further comprises at least one of the following:

(a) selecting the at least one classification threshold to substantially maximize a difference between the true positive rate and the false positive rate for the classifier;

(b) selecting the at least one classification threshold so that the false negative rate is substantially equal to the false positive rate for the classifier;

(c) selecting the at least one classification threshold so that the false positive rate is substantially equal to a fixed target value;

(d) selecting the at least one classification threshold so that the true positive rate is substantially equal to a fixed target value;

(e) selecting the at least one classification threshold so that a difference between a raw count and the product of the false positive rate and the true positive rate is substantially maximized;

(f) selecting the at least one classification threshold so that the difference between the true positive rate and the false positive rate is greater than a fixed target value; and

(g) selecting the at least one classification threshold so that the difference between a raw count and the false positive rate multiplied by the number of cases in the target set is greater than a fixed target value.

4. The computer-readable medium according to claim 2, wherein computing the count further comprises calculating the adjusted count of positive cases (PC) in the class through the following equation:

5. The computer-readable medium according to claim 2, wherein computing the count further comprises calculating an adjusted percentage of positive cases (pp′

) through the following equation;

6. The computer-readable medium according to claim 1, wherein the cases comprise calls received in a help center and the target set comprises a problem topic of the help center.

7. The computer-readable medium according to claim 1, further comprising:

providing a utility; and

wherein selecting the at least one classification threshold further comprises selecting the at least one classification threshold based on the utility and the one or more measures of behavior.

8. The computer-readable medium according to claim 1, further comprising:

selecting a plurality of alternative classification thresholds for the classifier;

computing a plurality of intermediate counts of the cases in the target set classified into the class based on the scores for the plurality of cases in the target set, the plurality of alternative classification thresholds, and the one or more measures of behavior; and

computing an adjusted count of the number of cases in the target set classified into the class based on the computed count and the plurality of computed intermediate counts.

9. The computer-readable medium according to claim 8, further comprising:

discarding intermediate counts that fall below a predetermined minimum value;

discarding intermediate counts that exceed a predetermined maximum value;

calculating at least one of the median of the remaining intermediate counts and the average of the remaining intermediate counts; and

wherein the adjusted count comprises the calculated at least one of the median of the remaining intermediate counts and the average of the remaining intermediate counts.

10. The computer-readable medium according to claim 8, further comprising:

discarding intermediate counts where a difference between the true positive rate and the false positive rate is below a predetermined threshold;

calculating at least one of the median of the remaining intermediate counts and the average of the remaining intermediate counts; and

wherein the adjusted count comprises the calculated at least one of the median of the remaining intermediate counts and the average of the remaining intermediate counts.

11. The computer-readable medium according to claim 1, further comprising:

computing counts of the cases in the target set classified into each of a plurality of classes; and

computing a distribution of computed counts across the plurality of classes using the computed counts.

12. The computer-readable medium according to claim 11, wherein computing the distribution of computed counts across the plurality of classes further comprises normalizing the computed counts across the plurality of classes.

13. A method for analyzing cases received in a help center, said method comprising:

providing a classifier capable of producing a score based on a case and a category of interest;

determining one or more measures of behavior of the classifier for a plurality of classification thresholds, wherein the one or more measures of behavior indicate an ability of the classifier to classify cases into the category of interest;

selecting at least one classification threshold for the classifier based on the one or more measures of behavior;

computing a score for a plurality of cases received in the help center using the classifier;

computing a count of the cases received in the help center classified into the category of interest based on the scores for the plurality of cases received in the help center, the selected at least one classification threshold, and the one or more measures of behavior;

analyzing the count of the cases to at least one of determine a frequency at which the cases are classified into the category of interest and to quantify an emerging category of interest in the help center; and

storing the analyzed count of the cases.

14. The method according to claim 13, wherein the cases comprise calls received in the help center and the category of interest comprises a problem topic.

15. The method according to claim 13, wherein determining one or more measures of behavior further comprises determining the one or more measures of behavior from:

a false negative rate comprising the percentage of true positive rates that are incorrectly predicted to be negative;

a false positive rate, wherein the false positive rate comprises the percentage of true false rates that are incorrectly predicted to be positive; and

a true positive rate, wherein the true positive rate comprises the percentage of true positive rates that are correctly predicted to be positive.

16. The method according to claim 15, wherein selecting at least one classification threshold further comprises at least one of:

(a) selecting the at least one classification threshold to maximize a difference between the true positive rate and the false positive rate for the classifier;

(b) selecting the at least one classification threshold so that the false negative rate is equal to the false positive rate for the classifier;

(c) selecting the at least one classification threshold so that the false positive rate is equal to a fixed target value;

(d) selecting the at least one classification threshold so that the true positive rate is equal to a fixed target value;

(e) selecting the at least one classification threshold so that a difference between a raw count and the product of the false positive rate and the true positive rate is maximized;

(f) selecting the at least one classification threshold so that the difference between the true positive rate and the false positive rate is greater than zero; and

(g) selecting the at least one classification threshold so that the difference between a raw count and the false positive rate multiplied by the number of cases in the target set is greater than zero.

17. The method according to claim 15, further comprising:

providing a utility; and

wherein selecting the at least one classification threshold further comprises selecting the at least one classification threshold based on the utility and the one or more measures of behavior.

18. The method according to claim 13, further comprising:

selecting a plurality of alternative classification thresholds for the classifier;

computing a plurality of intermediate counts of the cases received in the help center classified into the category of interest based on the scores for the plurality of cases in the category of interest, the plurality of alternative classification thresholds, and the one or more measures of behavior; and

computing an adjusted count of the cases received in the help center classified into the category of interest based on the computed count and the plurality of computed intermediate counts.

19. The method according to claim 13, further comprising:

computing counts of the cases received in the help center classified into each of a plurality of categories of interest; and

computing a distribution of computed counts across the plurality of categories of interest using the computed counts.

20. A computer readable storage medium on which is embedded one or more computer programs, said one or more computer programs implementing a method for computing a count estimate of cases in a target set classified in a class, said one or more computer programs comprising a set of instructions for:

providing a classifier capable of producing a score based on a case and the class;

determining one or more measures of behavior of the classifier for a plurality of classification thresholds, wherein the one or more measures of behavior indicate the ability of the classifier to classify cases into the class;

selecting at least one classification threshold for the classifier based on the one or more measures of behavior;

computing a score for a plurality of the cases in the target set using the classifier;

computing a count of the cases in the target set classified into the class based on the scores for the plurality of cases in the target set, the selected at least one classification threshold, and the one or more measures of behavior; and

adjusting the computed count of cases.