Comparing distributions of cases over groups of categories

• US 7,668,789 B1
• Filed: 03/30/2006
• Issued: 02/23/2010
• Est. Priority Date: 03/30/2006
• Status: Expired due to Fees

• Current Assignee: Micro Focus LLC (Open Text Corporation)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Kirshenbaum, Evan R., Suermondt, Henri J., Lillibridge, Mark D., Forman, George H.
• Primary Examiner(s): SMITH, CREIGHTON H
• Application Number:
  US11/393,487
• Time in Patent Office: 1,426 Days
• Field of Search: 706/20, 706/21, 706/15, 706/12, 706/14, 706/45
• US Class Current: 706/20
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation

Abstract

A first distribution of cases over a first group of categories is received. A categorizer trained using a search-and-confirm technique classifies the cases into a second group of categories. A second distribution of the cases over the second group of categories is generated using results of the classifying. The first and second distributions are compared to identify differences between the first and second distributions.

Claims

1. A method comprising:

receiving a first distribution of cases over a first group of categories;

classifying, with a categorizer trained using a search-and-confirm technique, the cases into a second group of categories;

generating a second distribution of the cases over the second group of categories using results of the classifying; and

comparing the first and second distributions to identify differences between the first and second distributions.

2. The method of claim 1, wherein classifying the cases into the second group of categories comprises classifying the cases into the second group of categories that are identical to the first group of categories.

3. The method of claim 1, wherein comparing the first and second distributions comprises using one of a confusion matrices technique, a chi-squared technique, a cross-entropy technique, a mutual information technique, an information gain technique, and a mutual scatter analysis technique.

4. The method of claim 1, further comprising, in response to the identified differences, performing at least one of:

identifying a remedial action to address the identified differences;

providing a report of the differences;

identifying a category in the first group for which the first distribution over-estimated or under-estimated a number of cases; and

identifying one or more example cases assigned to a first group category in the first distribution, which one or more example cases are not assigned to a corresponding second group category in the second distribution.

5. The method of claim 4, wherein identifying the remedial action comprises at least one of:

modifying the first group of categories, training users with respect to the first group of categories, developing documentation to assist users in assigning cases to the first group of categories, changing incentives given to users, correcting a measure estimated for a category in the first group, changing a workflow, and updating a report.

6. The method of claim 1, wherein the first and second groups of categories are different, the method further comprising providing a report of one or more categories in one of the first and second groups that are not part of the other of the first and second groups of categories.

7. The method of claim 1, wherein at least one of the first and second groups of categories comprises a hierarchy of categories.

8. The method of claim 7, further comprising providing a report of sub-categories in the second group not included in the first group.

9. The method of claim 1, wherein receiving the first distribution comprises receiving the first distribution based on initial categorizations performed by one of a human and another categorizer.

10. The method of claim 1, further comprising assessing correctness of the first distribution based on the identified differences.

11. A method comprising:

receiving a first distribution of cases over a first group of issue paths based on initial assignments;

training a categorizer, wherein the training comprises submitting one or more queries to identify matching cases and receiving indications of whether the identified matching cases belong or do not belong to at least one category in a group of categories;

classifying the cases of the first distribution into the group of categories using the trained categorizer;

generating a second distribution of the cases over the group of categories using results of the classifying;

comparing the first and second distributions; and

effecting an action to improve assignments of cases into a second group of issue paths.

12. The method of claim 11, wherein effecting action to improve assignments of cases to the second group of issue paths is with respect to the second group of issue paths that is identical to the first group of issue paths.

13. The method of claim 11, wherein effecting action to improve assignments of cases to the second group of issue paths is with respect to the second group of issue paths that is different from the first group of issue paths, wherein the second group of issue paths comprises the group of categories.

14. The method of claim 13, wherein the group of categories is initially based on the first group of issue paths, the method further comprising modifying the group of categories, wherein the second group of issue paths comprises the group of categories that has been modified.

15. An article comprising at least one storage medium containing instructions that when executed cause a system to:

receive a first distribution of cases over a first group of categories;

classify, with a categorizer trained using a search-and-confirm technique, the cases into a second group of categories;

generate a second distribution of the cases over the second group of categories using results of the classifying; and

compare the first and second distributions to identify differences between the first and second distributions.

16. The article of claim 15, wherein the instructions when executed cause the system to further assess correctness of the first distribution based on the identified differences.

17. The article of claim 15, wherein comparing the first and second distributions comprises using one of a confusion matrices technique, a chi-squared technique, a cross-entropy technique, a mutual information technique, an information gain technique, and a mutual scatter analysis technique.

18. The article of claim 15, wherein the instructions when executed cause the system to further identify a remedial action to address the identified differences.

19. The article of claim 18, wherein identifying the remedial action comprises at least one of:

modifying the first group of categories, training users with respect to the first group of categories, developing documentation to assist users in assigning cases to the first group of categories, changing incentives given to users, correcting a measure estimated for a category in the first group, changing a workflow, and updating a report.

20. The article of claim 15, wherein receiving the first distribution comprises receiving the first distribution based on initial categorizations performed by one of a human and another categorizer.