Reducing human overhead in text categorization

• US 20070183655A1
• Filed: 02/09/2006
• Published: 08/09/2007
• Est. Priority Date: 02/09/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Brill, Eric, Konig, Arnd
• Granted Patent: US 7,894,677 B2
• Time in Patent Office: Days
• US Class Current: 382/159
• CPC Class Codes: G06F 18/24323 Tree-organised classifiers

Abstract

A unique multi-stage classification system and method that facilitates reducing human resources or costs associated with text classification while still obtaining a desired level of accuracy is provided. The multi-stage classification system and method involve a pattern-based classifier and a machine learning classifier. The pattern-based classifier is trained on discriminative patterns as identified by humans rather than machines which allow a smaller training set to be employed. Given humans'"'"' superior abilities to reason over text, discriminative patterns can be more accurately and more readily identified by them. Unlabeled items can be initially processed by the pattern-based classifier and if no pattern match exists, then the unlabeled data can be processed by the machine learning classifier. By employing the classifiers in this manner, less human involvement is required in the classification process. Even more, classification accuracy is maintained and/or improved.

Claims

1. A multi-stage classification system that facilitates minimizing human effort in text classification while obtaining a desired level of accuracy comprising:

a pattern-based classifier that initially processes input to classify and assign the input a label; and

a learning-based classifier that processes the input for classification purposes when no label is assigned to it by the pattern-based classifier.

2. The system of claim 1, the pattern-based classifier is trained on a set of training items.

3. The system of claim 2, the training items comprise at least one of text documents, messages, and files that each labeled based on one or more text patterns.

4. The system of claim 2 further comprises a classifier construction component that builds the pattern-based classifier, the classifier construction component comprises a pattern identification component that extracts a plurality of patterns from the training items and determines a correlation between the patterns and corresponding labels.

5. The system of claim 4 further comprises a display component that presents at least a subset of extracted patterns to one or more users to facilitate determining which patterns are discriminative.

6. The system of claim 4, the pattern identification component identifies patterns in the training items in a scalable manner to facilitate conserving memory consumption.

7. The system of claim 1, the pattern-based classifier determines whether a subset of text taken from the input matches at least one pattern and assigns an appropriate label to the input based on the at least one pattern.

8. The system of claim 1, the learning-based classifier is a machine learning classifier that is trained on the same set of training items used to train the pattern-based classifier.

9. A classification method that facilitates minimizing human effort in text classification while obtaining a desired level of accuracy comprising:

receiving unlabeled input; and

classifying the unlabeled input using a multi-stage classifier, the multi-stage classifier comprising at least a pattern-based classifier.

10. The method of claim 9, classifying the unlabeled input using a multi-stage classifier comprises initially running the input through the pattern-based classifier.

11. The method of claim 9 further comprises training the pattern-based classifier which comprises providing a set of training data;

discovering patterns in the training data that correlate to labels assigned to the data;

presenting at least a subset of the patterns to a user; and

determining which patterns are discriminative.

12. The method of claim 11 further comprises storing any discriminative patterns and discarding any non-discriminative patterns.

13. The method of claim 11 further comprises selecting the subset of patterns for presentation to the user based at least in part on at least one of the following:

frequency and correlation threshold between the pattern and the label.

14. The method of claim 11, wherein discovering the patterns in the training data is scalable to minimize memory usage.

15. The method of claim 9, classifying the unlabeled input using the multi-stage classifier further comprises running the input through a machine learning classifier if no label is assigned by the pattern-based classifier.

16. The method of claim 15 further comprises training the machine learning classifier on training data used to train the pattern-based classifier.

17. A classification method that facilitates minimizing human effort in text classification while obtaining a desired level of accuracy comprising:

running one or more unlabeled text items through a pattern-based classifier to determine whether at least a portion of text in the item matches a discriminative pattern;

assigning a label to the one or more text items corresponding to the pattern matched when a pattern match is found; and

subsequently running the one or more unlabeled text items through a machine learning classifier when no pattern match is found.

18. The method of claim 17 further comprises training the pattern-based classifier wherein the training comprises:

providing a set of training documents;

identifying patterns in the documents;

selecting a subset of the patterns that satisfy at least one of a threshold frequency and a correlation threshold; and

determining which patterns from the subset of patterns are discriminative.

19. The method claim 17 further comprises training the machine learning classifier using a same set of training documents used by the pattern-based classifier.

20. The method of claim 17 further comprises storing patterns that have been labeled as discriminative.