Reconfigurable model for auto-classification system and method

• US 9,256,836 B2
• Filed: 10/31/2012
• Issued: 02/09/2016
• Est. Priority Date: 10/31/2012
• Status: Active Grant

• Current Assignee: Crowdstrike, Inc. (CrowdStrike Holdings Incorporated)
• Original Assignee: Open Text Corporation
• Inventors: Ludlow, Stephen, Pettigrew, Steve, Dowgailenko, Alex, Deligia, Agostino, Gigure, Isabelle
• Primary Examiner(s): Wong, Lut
• Application Number:
  US13/665,622
• Publication Number:
  US 20140122410A1
• Time in Patent Office: 1,196 Days
• Field of Search: 706/59, 706/12
• US Class Current: 1/1
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 99/00 Subject matter not provided for in other groups of this subclass

Abstract

A reconfigurable automatic document-classification system and method provides classification metrics to a user and enables the user to reconfigure the classification model. The user can refine the classification model by adding or removing exemplars, creating, editing or deleting rules, or performing other such adjustments to the classification model. This technology enhances the overall transparency and defensibility of the auto-classification process.

Claims

1. A computer-implemented method of automatically classifying digital content, the method comprising:

an auto-classification module, embodied on non-transitory computer memory, creating a content classification model for classifying content based on one or more selected documents that have been identified as exemplars, the exemplars representing a class, wherein creation of the content classification model comprises defining at least one content classification rule comprising a rule priority, a confidence level, or an applied classification, the at least one content classification rule having one or more parameters including at least one of a field name, an operator, or a value;

the auto-classification module displaying metrics for a set of documents that have been classified using the content classification model; and

the auto-classification module receiving user input to reconfigure the content classification model responsive to the displayed metrics;

wherein reconfiguring the content classification model includes changing one or more exemplars, one or more content classification rules, or a combination thereof.

2. The method as claimed in claim 1 wherein receiving the user input comprises receiving input that adds or removes exemplars.

3. The method as claimed in claim 1 wherein receiving the user input comprises receiving input that adds or removes one or more content classification rules.

4. The method as claimed in claim 1 wherein receiving the user input comprises receiving input that designates an auto-classified document as an exemplar.

5. The method as claimed in claim 1 wherein displaying metrics comprises displaying values for precision and recall, wherein the precision represents a frequency with which an assigned classification matches an expected classification and wherein the recall represents a frequency with which the expected classification is assigned across all processed documents.

6. The method as claimed in claim 1 wherein displaying metrics is in response to a review of a collection of classified documents by which an automatically assigned classification is accepted or rejected and for which an expected classification is specified for any misclassified documents.

7. The method as claimed in claim 1 further comprising automatically and randomly selecting a set of exemplars to use as test documents from a corpus of imported documents.

8. A non-transitory computer-readable medium comprising programmed instructions in code which, when loaded into a memory and executed by a processor of a computing device, causes the computing device to:

create a content classification model for classifying content based on one or more selected documents that have been identified as exemplars, the exemplars representing a class, wherein creation of the content classification model comprises defining at least one content classification rule comprising a rule priority, a confidence level, or an applied classification, the at least one content classification rule having one or more parameters including at least one of a field name, an operator, or a value;

display metrics for a set of documents that have been classified using the content classification model; and

receive user input to reconfigure the content classification model responsive to the displayed metrics;

wherein reconfiguring the content classification model includes changing one or more exemplars, one or more content classification rules, or a combination thereof.

9. The non-transitory computer-readable medium as claimed in claim 8 comprising code to receive input that adds or removes exemplars.

10. The non-transitory computer-readable medium as claimed in claim 8 comprising code to receive input that adds or removes one or more content classification rules.

11. The non-transitory computer-readable medium as claimed in claim 8 comprising code to receive input that designates an auto-classified document as an exemplar.

12. The non-transitory computer-readable medium as claimed in claim 8 comprising code to display values for precision and recall, wherein the precision represents a frequency with which an assigned classification matches an expected classification and wherein the recall represents a frequency with which the expected classification is assigned across all processed documents.

13. The non-transitory computer-readable medium as claimed in claim 8 wherein displaying metrics is in response to a review of a collection of classified documents by which an automatically assigned classification is accepted or rejected and for which an expected classification is specified for any misclassified documents.

14. The non-transitory computer-readable medium as claimed in claim 8 further comprising code for automatically and randomly selecting a set of exemplars to use as test documents from a corpus of imported documents.

15. An automatic document classification system comprising:

a processor coupled to a memory for creating a content classification model for classifying content based on one or more selected documents that have been identified as exemplars, the exemplars representing a class, wherein creation of the content classification model comprises defining at least one content classification rule comprising a rule priority, a confidence level, or an applied classification, the at least one content classification rule having one or more parameters including at least one of a field name, an operator, or a value; and

a display for displaying metrics for a set of documents that have been classified using the content classification model and for receiving user input to configure the content classification model responsive to the displayed metrics;

wherein configuring the content classification model includes changing one or more exemplars, one or more content classification rules, or a combination thereof.

16. The system as claimed in claim 15 wherein the user input adds or removes exemplars.

17. The system as claimed in claim 15 wherein the user input adds or removes one or more content classification rules.

18. The system as claimed in claim 15 wherein the user input designates an auto-classified document as an exemplar.

19. The system as claimed in claim 15 wherein the display presents values for precision and recall, wherein the precision represents a frequency with which an assigned classification matches an expected classification and wherein the recall represents a frequency with which the expected classification is assigned across all processed documents.

20. The system as claimed in claim 15 comprising a user input device to enable the user to perform a review of a collection of classified documents by which an automatically assigned classification is accepted or rejected and for which an expected classification is specified for any misclassified documents.

21. The system as claimed in claim 15 wherein the processor automatically and randomly selects a set of exemplars to use as test documents from a corpus of imported documents.

22. A computer-implemented method of automatically classifying digital content, the method comprising:

an auto-classification module generating a content classification model for classifying content based on one or more selected documents that have been identified as exemplars, the exemplars representing a class, wherein generation of the content classification model comprises defining at least one content classification rule comprising a rule priority, a confidence level, or an applied classification, the at least one content classification rule having one or more parameters including at least one of a field name, an operator, or a value;

the auto-classification module displaying metrics indicative of a precision of the content classification model for a plurality of other documents that have been classified using the content classification model; and

the auto-classification module receiving user input to reconfigure the content classification model by removing or adding exemplars or by removing or adding one or more classification rules responsive to the displayed metrics.

23. The method as claimed in claim 22 wherein displaying the metrics comprises presenting values for the precision and recall, wherein the precision represents a frequency with which an assigned classification matches an expected classification and wherein the recall represents a frequency with which the expected classification is assigned across all processed documents.

24. A computer-implemented method of automatically classifying digital content, the method comprising:

an auto-classification module receiving user input to identify one or more documents as exemplars, the exemplars representing a class of content;

the auto-classification module receiving user input to cause a content classification model to be generated based on the exemplars, wherein generation of the content classification model comprises defining at least one content classification rule comprising a rule priority, a confidence level, or an applied classification, the at least one content classification rule having one or more parameters including at least one of a field name, an operator, or a value;

the auto-classification module receiving user input to cause run a classification test using the content classification model;

the auto-classification module displaying metrics for the classification test; and

receiving user input to reconfigure the content classification model responsive to the displayed metrics;

wherein reconfiguring the content classification model includes changing one or more exemplars, one or more content classification rules, or a combination thereof.

25. The method as claimed in claim 24 wherein displaying the metrics comprises presenting values for precision and recall, wherein the precision represents a frequency with which an assigned classification matches an expected classification and wherein the recall represents a frequency with which the expected classification is assigned across all processed documents.