Auto-classification system and method with dynamic user feedback

• US 9,348,899 B2
• Filed: 10/31/2012
• Issued: 05/24/2016
• Est. Priority Date: 10/31/2012
• Status: Active Grant

• Current Assignee: Crowdstrike, Inc. (CrowdStrike Holdings Incorporated)
• Original Assignee: Open Text Corporation
• Inventors: Simard, Charles-Olivier, Bowyer, Alex, Leclerc, Daniel, Molloy, Steve
• Primary Examiner(s): Beausoliel, Jr., Robert
• Assistant Examiner(s): HOANG, HAU HAI
• Application Number:
  US13/665,607
• Publication Number:
  US 20140122486A1
• Time in Patent Office: 1,301 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 99/00 Subject matter not provided for in other groups of this subclass

Abstract

An auto-classification system and method provides dynamic user feedback in a guide that is presented to the user. The feedback presented in the guide enables the user to refine the classification model by adding or removing exemplars, creating, editing or deleting rules, or performing other such adjustments to the classification model. This technology enhances the overall transparency and defensibility of the auto-classification process.

Claims

1. A computer-implemented method of automatically classifying digital content, the method comprising:

creating, by an auto-classification system embodied on non-transitory computer memory, a classification model for classifying digital content, wherein creating the classification model comprises;

prompting a user via a user interface to enter a name and description for the classification model, and

responsive to user selection of one or more documents via the user interface, adding or importing, from a document source into a container, the selected one or more documents as exemplars for a category of digital content within the classification model;

subsequent to a user selecting a plurality of documents, the auto-classification system classifying the plurality of documents using characteristics of the exemplars;

during the classifying, after the classifying is complete, or both during the classifying and after the classifying is complete,

the auto-classification system determining performance metrics representing accuracy of the classification model in classifying the plurality of documents, the performance metrics including precision, recall, match, noise, and silence metrics;

the auto-classification system determining one or more recommended actions based on the performance metrics;

the auto-classification system displaying the performance metrics including the precision, recall, match, noise, and silence metrics for a number of the plurality of documents that have been classified using the classification model;

the auto-classification system displaying a user feedback guide that presents the one or more recommended actions to improve the accuracy of the classification model, wherein the one or more recommended actions comprise adding one or more exemplars from the number of the plurality of documents that have been classified using the classification model or specifying how to meet one or more associated content classification rules; and

providing a user interface element associated with the user feedback guide that, in response to receipt of user input, causes a recommended action to be performed on the classification model.

2. The method as claimed in claim 1 wherein displaying the user feedback guide comprises presenting a recommended action to change a number of exemplars.

3. The method as claimed in claim 1 wherein displaying the user feedback guide comprises presenting a recommended action to review a greater number of classified documents to improve a confidence level.

4. The method as claimed in claim 1 wherein displaying the performance metrics comprises displaying values for precision and recall, wherein the precision represents a frequency with which an assigned classification matches an expected classification and wherein the recall represents a frequency with which the expected classification is assigned across all processed documents.

5. The method as claimed in claim 1 wherein the performance metrics are displayed in response to a review of a collection of classified documents, wherein the review comprises accepting or rejecting an automatically assigned classification and wherein the review further comprises specifying an expected classification for any misclassified documents.

6. The method as claimed in claim 1 wherein creating the classification model further comprises defining at least one classification rule, wherein the rule comprises a rule priority determining an order in which rule is applied, a confidence level to be applied to a document when the document satisfies a condition specified by the rule and an applied classification that is to be applied to the document.

7. The method as claimed in claim 1 comprising prioritizing the recommended actions based on importance and displaying the recommended actions in order of priority.

8. A non-transitory computer-readable medium comprising programmed instructions in code which, when loaded into a memory and executed by a processor of a computing device, causes the computing device to:

create a classification model for classifying digital content, wherein creating the classification model comprises;

prompting a user via a user interface to enter a name and description for the classification model, and

responsive to user selection of one or more documents via the user interface, adding or importing, from a document source into a container, the selected one or more documents as exemplars for a category of digital content within the classification model;

subsequent to a user selecting a plurality of documents, the auto-classification system classify the plurality of documents using characteristics of the exemplars;

during the classifying, after the classifying is complete, or both during the classifying and after the classifying is complete,

determine performance metrics representing accuracy of the classification model in classifying the plurality of documents, the performance metrics including precision, recall, match, noise, and silence metrics;

determine one or more recommended actions based on the performance metrics;

display the performance metrics including the precision, recall, match, noise, and silence metrics for a number of the plurality of documents that have been classified using the classification model;

display a user feedback guide that presents the one or more recommended actions to improve the accuracy of the classification model, wherein the one or more recommended actions comprise adding one or more exemplars from the number of the plurality of documents that have been classified using the classification model or specifying how to meet one or more associated content classification rules; and

provide a user interface element associated with the user feedback guide that, in response to receipt of user input, causes a recommended action to be performed on the classification model.

9. The non-transitory computer-readable medium as claimed in claim 8 comprising code to present a recommended action to change a number of exemplars.

10. The non-transitory computer-readable medium as claimed in claim 8 comprising code to present a recommended action to review a greater number of classified documents to improve a confidence level.

11. The non-transitory computer-readable medium as claimed in claim 8 comprising code to display values for precision and recall, wherein the precision represents a frequency with which an assigned classification matches an expected classification and wherein the recall represents a frequency with which the expected classification is assigned across all processed documents.

12. The non-transitory computer-readable medium as claimed in claim 8 wherein the performance metrics are displayed in response to a review of a collection of classified documents, wherein the review comprises accepting or rejecting an automatically assigned classification and wherein the review further comprises specifying an expected classification for any misclassified documents.

13. The non-transitory computer-readable medium as claimed in claim 8 comprising code to define at least one classification rule, wherein the rule comprises a rule priority determining an order in which rule is applied, a confidence level to be applied to a document when the document satisfies a condition specified by the rule and an applied classification that is to be applied to the document.

14. The non-transitory computer-readable medium as claimed in claim 8 comprising code for prioritizing the recommended actions based on importance and for displaying the recommended actions in order of priority.

15. An automatic document classification system comprising:

a processor coupled to a memory;

stored instructions executable by the processor for;

creating a classification model for classifying digital content, wherein creating the classification model comprises;

prompting a user via a user interface to enter a name and description for the classification model, and

responsive to user selection of one or more documents via the user interface, adding or importing, from a document source into a container, the selected one or more documents as exemplars for a category of digital content within the classification model;

subsequent to a user selecting a plurality of documents, classifying the plurality of documents using characteristics of the exemplars;

determining performance metrics representing accuracy of the classification model in classifying the plurality of documents, the performance metrics including precision, recall, match, noise, and silence metrics; and

determining one or more recommended actions based on the performance metrics;

a display for displaying, during the classifying, after the classifying is complete, or both during the classifying and after the classifying is complete, the performance metrics including the precision, recall, match, noise, and silence metrics for a number of the plurality of documents that have been classified using the classification model and for displaying a user feedback guide that presents the one or more recommended actions to improve the accuracy of the classification model, wherein the one or more recommended actions comprise adding one or more exemplars from the number of the plurality of documents that have classified using the classification model or specifying how to meet one or more associated content classification rules; and

a user interface element associated with the user feedback guide that, in response to receipt of user input, causes a recommended action to be performed on the classification model.

16. The system as claimed in claim 15 wherein the user feedback guide presents a recommended action to change a number of exemplars.

17. The system as claimed in claim 15 wherein the user feedback guide presents a recommended action to review a greater number of classified documents to improve a confidence level.

18. The system as claimed in claim 15 wherein the display presents values for precision and recall, wherein the precision represents a frequency with which an assigned classification matches an expected classification and wherein the recall represents a frequency with which the expected classification is assigned across all processed documents.

19. The system as claimed in claim 15 comprising a user input device to enable the user to perform a review of a collection of classified documents by accepting or rejecting an automatically assigned classification and by specifying an expected classification for any misclassified documents.

20. The system as claimed in claim 15 comprising a user input device to enable the user to define at least one classification rule, wherein the rule comprises a rule priority determining an order in which rule is applied, a confidence level to be applied to a document when the document satisfies a condition specified by the rule and an applied classification that is to be applied to the document.

21. The system as claimed in claim 15 wherein the recommended actions are prioritized and displayed in order of priority.

22. A computer-implemented method of automatically classifying content, the method comprising:

generating, by an auto-classification system embodied on non-transitory computer memory, a classification model for classifying digital content, wherein generating the classification model comprises;

prompting a user via a user interface to enter a name and description for the classification model, and

responsive to user selection of one or more documents via the user interface, adding or importing, from a document source into a container, the selected one or more documents as exemplars for a category of digital content within the classification model;

subsequent to a user selecting a plurality of documents, the auto-classification system classifying the plurality of documents using characteristics of the exemplars;

the auto-classification system determining performance metrics representing accuracy of the classification model in classifying the plurality of documents, the performance metrics including precision, recall, match, noise, and silence metrics;

the auto-classification system displaying the performance metrics including the precision, recall, match, noise, and silence metrics for a number of the plurality of documents that have been classified using the classification model;

the auto-classification system determining one or more recommended actions based on the performance metrics to improve the accuracy of the classification model;

displaying a user feedback guide that presents the one or more recommended actions determined by the auto-classification system for improving the classification model; and

providing a user interface element associated with the user feedback guide that, in response to receipt of user input, causes a recommended action to be performed on the classification model.

23. The method as claimed in claim 22 wherein displaying the performance metrics comprises displaying values for precision and recall, wherein the precision represents a frequency with which an assigned classification matches an expected classification and wherein the recall represents a frequency with which the expected classification is assigned across all processed documents.

24. The method as claimed in claim 23 wherein displaying the user feedback guide comprises presenting a recommended action to change a number of exemplars.

25. The method as claimed in claim 23 wherein displaying the user feedback guide comprises presenting a recommended action to review a greater number of classified documents to improve a confidence level.