Auto-classification system and method with dynamic user feedback
First Claim
Patent Images
1. A computer-implemented method of automatically classifying digital content, the method comprising:
- creating, by an auto-classification system embodied on non-transitory computer memory, a classification model for classifying digital content, wherein creating the classification model comprises;
prompting a user via a user interface to enter a name and description for the classification model, andresponsive to user selection of one or more documents via the user interface, adding or importing, from a document source into a container, the selected one or more documents as exemplars for a category of digital content within the classification model;
subsequent to a user selecting a plurality of documents, the auto-classification system classifying the plurality of documents using characteristics of the exemplars;
during the classifying, after the classifying is complete, or both during the classifying and after the classifying is complete,the auto-classification system determining performance metrics representing accuracy of the classification model in classifying the plurality of documents, the performance metrics including precision, recall, match, noise, and silence metrics;
the auto-classification system determining one or more recommended actions based on the performance metrics;
the auto-classification system displaying the performance metrics including the precision, recall, match, noise, and silence metrics for a number of the plurality of documents that have been classified using the classification model;
the auto-classification system displaying a user feedback guide that presents the one or more recommended actions to improve the accuracy of the classification model, wherein the one or more recommended actions comprise adding one or more exemplars from the number of the plurality of documents that have been classified using the classification model or specifying how to meet one or more associated content classification rules; and
providing a user interface element associated with the user feedback guide that, in response to receipt of user input, causes a recommended action to be performed on the classification model.
2 Assignments
0 Petitions
Accused Products
Abstract
An auto-classification system and method provides dynamic user feedback in a guide that is presented to the user. The feedback presented in the guide enables the user to refine the classification model by adding or removing exemplars, creating, editing or deleting rules, or performing other such adjustments to the classification model. This technology enhances the overall transparency and defensibility of the auto-classification process.
-
Citations
25 Claims
-
1. A computer-implemented method of automatically classifying digital content, the method comprising:
-
creating, by an auto-classification system embodied on non-transitory computer memory, a classification model for classifying digital content, wherein creating the classification model comprises; prompting a user via a user interface to enter a name and description for the classification model, and responsive to user selection of one or more documents via the user interface, adding or importing, from a document source into a container, the selected one or more documents as exemplars for a category of digital content within the classification model; subsequent to a user selecting a plurality of documents, the auto-classification system classifying the plurality of documents using characteristics of the exemplars; during the classifying, after the classifying is complete, or both during the classifying and after the classifying is complete, the auto-classification system determining performance metrics representing accuracy of the classification model in classifying the plurality of documents, the performance metrics including precision, recall, match, noise, and silence metrics; the auto-classification system determining one or more recommended actions based on the performance metrics; the auto-classification system displaying the performance metrics including the precision, recall, match, noise, and silence metrics for a number of the plurality of documents that have been classified using the classification model; the auto-classification system displaying a user feedback guide that presents the one or more recommended actions to improve the accuracy of the classification model, wherein the one or more recommended actions comprise adding one or more exemplars from the number of the plurality of documents that have been classified using the classification model or specifying how to meet one or more associated content classification rules; and providing a user interface element associated with the user feedback guide that, in response to receipt of user input, causes a recommended action to be performed on the classification model. - View Dependent Claims (2, 3, 4, 5, 6, 7)
-
-
8. A non-transitory computer-readable medium comprising programmed instructions in code which, when loaded into a memory and executed by a processor of a computing device, causes the computing device to:
-
create a classification model for classifying digital content, wherein creating the classification model comprises; prompting a user via a user interface to enter a name and description for the classification model, and responsive to user selection of one or more documents via the user interface, adding or importing, from a document source into a container, the selected one or more documents as exemplars for a category of digital content within the classification model; subsequent to a user selecting a plurality of documents, the auto-classification system classify the plurality of documents using characteristics of the exemplars; during the classifying, after the classifying is complete, or both during the classifying and after the classifying is complete, determine performance metrics representing accuracy of the classification model in classifying the plurality of documents, the performance metrics including precision, recall, match, noise, and silence metrics; determine one or more recommended actions based on the performance metrics; display the performance metrics including the precision, recall, match, noise, and silence metrics for a number of the plurality of documents that have been classified using the classification model; display a user feedback guide that presents the one or more recommended actions to improve the accuracy of the classification model, wherein the one or more recommended actions comprise adding one or more exemplars from the number of the plurality of documents that have been classified using the classification model or specifying how to meet one or more associated content classification rules; and provide a user interface element associated with the user feedback guide that, in response to receipt of user input, causes a recommended action to be performed on the classification model. - View Dependent Claims (9, 10, 11, 12, 13, 14)
-
-
15. An automatic document classification system comprising:
-
a processor coupled to a memory; stored instructions executable by the processor for; creating a classification model for classifying digital content, wherein creating the classification model comprises; prompting a user via a user interface to enter a name and description for the classification model, and responsive to user selection of one or more documents via the user interface, adding or importing, from a document source into a container, the selected one or more documents as exemplars for a category of digital content within the classification model; subsequent to a user selecting a plurality of documents, classifying the plurality of documents using characteristics of the exemplars; determining performance metrics representing accuracy of the classification model in classifying the plurality of documents, the performance metrics including precision, recall, match, noise, and silence metrics; and determining one or more recommended actions based on the performance metrics; a display for displaying, during the classifying, after the classifying is complete, or both during the classifying and after the classifying is complete, the performance metrics including the precision, recall, match, noise, and silence metrics for a number of the plurality of documents that have been classified using the classification model and for displaying a user feedback guide that presents the one or more recommended actions to improve the accuracy of the classification model, wherein the one or more recommended actions comprise adding one or more exemplars from the number of the plurality of documents that have classified using the classification model or specifying how to meet one or more associated content classification rules; and a user interface element associated with the user feedback guide that, in response to receipt of user input, causes a recommended action to be performed on the classification model. - View Dependent Claims (16, 17, 18, 19, 20, 21)
-
-
22. A computer-implemented method of automatically classifying content, the method comprising:
-
generating, by an auto-classification system embodied on non-transitory computer memory, a classification model for classifying digital content, wherein generating the classification model comprises; prompting a user via a user interface to enter a name and description for the classification model, and responsive to user selection of one or more documents via the user interface, adding or importing, from a document source into a container, the selected one or more documents as exemplars for a category of digital content within the classification model; subsequent to a user selecting a plurality of documents, the auto-classification system classifying the plurality of documents using characteristics of the exemplars; the auto-classification system determining performance metrics representing accuracy of the classification model in classifying the plurality of documents, the performance metrics including precision, recall, match, noise, and silence metrics; the auto-classification system displaying the performance metrics including the precision, recall, match, noise, and silence metrics for a number of the plurality of documents that have been classified using the classification model; the auto-classification system determining one or more recommended actions based on the performance metrics to improve the accuracy of the classification model; displaying a user feedback guide that presents the one or more recommended actions determined by the auto-classification system for improving the classification model; and providing a user interface element associated with the user feedback guide that, in response to receipt of user input, causes a recommended action to be performed on the classification model. - View Dependent Claims (23, 24, 25)
-
Specification