Auto-classification system and method with dynamic user feedback

• US 10,235,453 B2
• Filed: 05/06/2016
• Issued: 03/19/2019
• Est. Priority Date: 10/31/2012
• Status: Active Grant

• Current Assignee: Open Text Corporation
• Original Assignee: Open Text Corporation
• Inventors: Simard, Charles-Olivier, Bowyer, Alex, Leclerc, Daniel, Molloy, Steve
• Primary Examiner(s): Beausoliel, Jr., Robert W
• Assistant Examiner(s): Hoang, Hau
• Application Number:
  US15/148,717
• Publication Number:
  US 20160253412A1
• Time in Patent Office: 1,047 Days
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 99/00 Subject matter not provided for in other groups of this subclass

Abstract

An auto-classification system and method provides dynamic user feedback in a guide that is presented to the user. The feedback presented in the guide enables the user to refine the classification model by adding or removing exemplars, creating, editing or deleting rules, or performing other such adjustments to the classification model. This technology enhances the overall transparency and defensibility of the auto-classification process.

Claims

1. A method for auto-classification of digital content, the method comprising:

receiving, over a network by a server machine embodying an auto-classification system through a user interface of the auto-classification system on a user device, an indication from a user to create or modify a classification model;

responsive to the indication, generating or accessing a container for the classification model, the container configured for holding classified documents for training the classification model, the generating or accessing performed by the server machine embodying the auto-classification system;

providing, by the server machine embodying the auto-classification system, a list of classification trees to the user device over the network for display on the user device, the list of classification trees residing on a content source communicatively connected to the auto-classification system;

receiving, by the server machine embodying the auto-classification system through the user interface on the user device, a selection of a specific classification or a classification tree from the list of classification trees displayed on the user device, the classification tree comprising individual classifications;

importing one or more content source documents that have been assigned with the specific classification or the classification tree from the content source into the container for the classification model, the importing performed by the server machine embodying the auto-classification system;

determining, by the server machine embodying the auto-classification system, a plurality of documents to be automatically classified, the plurality of documents residing on the content source or a data source communicatively connected to the auto-classification system;

automatically classifying the plurality of documents residing on the content source or the data source utilizing the one or more content source documents in the container for the classification model, the automatically classifying performed by the server machine embodying the auto-classification system and comprising;

comparing characteristics of the one or more content source documents in the container for the classification model and characteristics of a document of the plurality of documents residing on the content source or the data source; and

automatically assigning the document to the specific classification or the individual classifications of the classification tree of the one or more content source documents in the container for the classification model responsive to the characteristics of the document sufficiently matching the characteristics of the one or more content source documents in the container for the classification model;

activating a dynamic user feedback guide in the user interface;

determining at least one recommended action, suggestion, tip, or piece of information for refining the classification model; and

displaying the at least one recommended action, suggestion, tip, or piece of information in the dynamic user feedback guide of the user interface on the user device.

2. The method according to claim 1, wherein the content source comprises at least one of a content server, a content repository, a content management system, or a file system.

3. The method according to claim 1, further composing:

automatically selecting a random set of documents from content source documents being imported into the container for the classification model; and

converting the random set of documents to test documents for assessing accuracy of auto-classification by the classification model.

4. The method according to claim 1, further comprising:

responsive to a command to delete or remove an exemplar from a classification in the classification model, determining whether the exemplar is last exemplar for the classification in the classification model; and

responsive to the exemplar being the last exemplar for the classification in the classification model, displaying a warning message that the classification is to be deleted or removed from the classification model along with the exemplar.

5. The method according to claim 1, wherein the user interface of the auto-classification system comprises a review function through which classifications automatically assigned by the auto-classification system to the plurality of documents residing on the content source or the data source are reviewed, accepted, rejected, or corrected.

6. The method according to claim 1, further comprising:

automatically performing the activating.

7. The method according to claim 1, further comprising:

performing the activating responsive to user input on a user-selectable interface element of the user interface.

8. A computer program product for auto-classification of digital content, the computer program product comprising a non-transitory computer-readable medium comprising instructions translatable by a server machine running an auto-classification system for:

receiving, over a network through a user interface of the auto-classification system on the user device, an indication from a user to create or modify a classification model;

responsive to the indication, generating or accessing a container for the classification model, the container configured for holding classified documents for training the classification model;

providing a list of classification trees to the user device over the network for display on the user device, the list of classification trees residing on a content source communicatively connected to the auto-classification system;

receiving, through the user interface on the user device, a selection of a specific classification or a classification tree from the list of classification trees displayed on the user device, the classification tree comprising individual classifications;

importing one or more content source documents that have been assigned with the specific classification or the classification tree from the content source into the container for the classification model;

determining a plurality of documents to be automatically classified, the plurality of documents residing on the content source or a data source communicatively connected to the auto-classification system;

automatically classifying the plurality of documents residing on the content source or the data source utilizing the one or more content source documents in the container for the classification model, the automatically classifying comprising;

comparing characteristics of the one or more content source documents in the container for the classification model and characteristics of a document of the plurality of documents residing on the content source or the data source; and

automatically assigning the document to the specific classification or the individual classifications of the classification tree of the one or more content source documents in the container for the classification model responsive to the characteristics of the document sufficiently matching the characteristics of the one or more content source documents in the container for the classification model;

activating a dynamic user feedback guide in the user interface;

determining at least one recommended action suggestion, tip, or piece of information for refining the classification model; and

displaying the at least one recommended action, suggestion, tip, or piece of information in the dynamic user feedback guide of the user interface on the user device.

9. The computer program product of claim 8, wherein the content source comprises at least one of a content server, a content repository, a content management system, or a file system.

10. The computer program product of claim 8, wherein the instructions are further translatable by the server machine running the auto-classification system for:

automatically selecting a random set of documents from content source documents being imported into the container for the classification model; and

converting the random set of documents to test documents for assessing accuracy of auto-classification by the classification model.

11. The computer program product of claim 8, wherein the instructions are further translatable by the server machine running the auto-classification system for:

responsive to a command to delete or remove an exemplar from a classification in the classification model, determining whether the exemplar is last exemplar for the classification in the classification model; and

responsive to the exemplar being the last exemplar for the classification in the classification model, displaying a warning message that the classification is to be deleted or removed from the classification model along with the exemplar.

12. The computer program product of claim 8, wherein the user interface of the auto-classification system comprises a review function through which classifications automatically assigned by the auto-classification system to the plurality of documents residing on the content source or the data source are reviewed, accepted, rejected, or corrected.

13. The computer program product of claim 8, wherein the instructions are further translatable by the server machine running the auto-classification system for:

automatically performing the activating.

14. The computer program product of claim 8, wherein the instructions are further translatable by the server machine running, the auto-classification system for:

performing the activating responsive to user input on a user-selectable interface element of the user interface.

15. An auto-classification system for auto-classification of digital content, the auto-classification system comprising:

a processor;

a non-transitory computer-readable medium; and

stored instructions translatable by the processor for;

receiving, over a network through a user interface of the auto-classification system on the user device, an indication from a user to create or modify a classification model;

responsive to the indication, generating or accessing a container for the classification model;

providing a list of classification trees to the user device over the network for display on the user device, the list of classification trees residing on a content source communicatively connected to the auto-classification system;

receiving, through the user interface on the user device, a selection of a specific classification or a classification tree from the list of classification trees displayed on the user device, the classification tree comprising individual classifications;

importing one or more content source documents that have been assigned with the specific classification or the classification tree from the content source into the container for the classification model;

determining a plurality of documents to be automatically classified, the plurality of documents residing on the content source or a data source communicatively connected to the auto-classification system;

automatically classifying the plurality of documents residing on the content source or the data source utilizing the one or more content source documents in the container for the classification model, the automatically classifying comprising;

comparing characteristics of the one or more content source documents in the container for the classification model and characteristics of a document of the plurality of documents residing on the content source or the data source; and

automatically assigning the document to the specific classification or the individual classifications of the classification tree of the one or more content source documents in the container for the classification model responsive to the characteristics of the document sufficiently matching the characteristics of the one or more content source documents in the container for the classification model;

activating a dynamic user feedback guide in the user interface;

determining at least one recommended action, suggestion, tip, or piece of information for refining the classification model; and

displaying the at least one recommended action suggestion, tip, or piece of information in the dynamic user feedback guide of the user interface on the user device.

16. The auto-classification system of claim 15, wherein the content source comprises at least one of a content server, a content repository, a content management system, or a file system.

17. The auto-classification system of claim 15, wherein the instructions are further translatable by the processor for:

automatically selecting a random set of documents from content source documents being imported into the container for the classification model; and

converting the random set of documents to test documents for assessing accuracy of auto-classification by the classification model.

18. The auto-classification system of claim 15, wherein the instructions are further translatable by the processor for:

responsive to a command to delete or remove en exemplar from a classification in the classification model, determining whether the exemplar is last exemplar for the classification in the classification model; and

responsive to the exemplar being the last exemplar for the classification in the classification model, displaying a warning message that the classification is to be deleted or removed from the classification model along with the exemplar.

19. The auto-classification system of claim 15, wherein the user interface of the auto-classification system comprises a review function through which classifications automatically assigned by the auto-classification system to the plurality of documents residing on the content source or the data source are reviewed, accepted, rejected, or corrected.

20. The auto-classification system of claim 15, wherein the instructions are further translatable by the processor for:

automatically, performing the activating.

21. The auto-classification system of claim 15, wherein the instructions are further translatable by the processor for:

performing the activating responsive to user input on a user-selectable interface element of the user interface.