Determining document classification probabilistically through classification rule analysis

• US 9,495,639 B2
• Filed: 02/25/2015
• Issued: 11/15/2016
• Est. Priority Date: 06/19/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Lamanna, Charles, Gandhi, Mauktik, Shur, Andrey, Kanade, Abhijat, Brown, Jeffrey, Pontes-Filho, Robledo
• Primary Examiner(s): SITIRICHE, LUIS A
• Application Number:
  US14/631,430
• Publication Number:
  US 20150170036A1
• Time in Patent Office: 629 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06N 5/027 Frames
• CPC Class Codes: G06N 5/047 Pattern matching networks; Rete networks
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

A classification application identifies patterns and evidences within representative documents. The application constructs a classification rule according to an entity and an affinity determined from the patterns and evidences. The application processes the representative documents with the classification rule to evaluate whether the rules meet acceptance requirements. Subsequent to a successful evaluation, the application identifies confidence levels for patterns and evidences within other documents.

Claims

1. A system for determining document classification probabilistically through classification rule analysis, the system comprising:

at least one server comprising;

a memory configured to store instructions; and

a processor configured to execute an application in conjunction with the instructions stored in the memory, wherein the application is configured to;

identify patterns and evidences within content of representative documents;

construct a classification rule based on an entity determined according to an analysis of the patterns and an affinity determined according to an analysis of the evidences; and

process the content with the classification rule to;

determine an entity count and an entity confidence level for the entity;

determine an affinity presence and an affinity confidence level for the affinity, wherein the affinity confidence level is determined from a probability of at least one of the evidences being within a proximity window of a presence of the affinity, and the proximity window includes a window of the content used to scan the content for the affinity;

aggregate the entity count, the entity confidence level, the affinity presence, and the affinity confidence level to returned results;

compare the returned results to expected results to evaluate the classification rule against acceptance requirements; and

in response to a determination that the classification rule meets the acceptance requirements, identify confidence levels for the patterns and the evidences;

else

edit the classification rule.

2. The system of claim 1, wherein the application is further configured to:

in response to a determination that an entity rule meets acceptance requirements, process the entity rule against another document, wherein the acceptance requirements include precision requirements for established thresholds and recall rates for established thresholds.

3. The system of claim 1, wherein the application is further configured to:

in response to a determination that an affinity rule meets acceptance requirements, process the affinity rule against another document, wherein the acceptance requirements include precision requirements for established thresholds and recall rates for established thresholds.

4. The system of claim 1, wherein the application is further configured to:

in response to a determination that an entity rule fails to meet one or more of precision requirements for established thresholds and recall rates for established thresholds, one or more of;

edit the entity rule by adding corroborative evidence, edit the entity rule by removing pattern evidence, and edit the entity rule by adding pattern evidence.

5. The system of claim 4, wherein the application is further configured to:

reprocess the edited entity rule.

6. The system of claim 1, wherein the application is further configured to:

in response to a determination that an affinity rule fails to meet one or more precision requirements for established thresholds and recall rates for established thresholds, one or more of;

edit the affinity rule by adding corroborative evidence, edit the affinity rule by removing pattern evidence, and edit the affinity rule by adding pattern evidence.

7. The system of claim 6, wherein the application is further configured to:

reprocess the edited affinity rule.

8. A computing device for determining document classification probabilistically through classification rule analysis, the computing device comprising:

a memory configured to store instructions; and

a processor coupled to the memory, the processor executing an application in conjunction with the instructions stored in the memory, wherein the application is configured to;

identify patterns and evidences within content of documents;

determine a confidence level of an affinity from a probability of at least one of the evidences being within a proximity window of a presence of the affinity, wherein the proximity window includes a window of the content used to scan the content for the affinity;

construct a classification rule based on an entity determined according to an analysis of the patterns and the affinity determined according to an analysis of the evidences;

define the entity through the patterns, wherein each pattern is a collection of text markers with a probability of finding the entity including one or more of;

a regular expression match and a keyword match;

process the content with the classification rule to collect returned results; and

compare the returned results to expected results to evaluate the classification rule against acceptance requirements.

9. The computing device of claim 8, wherein the application is further configured to:

accept the entity subsequent to determining an entity environment value of the entity equal or greater than an entity precision value of the entity and an entity recall value of the entity equal or greater than the entity recall value.

10. The computing device of claim 8, wherein the application is further configured to:

accept the affinity subsequent to determining an affinity environment value of the affinity equal or greater than an affinity precision value of the affinity and an affinity recall value of the affinity equal or greater than the affinity recall value.

11. The computing device of claim 8, wherein the application is further configured to:

evaluate the classification rule to determine if the classification rule meets acceptance requirements, wherein the acceptance requirements include precision thresholds and recall thresholds.

12. The computing device of claim 11, wherein the application is further configured to:

in response to a positive evaluation of the acceptance requirements, identify confidence levels for the patterns and the evidences; and

in response to a negative determination of the acceptance requirements, edit the classification rule and reevaluate the classification rule against acceptance requirements.

13. A method executed on a computing device for determining document classification probabilistically through classification rule analysis, the method comprising:

identifying patterns and evidences within content of documents, wherein each one of the evidences includes an aggregate of keyword matches that are in proximity;

determining a confidence level of an affinity from a probability of at least one of the evidences being within a proximity window of a presence of the affinity, wherein the proximity window includes a window of the content used to scan the content for the affinity;

constructing a classification rule based on an entity determined according to an analysis of the patterns and the affinity determined according to an analysis of the evidences;

processing the content with the classification rule to collect returned results; and

comparing the returned results to expected results to evaluate the classification rule against acceptance requirements.

14. The method of claim 13, further comprising:

defining the entity through the patterns, wherein each pattern is a collection of text markers with a probability of finding the entity including one or more of;

a regular expression match and a keyword match.

15. The method of claim 13, further comprising:

determining a confidence level of the entity from a probability of matching at least one of the patterns to indicate a presence of the entity.

16. The method of claim 15, further comprising:

in response to a detection of the entity within the document having the confidence level as a moderate confidence level, warning a user that the document includes the entity.

17. The method of claim 15, further comprising:

in response to a detection of the entity within the document having the confidence level as a high confidence level, encrypting the entity.

18. The method of claim 13, further comprising:

determining a confidence level of the entity and a confidence level of the affinity as a number of true positives divided by a sum of a number of true positives and a number of false positives.

19. The method of claim 13, further comprising:

calibrating the classification rule with keywords of the content.