Determining document classification probabilistically through classification rule analysis

• US 8,972,328 B2
• Filed: 06/19/2012
• Issued: 03/03/2015
• Est. Priority Date: 06/19/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Lamanna, Charles, Gandhi, Mauktik, Shur, Andrey, Kanade, Abhijat, Brown, Jeffrey, Pontes-Filho, Robledo
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): SITIRICHE, LUIS A
• Application Number:
  US13/527,267
• Publication Number:
  US 20130339288A1
• Time in Patent Office: 987 Days
• Field of Search: None
• US Class Current: 706/48
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06N 5/027 Frames
• CPC Class Codes: G06N 5/047 Pattern matching networks; Rete networks
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

A classification application identifies patterns and evidences within representative documents. The application constructs a classification rule according to an entity and an affinity determined from the patterns and evidences. The application processes the representative documents with the classification rule to evaluate whether the rules meet acceptance requirements. Subsequent to a successful evaluation, the application identifies confidence levels for patterns and evidences within other documents.

Claims

1. A method executed on a computing device for determining document classification probabilistically through classification rule analysis, the method comprising:

identifying patterns and evidences within content of representative documents;

constructing a classification rule based on an entity determined according to an analysis of the patterns and an affinity determined according to an analysis of the evidences;

processing the content with the classification rule to;

determine an entity count and an entity confidence level for the entity;

determine an affinity presence and an affinity confidence level for the affinity;

aggregate the entity count, the entity confidence level, the affinity presence, and

the affinity confidence level to returned results;

comparing the returned results to expected results to evaluate the classification rule against acceptance requirements; and

if the classification rule meets the acceptance requirements, identifying confidence levels for the patterns and the evidences;

else

editing the classification rule.

2. The method of claim 1, further comprising:

defining the entity through a collection of the patterns.

3. The method of claim 2, wherein each pattern is a collection of text markers with a probability of finding the entity including at least one from a set of:

a regular expression match and a keyword match.

4. The method of claim 1, wherein the entity count is a sum of each pattern'"'"'s count.

5. The method of claim 1, wherein the entity confidence level is a probability of finding at least one of the patterns in the content.

6. The method of claim 1, wherein the entity confidence level is a combination of each pattern'"'"'s confidence level.

7. The method of claim 1, further comprising:

defining the affinity through a collection of the evidences.

8. The method of claim 1, wherein the affinity confidence level is a probability of finding at least one of the evidences in the content.

9. The method of claim 1, further comprising:

determining the affinity confidence level by evaluating a local confidence level as a probability of at least one of the evidences within a proximity window of the affinity presence.

10. A computing device for determining document classification probabilistically through classification rule analysis, the computing device comprising:

a memory configured to store instructions; and

a processor coupled to the memory, the processor executing an application in conjunction with the instructions stored in the memory, wherein the application is configured to;

identify patterns and evidences within content of representative documents;

construct a classification rule based on an entity determined according to an analysis of the patterns and an affinity determined according to an analysis of the evidences;

process the content with the classification rule to;

determine an entity count and an entity confidence level for the entity;

determine an affinity presence and an affinity confidence level for the affinity;

aggregate the entity count, the entity confidence level, the affinity presence, and the affinity confidence level to returned results;

compare the returned results to expected results to evaluate the classification rule against acceptance requirements; and

if the classification rule meets the acceptance requirements, identify confidence levels for the patterns and the evidences;

else

edit the classification rule.

11. The computing device of claim 10, wherein the application is further configured to:

determine an entity environment value from a subset of the content in which at least one of the patterns is detected; and

determine an entity recall value from another subset of the content in which the entity is present.

12. The computing device of claim 11, wherein the application is further configured to:

accept the entity subsequent to determining the entity environment value equal or greater than a predetermined entity precision value and the entity recall value equal or greater than a predetermined entity recall value.

13. The computing device of claim 10, wherein the application is further configured to:

determine an affinity environment value from a subset of the content in which at least one of the evidences is detected; and

determine an affinity recall value from another subset of the content in which the affinity is present.

14. The computing device of claim 13, wherein the application is further configured to:

accept the affinity subsequent to determining the affinity environment value equal or greater than a predetermined affinity precision value and the affinity recall value equal or greater than a predetermined affinity recall value.

15. A method executed on a computing device to determine document classification probabilistically through classification rule analysis, the method comprising:

identifying patterns and evidences within content of representative documents;

constructing a classification rule based on an entity determined according to an analysis of the patterns and an affinity determined according to an analysis of the evidences;

processing the content with the classification rule to collect returned results by;

determining an entity count and an entity confidence level for the entity;

determining an affinity presence and an affinity confidence level for the affinity;

aggregating the entity count, the entity confidence level, the affinity presence, and the affinity confidence level to returned results;

comparing the returned results to expected results to evaluate the classification rule against acceptance requirements by;

accepting the entity subsequent to determining an entity environment value equal or greater than a predetermined entity precision value and an entity recall value equal or greater than a predetermined entity recall value;

accepting the affinity subsequent to determining an affinity environment value equal or greater than a predetermined affinity precision value and an affinity recall value equal or greater than a predetermined affinity recall value;

if the classification rule meets the acceptance requirements, identifying confidence levels for the patterns and the evidences;

else

editing the classification rule.

16. The method of claim 15, further comprising:

utilizing the entity confidence level and the affinity confidence level along with thresholds to recommend decisions.

17. The method of claim 15, further comprising:

including thresholds for the predetermined entity precision value, the predetermined affinity precision value, the predetermined affinity recall value, and the predetermined entity recall value within the acceptance requirements.

18. The method of claim 15, further comprising:

determining each pattern confidence level and each evidence confidence level as a number of true positives divided by a sum of a number of true positives and a number of false positives.