System and method for classification of documents

• US 7,283,998 B2
• Filed: 09/02/2003
• Issued: 10/16/2007
• Est. Priority Date: 09/03/2002
• Status: Active Grant

• Current Assignee: Fair Isaac Corporation
• Original Assignee: Infoglide Software Corp. (Fair Isaac Corporation)
• Inventors: Torossian, Vasken, Moon, Charles, Zrubek, Michael
• Primary Examiner(s): LeRoux; Etienne
• Application Number:
  US10/653,432
• Publication Number:
  US 20040078386A1
• Time in Patent Office: 1,505 Days
• Field of Search: 707/1, 707/2, 707/10, 707/200, 707/530, 707/5, 715/513
• US Class Current: 707/740
• CPC Class Codes: G06F 16/30 of unstructured textual data document management systems G06F16/93
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

The invention provides a classification engine for classifying documents that makes use of functions included in a similarity search engine. The classification engine executes a classify command from a client that makes use of similarity search results, and rules files, classes files, and a classification profile embedded in the classification command. When the classification receives a classify command from a client, it retrieves a classification profile and input documents to be classified, sends extracted values from the input documents based on anchor values to a XML transformation engine to obtain a search schema, requests a similarity search by a search manager to determine the similarity between input documents and anchor values, and classifies the input documents according to the rules files, classes files, and the classification profile. The client is then notified that the classify command has been completed and the classification results are stored in a database.

Claims

1. A method for classification of documents, comprising the steps of:

receiving a classify command from a client for initiating a classification of documents, the classify command identifying input documents to be classified, a classification profile, and an anchor document containing anchor values for identifying input document values to be used as a search criteria;

retrieving the classification profile and input documents;

extracting input values from each input document based on the anchor values;

structuring the input values according to a source schema identified in the classification profile;

performing similarity searches to determine similarity scores between one or more target database documents and the input values of each input document;

performing external analysis of the input documents to determine external analytic scores between one or more target database documents and the input values of each input document;

applying rules to the similarity scores and external analytic scores to determine rules results;

classifying the input documents into classes based on the classification profile, external analytic scores, similarity scores and rules results identified in the classification profile; and

sending the client a response at the completion of the classify command.

2. The method of claim 1, wherein the step of performing similarity searches comprises performing similarity searches for determining normalized similarity scores having values of between 0.00 and 1.00 for the one or more database documents for indicating a degree of similarity between the one or more database documents and each input document, whereby a normalized similarity score of 0.00 represents no similarity matching, a value of 1.00 represents exact similarity matching, and scores between 0.00 and 1.00 represent degrees of similarity matching.

3. The method of claim 1, wherein the step of retrieving the classification profile and input documents comprises retrieving the classification profile and input documents having repeating groups.

4. The method of claim 1, wherein the step of classifying further comprises scoring the input documents relative to other input documents in a same class according to predetermined scoring thresholds.

5. The method of claim 4, further comprising defining an upper and a lower threshold for scores associated with a class.

6. The method of claim 1, further comprising the step of storing the classified input documents as a classification results file in a results database.

7. The method of claim 6, wherein the step of storing the classified input documents comprises storing the classified input documents in an output target database identified in the classification profile.

8. The method of claim 1, wherein each of the classes identified in a class rules map in the-classification profile comprises a class identification attribute, a rule identification attribute, a low score attribute and a high score attribute.

9. The method of claim 8, further comprising a low score element and a high score element for defining lower and upper thresholds for similarity scores associated with the class.

10. The method of claim 1, wherein each of the rules identified in the classification profile comprises an identification attribute, a description element, and a condition element.

11. The method of claim 10, further comprising property elements for describing conditions for including a document in a parent class.

12. The method of claim 1, further comprising the step of mapping between defined classes and defined rules using class rule map files.

13. The method of claim 1, wherein the step of classifying the input documents is selected from the group consisting of classifying the input documents based on a threshold using a top score from results of more than one search schema, classifying the input documents based on a logical relationship and a threshold using a top score from more results of more than one search schema, classifying the input documents based on a number of search results for a single schema that have scores greater than a threshold, classifying the input documents based on a number of search results from multiple schemas having scores above a threshold, classifying the input documents based on external analytics for determining a document score, and classifying the input documents according to score rankings based on external analytics for determining a document score.

14. The method of claim 1, wherein the step of classifying the input documents further comprises classifying the multiple input documents based on profile, external analytic scores, and the similarity scores using classes and rules identified in the classification profile using a classify utility.

15. A computer-readable medium containing instructions for controlling a computer system to implement the method of claim 1.

16. A system for classification of documents, comprising:

a classification engine for receiving a classify command from a client for initiating a classification of documents, the classify command identifying input documents to be classified, a classification profile, and an anchor document containing anchor values for identifying input document values to be used as a search criteria;

the classification engine for retrieving the classification profile and input documents from a virtual document manager;

the classification engine for extracting input values from each input document based on the anchor values;

an XML transformation engine for structuring the input values according to a source schema identified in the classification profile;

a search manager for performing similarity searches to determine similarity scores between one or more target database documents and the input values of each input document;

external analytics for performing external analysis of the input documents to determine external analytic scores between one or more target database documents and the input values of each input document;

the classification engine for applying rules to the similarity scores and external analytic scores to determine rules results;

the classification engine for classifying the input documents into classes based on the classification profile, external analytic scores, similarity scores and rules results identified in the classification profile; and

means for sending the client a response at the completion of the classify command.

17. The system of claim 16, further comprising the search manager for performing similarity searches for determining normalized similarity scores having values of between 0.00 and 1.00 the one or more database documents for indicating a degree of similarity between the one or more database documents and each input document, whereby a normalized similarity score of 0.00 represents no similarity matching, a value of 1.00 represents exact similarity matching, and scores between 0.00 and 1.00 represent degrees of similarity matching.

18. The system of claim 16, further comprising the classification engine for retrieving the classification and input documents having repeating groups.

19. The system of claim 16, further comprising the classification engine for scoring the input documents relative to other input documents in a same class according to predetermined scoring thresholds.

20. The system of claim 16, further comprising the classification engine for storing the classified database documents as a classification results file in a results database.

21. The system of claim 20, wherein the classification engine stores the classified input documents in an output target database identified in the classification profile.

22. The system of claim 16, wherein each of the classes identified in a class rule maps in the-classification profile comprises a class identification attribute, a rule identification attribute, a low score attribute and a high score attribute.

23. The system of claim 22, further comprising a low score element and a high score element for defining lower and upper thresholds for similarity scores associated with the class.

24. The system of claim 16, wherein each of the rules identified in the classification profile comprises an identification attribute, a description element, and a condition element.

25. The system of claim 24, further comprising property elements for describing conditions for including a document in a parent class.

26. The system of claim 16, further comprising the classification engine for mapping between defined classes and defined rules using class rule map files.

27. The system of claim 16, wherein the classification engine for classifying the input documents is selected from the group consisting of means for classifying the input documents based on a threshold using a top score from results of more than one search schema, means for classifying the input documents based on a logical relationship and a threshold using a top score from more results of more than one search schema, means for classifying the input documents based on a number of search results for a single schema that have scores greater than a threshold, means for classifying the input documents based on a number of search results from multiple schemas having scores above a threshold, means for classifying the input documents based on external analytics for determining a document score, and means for classifying the input documents according to score rankings based on external analytics for determining a document score.

28. The system of claim 16, wherein the classification engine further comprises means for classifying the multiple input documents based on profile, external analytics, and the similarity scores using classes and rules identified in the classification profile using a classify utility.

29. A method for classification of documents, comprising:

receiving a classify command from a client, the classify command designating input document elements for names and search schema, anchor document structure, external analytics and values to be used as classification filters, and a classification profile;

retrieving the designated classification profile, the classification profile designating classes files for name, rank and score thresholds, rules files for nested conditions, properties, schema mapping, score threshold ranges and number of required documents, and class rules maps for class identification, class type, rule identification, description, property, score threshold ranges and document count;

retrieving the designated input documents;

identifying a schema mapping file for each input document;

determining a degree of similarity between each input document and one or more anchor documents;

determining analytic scores for each input document;

classifying the input documents according to the designated classes files, analytic scores and rules files;

creating and storing a classification results file in a database; and

notifying the client of completion of the classify command.

30. The method of claim 29, wherein the number of input documents classified based on rules is designated in the rules files.