Method for automatic deduction of rules for matching content to categories

• US 20040139059A1
• Filed: 12/31/2002
• Published: 07/15/2004
• Est. Priority Date: 12/31/2002
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Gosby, Desiree D. G., Conroy, William F.
• Granted Patent: US 7,047,236 B2
• Time in Patent Office: Days
• US Class Current: 707/3
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

Accordingly, the invention is a method for automatic deduction of rules for matching document content to a category within a strange taxonomy, which allows the document to be automatically classified into a proper category for storage in that strange taxonomy. The method includes the steps of spidering the taxonomy to determine its structure and contents, extracting keywords from documents within the strange taxonomy, formulating rules for determining the category from the extracted keywords, and applying the rules to classify a new document whose keywords have been extracted. The taxonomy is strange because the user has no knowledge of its internal structure and needs no such knowledge. The taxonomy may be flat or may be hierarchal, the later having rules formulated at each level for proceeding to the next level. Variations for creating new and refurbishing old document management systems are disclosed.

Claims

1. A method of classifying document content within a taxonomy, the taxonomy comprising a plurality of first categories in a computer document storage organizational scheme and a plurality of first documents, each first document tagged with at least one first category according to the taxonomy, the method comprising the steps of:

spidering the taxonomy to generate at least one pairing of each first document with each first category with which the each first document is tagged;

creating a rule generation document representing each of the at least one pairings;

parsing a second document according to the rule generation document; and

classifying the parsed second document into a particular first category.

2. The method of claim 1, wherein the step of spidering the plurality of first documents comprises spidering to retrieve at least one of metadata, a storage location, and a category tag.

3. The method of claim 1, wherein the taxonomy comprises a strange taxonomy and wherein the step of spidering the plurality of first documents tagged with at least one first category according to the taxonomy comprises the steps of:

spidering the strange taxonomy with a first spider, the first spider adapted to the strange taxonomy being spidered;

creating a third document using the first spider, the third document describing the strange taxonomy, the third document comprising a link to each of the first documents; and

spidering the strange taxonomy with a second spider by spidering the third document created by the first spider, the second spider operable to access each of the first documents through the links in the third document.

4. The method of claim 3, wherein the step of creating the third document comprises creating an XML document.

5. The method of claim 3, wherein the steps of spidering the strange taxonomy with the first spider and creating a third document comprise steps taken after the second document is classified into the taxonomy, the second document thereby becoming a first document within the plurality of first documents.

6. The method of claim 3, wherein the step of spidering the strange taxonomy with a second spider comprises the step of spidering the strange taxonomy with a second spider after the second document is presented for classification within the taxonomy.

7. The method of claim 3, further comprising making the third document available for use by document-searching software.

8. The method of claim 1, wherein the step of creating a rule generation document comprises the steps of:

receiving a plurality of first-document-category pairings produced by the spidering step;

extracting at least one of a keyword and a pattern of keywords from each of the first documents within the plurality of first documents;

associating each at least one of a keyword and a pattern of keywords in each of the first documents with the at least one first category of the first document from which the at least one of a keyword and a pattern of keywords was extracted; and

generating rules for mapping at least one of a keyword and a pattern of keywords to the first category.

9. The method of claim 8, wherein the step of associating each at least one of a keyword and a pattern of keywords in each of the first documents with the at least one first category of the first document from which the at least one of a keyword and a pattern of keywords was extracted further comprises parsing each first document.

10. The method of claim 8, wherein the step of associating each at least one of a keyword and a pattern of keywords in each of the first documents with the at least one first category of the first document from which the at least one of a keyword and a pattern of keywords was extracted further comprises reading keywords from the metadata of each first document.

11. The method of claim 1, wherein the rule generation document comprises rules for mapping from at least one of a keyword and a pattern of keywords to one or more first categories, the step of parsing a second document according to the rule generation document comprises the steps of:

parsing the second document to determine at least one of a keyword and a pattern of keywords;

looking up the at least one of a keyword and a pattern of keywords of the second document in the rule generation document to find at least one of the first categories associated with the at least one of a keyword and a pattern of keywords of the second document;

scoring the found at least one first category according to a predetermined criteria; and

determining from the scoring the at least one first category comprising the classification of the second document.

12. The method of claim 11, wherein the step of scoring according to a predetermined criteria comprises scoring by at least one of:

similarity to at least one pattern of keywords associated with a first category;

frequency of keywords in a first category;

commonality of keywords among documents in a first category;

absence of particular keywords among documents in a first category; and

uniqueness of keywords in a first category.

13. The method of claim 12, wherein the step of determining from the scoring at least one first category further comprises the steps of selecting one of:

a) the at least one first category having a score comprising an extrema among the alternatives;

b) at least one first category having a score in a predetermined relationship to a predetermined threshold score; and

c) at least one first category having a particular predetermined score.

14. The method of claim 13, wherein the step of selecting further comprises selecting the at least one first category having the first-in-time score meeting the selection criteria.

15. The method of claim 1, wherein the step of classifying the parsed second document into at least one first category comprises submitting the parsed second document to a classification engine.

16. The method of claim 1, wherein the step of classifying the parsed second document into at least one first category comprises at least one of the steps of adding data to the metadata of the second document identifying the at least one first category, tagging the second document according to the taxonomy, and storing the second document in a location associated with the at least one first category.

17. The method of claim 1, wherein the taxonomy comprises a plurality of strange taxonomies, and further wherein:

the step of creating a rule generation document comprises generating a single rule generation document for the plurality of strange taxonomies; and

the step of classifying the parsed second document into at least one first category comprises the steps of;

classifying the parsed second document into one strange taxonomy within the plurality of strange taxonomies; and

classifying the parsed second document into one category within the plurality of categories within the strange taxonomy;

the method operable to select one strange taxonomy among the plurality of strange taxonomies within which to classify the second document.

18. The method of claim 1, wherein the taxonomy comprises a hierarchy of strange taxonomies, and further wherein:

the step of creating a rule generation document comprises at least one of;

generating at least one rule within the rule generation document for each strange taxonomy within the hierarchy of strange taxonomies; and

creating a rule generation document for each level of the hierarchy of strange taxonomies; and

the step of classifying the parsed second document into at least one first category comprises the steps of;

classifying the parsed second document into at least one strange taxonomy within the hierarchy of strange taxonomies; and

classifying the parsed second document into at least one first category within the at least one strange taxonomy within the hierarchy of strange taxonomies.

19. The method of claim 1, wherein the rule generation document comprises rules for mapping from at least one of a keyword and a pattern of keywords to one or more first categories, and wherein the step of parsing the second document according to the rule generation document comprises the steps of:

finding no keywords in the parsed second document similar to keywords in the rule generation document;

creating a new category within the taxonomy; and

classifying the second document in the new category.

20. The method of claim 1, wherein the step of classifying the parsed second document into a first category further comprises tagging the parsed second document.

23. The method of claim 20, wherein the step of providing a new document to a rule-deducing document classification and storage computer program comprises the execution of a “

save”

command referenced to the new document.

21. A method for categorizing the content of a new document within a strange taxonomy, the strange taxonomy comprising a plurality of first categories and a plurality of first documents within at least one of the first categories, wherein a root node for the strange taxonomy has been provided, the method comprising the steps of:

automatically spidering the strange taxonomy to identify each first category and each document among the plurality of first document classified within each respective first category;

automatically forming pairs for each of the first documents, each pair comprising one of the first documents and the category within which the one of the first documents is classified;

automatically extracting at least one of a keyword and a pattern of keywords from each of the first documents in each of the first categories;

automatically associating at least one of a keyword and a pattern of keywords extracted from each of the first documents within each of the first categories with the first category in which the first documents are classified;

automatically generating rules, each rule mapping at least one of a keyword and patterns of keywords to the first category in which the first documents containing the at least one of a keywords and a pattern of keywords are classified;

automatically parsing an unclassified document to determine new keywords therein; and

automatically classifying the unclassified document into at least one of a new category and a first category having documents containing at least one of keywords and patterns of keywords similar to the new keywords.

22. A method of storing a new document according to a strange taxonomy, the method comprising the step of providing the new document and a starting point in the strange taxonomy to a rule-deducing document classification and storage computer program, the program automatically spidering the strange taxonomy and tagged documents classified therein, automatically deducing rules for classification of the new document, automatically classifying the new document according to the rules deduced, and automatically storing the new document according to the classification of the new document.

25. The apparatus of claim 22, wherein the rule-deducing content classification mechanism comprises a mechanism operable to spider the taxonomy and the tagged documents classified therein to produce pairings of tagged documents and the categories with which the documents are tagged, to deduce rules for classifying content within the taxonomy from at least one of a keyword and a pattern of keywords from the document, and to classify a new document according to the taxonomy based on the deduced rules as applied to the content of the new document.

24. An apparatus comprising:

at least one processor;

a memory coupled to the at least one processor;

computer-readable data storage media coupled to the at least one processor;

a plurality of documents tagged according to a taxonomy, the documents residing in the computer readable data storage media, the documents comprising content, the content comprising at least one of a keyword and a pattern of keywords; and

a rule-deducing content classification mechanism residing in memory.

26. A program product comprising:

a rule-deducing classification mechanism residing in memory, the rule-deducing classification mechanism operable to automatically spider a strange taxonomy and the tagged documents classified therein, to deduce rules for classifying documents within the strange taxonomy, and to classify a new document according to the strange taxonomy; and

computer-readable signal bearing media bearing the rule-deducing classification mechanism.

27. A method of classifying document content within at least one taxonomy, the at least one taxonomy comprising a plurality of first categories in a computer document storage taxonomy and at least one first document tagged according to the taxonomy, the at least one first document classified within at least one first category within the plurality of first categories of the at least one taxonomy, the method comprising the steps of:

at least one of spidering and crawling the at least one taxonomy and the at least one first document tagged according to the at least one taxonomy to generate at least one pairing of at least one first document with the at least one first category in which the at least one first document is classified within the at least one taxonomy;

creating a rule generation document representing the at least one pairing of at least one first document with the at least one first category;

parsing a second document according to the rule generation document; and

classifying the parsed second document into at least one first category in the at least one taxonomy.

28. A method of finding documents in a computerized document management system, wherein the lost documents are lost to search engines because of incorrect filing, the method comprising the steps of:

retrieving each document; and

saving each document under a new taxonomical root using a rule-deducing classification mechanism.