Document analysis and retrieval

• US 20040139058A1
• Filed: 12/30/2002
• Published: 07/15/2004
• Est. Priority Date: 12/30/2002
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Gosby, Desiree D. G., Ito, Keith I.
• Granted Patent: US 7,412,453 B2
• Time in Patent Office: Days
• US Class Current: 707/3
• CPC Class Codes: G06F 16/3347 using vector based model
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06F 16/38 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

A method and system for document analysis and retrieval. A document that includes text is received from a host. Document keys (i.e., keywords and keyphrases) associated with the text are generated. In first embodiments, a document taxonomy is provided. The taxonomy has categories and associated category keys (i.e., keywords and keyphrases). The category keys of each category are compared with the document keys to determine a distance between the document and each category as a measure of how close the document is to each category. A subset of the categories is returned to the host, wherein the subset of the categories reflects the determined distances. In second embodiments, a search string is created as a logical function of a subset of the document keys. The search string is submitted to a search engine. Links to related documents are received from the search engine and returned to the host.

Claims

1. A method for document analysis and retrieval, comprising the steps of:

receiving a document having text therein from a host of a first computing system;

generating document keys associated with said text from analysis of said text, each said document key selected from the group consisting of a keyword of said text and a keyphrase of said text;

providing a document taxonomy having categories, each category having category keys, each said category key selected from the group consisting of a keyword of said category and a keyphrase of said category;

comparing the category keys of each category with said document keys to make a determination of a distance between the document and each category as a measure of how close the document is to each category; and

returning a subset of said categories to said host, wherein said subset of said categories reflects said determination.

2. The method of claim 1, wherein said generating document keys comprises:

extracting said text from said document if said text appears in a non-text format within said document;

generating tokens of said text such that stop words do not appear in said tokens; and

stemming said tokens to generate said document keys from said tokens.

3. The method of claim 1, wherein the steps of the method are performed by the first computing system.

4. The method of claim 1, wherein the steps of the method are performed by a second computing system that is remote to the first computing system.

5. The method of claim 1, wherein said comparing comprises computing said distance for each category as a dot product of a vector of said document keys and a vector of said category keys for each category.

6. The method of claim 1, further comprising:

generating document weights, wherein each document weight is associated with a corresponding document key of said document keys, and wherein each document weight is a measure of the importance its corresponding document key in comparison with the other document keys of the document; and

providing category weights for each category, wherein each category weight is associated with a corresponding category key of said category keys, wherein each category weight is a measure of the importance its corresponding category key in comparison with the other category keys of the category, and wherein said comparing comprises computing said distance for each category as a dot product of a vector of said document weights and a vector of said category weights for each category.

7. The method of claim 6, wherein the document weights are a linear function of the number of appearances of the corresponding document keys in the document.

8. The method of claim 6, wherein the document weights are a nonlinear function of the number of appearances of the corresponding document keys in the document.

9. A method for document analysis and retrieval, comprising the steps of:

receiving a document having text therein from a host of a first computing system;

generating document keys associated with said text from analysis of said text, each said document key selected from the group consisting of a keyword of said text and a keyphrase of said text;

creating a search string, said search string comprising a logical function of a subset of said document keys;

submitting said search string to a search engine;

receiving links to related documents from said search engine, said links being based on said search string; and

returning said links to said host.

10. The method of claim 9, wherein said generating document keys comprises:

extracting said text from said document if said text appears in a non-text format within said document;

generating tokens of said text such that stop words do not appear in said tokens; and

stemming said tokens to generate said document keys from said tokens.

11. The method of claim 9, wherein the steps of the method are performed by a computing system that is local to the first computing system.

12. The method of claim 9, wherein the steps of the method are performed by a computing system that is remote to the first computing system.

13. The method of claim 9, wherein the search engine is local to the first computing system.

14. The method of claim 9, wherein the search engine is external to the first computing system.

15. The method of claim 9, wherein generating the search string comprises automatically generating the search string in accordance with a default algorithm.

16. The method of claim 9, further comprising generating document weights, wherein each document weight is associated with a corresponding document key of said document keys, wherein each document weight is a measure of the importance its corresponding document key in comparison with the other document keys of the document, and wherein creating the search string comprises creating the search string in dependence on said document weights.

17. The method of claim 16, wherein the document weights are a linear function of the number of appearances of the corresponding document keys in the document.

18. The method of claim 16, wherein the document weights are a nonlinear function of the number of appearances of the corresponding document keys in the document.

19. The method of claim 9, further comprising:

selecting a related document of said related documents; and

applying to said related document in substitution for said document;

said generating, said creating, said submitting, said receiving links, and said returning steps.

20. A computer program product, comprising a computer usable medium having a computer readable program code embodied therein for document analysis and retrieval, wherein the computer readable program code comprises an algorithm adapted to:

receive a document having text therein from a host of a first computing system;

generate document keys associated with said text from analysis of said text, each said document key selected from the group consisting of a keyword of said text and a keyphrase of said text;

provide a document taxonomy having categories, each category having category keys, each said category key selected from the group consisting of a keyword of said category and a keyphrase of said category;

compare the category keys of each category with said document keys to make a determination of a distance between the document and each category as a measure of how close the document is to each category; and

return a subset of said categories to said host, wherein said subset of said categories reflects said determination.

21. A computer program product, comprising a computer usable medium having a computer readable program code embodied therein for document analysis and retrieval, wherein the computer readable program code comprises an algorithm adapted to:

receive a document having text therein from a host of a first computing system;

generate document keys associated with said text from analysis of said text, each said document key selected from the group consisting of a keyword of said text and a keyphrase of said text;

create a search string, said search string comprising a logical function of a subset of said document keys;

submit said search string to a search engine;

receive links to related documents from said search engine, said links being based on said search string; and

return said links to said host.