Method and system for identifying relationships between text documents and structured variables pertaining to the text documents

• US 7,810,029 B2
• Filed: 11/06/2006
• Issued: 10/05/2010
• Est. Priority Date: 04/19/2001
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Holland, Karen Mae, Spangler, William Scott, Kreulen, Jeffrey Thomas
• Primary Examiner(s): LEWIS, CHERYL RENEA
• Application Number:
  US11/592,957
• Publication Number:
  US 20070061348A1
• Time in Patent Office: 1,429 Days
• Field of Search: 707 1- 3, 707/5, 707/7, 707/203, 707/765, 707/802, 707/999.005, 715/256, 715/259, 382/306
• US Class Current: 715/256
• CPC Class Codes: G06F 40/237 Lexical tools
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

A method and system for interesting relationships in text documents includes generating a dictionary of keywords in the text documents, forming categories of the text documents using the dictionary and an automated algorithm, counting occurrences of the structured variables, categories and structured variable/category combinations in the text documents, and calculating probabilities of occurrences of the structured variable/category combinations.

Claims

1. A computer implemented method for identifying relationships between text documents and structured variables pertaining to said text documents, comprising:

providing a dictionary of keywords in said text documents;

forming categories of said text documents using said dictionary and an automated algorithm;

counting occurrences of said structured variables, said categories and combinations of said structured variables and said categories for said said text documents;

calculating probabilities of occurrences of said combinations of structured variables and categories; and

identifying a relationship between a structured variable of said structured variables and text documents included in a category of said categories based on a probability of occurrence of a combination of said structured variable and said category.

2. The method according to claim 1, wherein said algorithm comprises a keyword occurrence algorithm and wherein each of said categories comprises a category of text documents in which a particular keyword occurs.

3. The method according to claim 2, wherein said forming said categories comprises:

generating a sparse matrix array containing a count of each of said keywords in each of said text documents.

4. The method according to claim 1, wherein said algorithm comprises a clustering algorithm and wherein each of said categories comprises a category of said text documents containing a particular cluster.

5. The method according to claim 4, wherein said clustering algorithm comprises a k means algorithm.

6. The method according to claim 4, wherein said forming said categories comprises inputting a predetermined number of categories.

7. The method according to claim 1, wherein said keywords comprise at least one of words and phrases which occur a predetermined number of times in said text documents.

8. The method of claim 1, wherein said calculating probabilities comprises using a Chi squared function.

9. The method of claim 1, wherein said providing said dictionary of keywords comprises:

first parsing text in said text document to identify and count occurrences of words;

storing a predetermined number of frequently occurring words;

second parsing text in said text documents to identify and count occurrences of phrases; and

storing a predetermined number of frequently occurring phrases.

10. The method according to claim 9, wherein said frequently occurring words and phrases are stored in a hash table.

11. The method according to claim 6, wherein said generating a sparse matrix array comprises:

third parsing text in said text documents to count a number of times that each of said keywords occurs in each of said text documents.

12. The method according to claim 1, wherein said relationships comprise said combinations of structured variables and categories having a lowest probability of occurrence.

13. The method according to claim 1, wherein said method calculates a probability that a given co-occurrence of a structured variable and a category would have occurred as a purely random event.

14. The method according to claim 1, wherein said structured variables comprise predetermined time intervals.

15. The method according to claim 14, wherein said predetermined time intervals comprise one of days, weeks, months and years.

16. The method according to claim 1, wherein said structured variables comprise structured data,

wherein said calculating said probabilities comprises using a result of said counting said occurrences of said combinations of structured variables and categories, and

wherein said identifying said relationship comprises:

determining whether said probability of occurrence of said combination of said structured variable and said category is below a predetermined value; and

if said probability is below said predetermined value, designating said relationship as an interesting relationship.

17. A system for identifying relationships between text documents and structured variables pertaining to said text documents, comprising:

an input device for inputting text documents;

a processor for;

forming categories of said text documents;

counting occurrences of said structured variables, categories and combinations of structured variables and categories;

calculating probabilities of occurrence of said combinations of structured variables and categories; and

identifying a relationship between a structured variable of said structured variables and text documents included in a category of said categories based on a probability of occurrence of a combination of said structured variable and said category; and

a display, for displaying said probabilities.

18. The system according to claim 17, further comprising:

a memory for storing occurrences of said structured variables, categories combinations of structured variables and categories, and probabilities of occurrences of said combinations of structured variables and categories.

19. The system according to claim 17, wherein said structured variables comprise predetermined time intervals.

20. The system according to claim 19, wherein said predetermined time intervals comprise one of days, weeks, months and years.

21. The system according to claim 17, wherein said probabilities of occurrence of said combinations of structured variables and categories comprise a probability that a given co-occurrence of a structured variable and a category would have occurred as a purely random event.

22. The system according to claim 17, wherein said relationships comprise statistically significant relationships.

23. A programmable storage medium tangibly embodying a program of machine-readable instructions executable by a digital processing apparatus to perform a method for identifying relationships between text documents and structured variables pertaining to said text documents, said method comprising:

providing a dictionary of keywords in said text documents;

forming categories of said text documents using said dictionary and an automated algorithm;

counting occurrences of said structured variables, said categories and said combinations of structured variables and categories in said text documents;

calculating probabilities of occurrences of said structured variable/category combinations; and

identifying a relationship between a structured variable of said structured variables and text documents included in a category of said categories based on a probability of occurrence of a combination of said structured variable and said category.