Method and system for analyzing unstructured text in data warehouse

• US 20050262039A1
• Filed: 05/20/2004
• Published: 11/24/2005
• Est. Priority Date: 05/20/2004
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Spangler, William Scott, Rhodes, James J., Kreulen, Jeffrey Thomas
• Application Number:
  US10/851,754
• Publication Number:
  US 20050262039A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/31 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/355 Class or cluster creation or modification

Abstract

A user initially analyzes a statistically significant sample of documents randomly drawn from a data warehouse to create a cached feature space and text classifier, which can then be used to establish a classification dimension in the data warehouse for in depth and detailed text analysis of the entire data set.

Claims

1. A computer-implemented method for analyzing information in a data warehouse, comprising:

selecting a sample of documents from the data warehouse;

generating at least one feature space of terms of interest in unstructured text fields of the documents using the sample;

generating at least one default classification using the feature space;

modifying the default classification to render a modified classification;

establishing at least one classifier using the modified classification; and

establishing a classification dimension in the data warehouse using the classifier.

2. The method of claim 1, further comprising adding documents not in the sample to the classification dimension, whereby the classification dimension is useful for searching for documents by classification.

3. The method of claim 1, wherein the classifier includes a machine-implementable set of rules that can be applied to any data element in the warehouse to generate a label.

4. The method of claim 1, wherein the sample is pseudo-randomly selected.

5. The method of claim 1, comprising displaying output using an on-line analytical tool (OLAP) tool.

6. The method of claim 1, wherein at least the act of generating a default classifier is undertaken using an e-classifier tool.

7. The method of claim 1, comprising:

identifying a subset of documents in the warehouse;

selecting features from the feature space that are relevant to the subset; and

comparing the subset with the sample using the features from the feature space that are relevant to the subset.

8. A service for analyzing information in a data warehouse of a customer, comprising:

receiving a sample of documents in the warehouse;

based on the sample, generating at least one initial classification;

using the initial classification to generate a classifier;

using the classifier to add documents not in the sample to a classification dimension; and

returning at least one of;

the classification dimension, and an analysis rendered by using the classification dimension, to the customer.

9. The service of claim 8, comprising allowing a user to modify the initial classification.

10. The service of claim 8, wherein the classification dimension is useful for searching for documents by classification.

11. The service of claim 8, wherein the classifier includes a machine-implementable set of rules that can be applied to any data element in the warehouse to generate a label.

12. The service of claim 8, wherein the sample is pseudo-randomly selected.

13. The service of claim 8, comprising displaying output using an on-line analytical tool (OLAP) tool.

14. The service of claim 8, wherein at least the act of generating an initial classifier is undertaken using an e-classifier tool.

15. The service of claim 8, comprising:

identifying a subset of documents in the warehouse;

selecting features from the feature space that are relevant to the subset; and

comparing the subset with the sample using the features from the feature space that are relevant to the subset.

16. A computer executing logic for analyzing unstructured text in documents in a data warehouse, the logic comprising:

establishing, based on only a sample of documents in the warehouse, a classification dimension listing all documents in the warehouse, the classification dimension being based on words in unstructured text fields in the documents.

17. The computer of claim 16, wherein the establishing act undertaken by the logic includes:

selecting a sample of documents from the data warehouse;

generating at least one feature space of terms of interest using the sample;

generating at least one default classification using the feature space;

modifying the default classification to render a modified classification;

establishing at least one classifier using the modified classification; and

implementing the classification dimension in the data warehouse using the classifier.

18. The computer of claim 17, wherein the logic executed by the computer further comprises adding documents not in the sample to the classification dimension, whereby the classification dimension is useful for searching for documents by classification.

19. The computer of claim 17, wherein the classifier includes a machine-implementable set of rules that can be applied to any data element in the warehouse to generate a label.

20. The computer of claim 17, wherein the sample is pseudo-randomly selected.

21. The computer of claim 17, comprising displaying output using an on-line analytical tool (OLAP) tool.

22. The computer of claim 17, wherein at least the act of generating a default classifier is undertaken using an e-classifier tool.

23. The computer of claim 17, wherein the logic executed by the computer includes:

identifying a subset of documents in the warehouse;

selecting features from the feature space that are relevant to the subset; and

comparing the subset with the sample using the features from the feature space that are relevant to the subset.

24. A computer program product having means executable by a digital processing apparatus to analyze data in a data warehouse, comprising:

means for selecting a sample of documents from the data warehouse;

means for generating at least one feature space of terms of interest in unstructured text fields of the documents using the sample;

means for generating at least one classification using the feature space;

means for establishing at least one classifier using the classification;

means for identifying a subset of documents in the warehouse;

means for selecting features from the feature space that are relevant to the subset; and

means for comparing the subset with the sample using the features from the feature space that are relevant to the subset.

25. The computer program product of claim 24, comprising:

means for implementing a classification dimension in the data warehouse using the classifier.

26. The computer program product of claim 25, further comprising means for adding documents not in the sample to the classification dimension.

27. The computer program product of claim 24, wherein the classifier includes a machine-implementable set of rules that can be applied to any data element in the warehouse to generate a label.

28. The computer program product of claim 24, wherein the sample is pseudo-randomly selected.

29. The computer program product of claim 24, comprising means for displaying output using an on-line analytical tool (OLAP) tool.

30. The computer program product of claim 24, wherein at least the means for generating a default classifier includes an e-classifier tool.