Textual on-line analytical processing method and system

• US 20040049505A1
• Filed: 09/11/2002
• Published: 03/11/2004
• Est. Priority Date: 09/11/2002
• Status: Abandoned Application

• Current Assignee: Intelligent Results
• Original Assignee: Intelligent Results
• Inventors: Pennock, Kelly
• Application Number:
  US10/241,981
• Publication Number:
  US 20040049505A1
• Time in Patent Office: Days
• US Class Current: 707/5
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06F 16/30 of unstructured textual data document management systems G06F16/93
• CPC Class Codes: G06F 2216/03 Data mining

Abstract

The present invention provides for a system and method that allows OLAP analysis of unstructured content. This is accomplished by transforming isolated, unstructured content into quantifiable structured data, thereby creating a common measure for performing OLAP analysis. This allows the seamless integration of unstructured content with structured data sources. It also allows for the ability to query what was before unqueriable information that enterprises were in possession of.

Claims

1. A method for processing unstructured documents to populate an OLAP data structure, the method comprising:

selecting a plurality of unstructured documents from a corpus of unstructured documents;

computing a document representation for each selected document;

organizing said selected documents into a hierarchy of document clusters based on said document representations;

populating the OLAP data structure using said hierarchy of document clusters, and;

computing a document measure for each selected document.

2. The method of claim 1, wherein said document representation is a document vector.

3. The method of claim 1, wherein said document representation for an selected document is computed by:

filtering features of interest in said selected documents;

weighting said filtered features of interest; and

determining a value for said document representation based on said weighted features of interest.

4. The method of claim 3, wherein filtering features of interest in said selected documents comprises:

generating an inverted file index for said selected documents, wherein said inverted file index identifies each feature of interest, the selected document or documents in which each feature of interest occurs, and the frequency in which each feature of interest occurs in said selected documents; and

removing features of interest based on the frequency in which said features of interest occur in said selected documents.

5. The method of claim 4, wherein filtering features of interest further comprises normalizing related features of interest into a common feature of interest.

6. The method of claim 4, wherein removing features of interest based on the frequency in which said features of interest occur in said selected documents comprises removing features of interest that occur at a frequency above a predetermined threshold.

7. The method of claim 4, wherein removing features of interest based on the frequency at which said features of interest occur in said selected documents comprises removing features of interest that occur at a frequency below a predetermined threshold.

8. The method of claim 4, wherein at least some of said features of interest are word features.

9. The method of claim 8, wherein said word features removed are function words.

10. The method of claim 8, wherein said word features removed are stop words.

11. The method of claim 8, wherein word features removed are case variations of the same word.

12. The method of claim 4, wherein at least some of said features of interest are non-word features.

13. The method of claim 3, wherein weighting said filtered features of interest comprises assigning a greater weight to those features of interest that occur at a higher frequency within a particular document.

14. The method of claim 2, wherein the direction and magnitude of said document vector are determined by cosine measure.

15. The method of claim 1, wherein said document measure is an attribute score.

16. The method of claim 1, wherein organizing said selected documents into a hierarchy of document clusters comprises:

(a) forming a first prior level of document clusters based on similarities between the respective document measures of said selected documents;

(b) computing an average document measure for each document cluster in the prior level of document clusters, and

(c) forming a next level of document clusters based on similarities between the respective average document measures of the document clusters in the prior level of document clusters.

17. The method of claim 16 further comprising repeating (b) and (c) until the next level of document clusters forms a root document cluster.

18. The method of claim 16, wherein each document cluster in the first prior level of document clusters is formed by grouping together selected documents with similar document measures.

19. The method of claim 16, wherein each document cluster in the next level of document clusters is formed by grouping together document clusters from the prior level with similar average document measures.

20. The method of claim 1 further comprising filtering said selected documents.

21. The method of claim 1 further comprising applying an OLAP tool to the OLAP data structure.

22. The method of claim 21, wherein said OLAP tool is a drill-down tool.

23. The method of claim 21, wherein said OLAP tool is a roll-up tool.

24. The method of claim 1 further comprising obtaining information from selected documents by querying the OLAP data structure.

25. The method of claim 24, wherein said queried information is depicted in a pivot table.

26. The method of claim 24, wherein said queried information is depicted in a chart.

27. A computer readable medium containing computer executable instructions for processing unstructured documents to populate an OLAP data structure, the computer readable medium comprising:

a selection module for;

selecting a plurality of unstructured documents from a corpus of unstructured documents;

a representation module for;

computing a document representation for each selected document; and

an organization module for;

organizing said selected documents into a hierarchy of document clusters based on said document representations;

populating the OLAP data structure using said hierarchy of document clusters, and;

computing a document measure for each selected document.

28. The computer readable medium of claim 27, wherein said document representation is a document vector.

29. The computer readable medium of claim 27, wherein representation module further comprises instructions for:

filtering features of interest in said selected documents;

weighting said filtered features of interest; and

determining a value for said document representation based on said weighted features of interest.

30. The computer readable medium of claim 29, wherein filtering features of interest in said selected documents comprises:

generating an inverted file index for said selected documents, wherein said inverted file index identifies each feature of interest, the selected document or documents in which each feature of interest occurs, and the frequency in which each feature of interest occurs in said selected documents; and

removing features of interest based on the frequency in which said features of interest occur in said selected documents.

31. The computer readable medium of claim 30, wherein filtering features of interest further comprises normalizing related features of interest into a common feature of interest.

32. The computer readable medium of claim 30, wherein removing features of interest based on the frequency in which said features of interest occur in said selected documents comprises removing features of interest that occur at a frequency above a predetermined threshold.

33. The computer readable medium of claim 30, wherein removing features of interest based on the frequency at which said features of interest occur in said selected documents comprises removing features of interest that occur at a frequency below a predetermined threshold.

34. The computer readable medium of claim 30, wherein at least some of said features of interest are word features.

35. The computer readable medium of claim 34, wherein said word features removed are function words.

36. The computer readable medium of claim 34, wherein said word features removed are stop words.

37. The computer readable medium of claim 34, wherein word features removed are case variations of the same word.

38. The computer readable medium of claim 30, wherein at least some of said features of interest are non-word features.

39. The computer readable medium of claim 29, wherein weighting said filtered features of interest comprises assigning a greater weight to those features of interest that occur at a higher frequency within a particular document.

40. The computer readable medium of claim 28, wherein the direction and magnitude of said document vector are determined by cosine measure.

41. The computer readable medium of claim 27, wherein said document measure is an attribute score.

42. The computer readable medium of claim 27, wherein the organization module organizes documents into hierarchies by:

(a) forming a first prior level of document clusters based on similarities between the respective document measures of said selected documents;

(b) computing an average document measure for each document cluster in the prior level of document clusters, and

(c) forming a next level of document clusters based on similarities between the respective average document measures of the document clusters in the prior level of document clusters.

43. The computer readable medium of claim 42 further comprising repeating (b) and (c) until the next level of document clusters forms a root document cluster.

44. The computer readable medium of claim 42, wherein each document cluster in the first prior level of document clusters is formed by grouping together selected documents with similar document measures.

45. The computer readable medium of claim 42, wherein each document cluster in the next level of document clusters is formed by grouping together document clusters from the prior level with similar average document measures.

46. The computer readable medium of claim 27 wherein the selection module further comprises filtering said selected documents.

47. The computer readable medium of claim 27 further comprising a query module for applying an OLAP tool to the OLAP data structure.

48. The computer readable medium of claim 47, wherein said OLAP tool is a drill-down tool.

49. The computer readable medium of claim 47, wherein said OLAP tool is a roll-up tool.

50. The computer readable medium of claim 27 further comprising a query module for obtaining information from selected documents by querying the OLAP data structure.

51. The computer readable medium of claim 50, wherein said queried information is depicted in a pivot table.

52. The computer readable medium of claim 50, wherein said queried information is depicted in a chart.

53. A computing apparatus for processing unstructured documents to populate an OLAP data structure, the computing apparatus operative to:

select a plurality of unstructured documents from a corpus of unstructured documents;

compute a document representation for each selected document;

organize said selected documents into a hierarchy of document clusters based on said document representations;

populate the OLAP data structure using said hierarchy of document clusters, and;

compute a document measure for each selected document.

54. The computing apparatus of claim 53, wherein said document representation is a document vector.

55. The computing apparatus of claim 53, wherein said document representation for an selected document is computed by:

filtering features of interest in said selected documents;

weighting said filtered features of interest; and

determining a value for said document representation based on said weighted features of interest.

56. The computing apparatus of claim 55 wherein filtering features of interest in said selected documents comprises:

generating an inverted file index for said selected documents, wherein said inverted file index identifies each feature of interest, the selected document or documents in which each feature of interest occurs, and the frequency in which each feature of interest occurs in said selected documents; and

removing features of interest based on the frequency in which said features of interest occur in said selected documents.

57. The computing apparatus of claim 56, wherein filtering features of interest further comprises normalizing related features of interest into a common feature of interest.

58. The computing apparatus of claim 56, wherein removing features of interest based on the frequency in which said features of interest occur in said selected documents comprises removing features of interest that occur at a frequency above a predetermined threshold.

59. The computing apparatus of claim 56, wherein removing features of interest based on the frequency at which said features of interest occur in said selected documents comprises removing features of interest that occur at a frequency below a predetermined threshold.

60. The computing apparatus of claim 56, wherein at least some of said features of interest are word features.

61. The computing apparatus of claim 60, wherein said word features removed are function words.

62. The computing apparatus of claim 60, wherein said word features removed are stop words.

63. The computing apparatus of claim 60, wherein word features removed are case variations of the same word.

64. The computing apparatus of claim 56, wherein at least some of said features of interest are non-word features.

65. The computing apparatus of claim 55, wherein weighting said filtered features of interest comprises assigning a greater weight to those features of interest that occur at a higher frequency within a particular document.

66. The computing apparatus of claim 54, wherein the direction and magnitude of said document vector are determined by cosine measure.

67. The computing apparatus of claim 53, wherein said document measure is an attribute score.

68. The computing apparatus of claim 53, wherein organizing said selected documents into a hierarchy of document clusters comprises:

(a) forming a first prior level of document clusters based on similarities between the respective document measures of said selected documents;

(b) computing an average document measure for each document cluster in the prior level of document clusters, and

(c) forming a next level of document clusters based on similarities between the respective average document measures of the document clusters in the prior level of document clusters.

69. The computing apparatus of claim 68 further operative to repeat (b) and (c) until the next level of document clusters forms a root document cluster.

70. The computing apparatus of claim 68, wherein each document cluster in the first prior level of document clusters is formed by grouping together selected documents with similar document measures.

71. The computing apparatus of claim 68, wherein each document cluster in the next level of document clusters is formed by grouping together document clusters from the prior level with similar average document measures.

72. The computing apparatus of claim 53 further operative to filter said selected documents.

73. The computing apparatus of claim 53 further operative to apply an OLAP tool to the OLAP data structure.

74. The computing apparatus of claim 73, wherein said OLAP tool is a drill-down tool.

75. The computing apparatus of claim 73, wherein said OLAP tool is a roll-up tool.

76. The computing apparatus of claim 53 further operative to obtain information from selected documents by querying the OLAP data structure.

77. The computing apparatus of claim 76, wherein said queried information is depicted in a pivot table.

78. The computing apparatus of claim 76, wherein said queried information is depicted in a chart.