System with a data aggregation module generating aggregated data for responding to OLAP analysis queries in a user transparent manner

• US 20070233644A1
• Filed: 02/09/2007
• Published: 10/04/2007
• Est. Priority Date: 02/28/2000
• Status: Abandoned Application

• Current Assignee: Yanicklo Technology LLC (Intellectual Ventures LLC), MEC Management, LLC (Three Affiliated Tribes of the Fort Berthold Reservation, North Dakota)
• Original Assignee: MEC Management, LLC (Three Affiliated Tribes of the Fort Berthold Reservation, North Dakota)
• Inventors: Bakalash, Reuven, Shaked, Guy, Caspi, Joseph
• Application Number:
  US11/704,712
• Publication Number:
  US 20070233644A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24539 using cached or materialised query results
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06F 16/30 of unstructured textual data document management systems G06F16/93
• CPC Class Codes: Y10S 707/954 Relational
• CPC Class Codes: Y10S 707/957 Multidimensional
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

A system for supporting OLAP analysis over a network. The system comprises an OLAP server for enabling an OLAP user to perform OLAP analysis via interaction with a client machine on the network. The system also includes a data aggregation module comprising a multi-dimensional datastore, an aggregation engine integrated with the multi-dimensional datastore, and a first interface for loading base data from a data source to the aggregation engine. The aggregation engine performs data aggregation operations on loaded base data, generates aggregated data from the base data, and stores the aggregated data in the multi-dimensional datastore. A second interface receives requests for OLAP analysis from the OLAP server, accesses the aggregation engine to retrieve from the multi-dimensional datastore, aggregated data corresponding to requests, and communicates the retrieved aggregated data to the OLAP server for query servicing, in a manner transparent to the OLAP user.

Claims

1. An on-line analytical processing (OLAP) system, comprising:

a data aggregation module for servicing queries directed towards high dimensionality sparse data sets, said data aggregation module including;

(1) a multi-dimensional datastore,

(2) an hierarchy transformation module for receiving an original hierarchical database structure of said OLAP system defining parent-child relationships of levels within dimensions and converting said hierarchical database structure into a functionally equivalent hierarchical database structure optimized for rapid aggregation, storage and retrieval of sparse data; and

(3) an aggregation engine for aggregating said large data sets, including sparse data, according to said functionally equivalent hierarchy.

2. The OLAP system of claim 1, further comprising a load and indexing module for organizing said multi-dimensional data store into autonomic segments capable of being rolled up in different sequences according to said functionally equivalent hierarchy, said autonomic segments being stored as records indexed such that each data segment is capable of being independently loaded into a main memory and each data record has a size that is comparatively small compared to a maximum data size of said disk memory space.

3. The OLAP system of claim 2, wherein said aggregation module has a mode of operation in which data is aggregated on-the-fly by said aggregation engine to service a query statement by determining a roll-up order based on any previously pre-aggregated sparse data and loading into said main memory a set of memory data segments having data points for performing aggregation on-the-fly to service the query statement.

4. The OLAP system of claim 1 further comprising an OLAP server, wherein said aggregation module serves as a complimentary accelerator to said OLAP server.

5. An on-line analytical processing (OLAP) system, comprising:

a data aggregation module to service queries directed towards high dimensionality sparse data sets, said data aggregation module including;

(1) a multi-dimensional datastore,

(2) an hierarchy transformation module a for converting an original hierarchical database structure of said OLAP system into a functionally equivalent hierarchical database structure optimized for large data sets, including sparse data having a low density of data points, said functionally equivalent hierarchical database structure being used to perform data indexing and aggregation operations such that groups of related data points at different stages of an aggregation process are organized into sub-units of memory storage that are individually accessible from a memory space of said multi-dimensional datastore; and

(3) an aggregation engine having a mode of operation in which data is aggregated on-the-fly to service a query statement by identifying a set of said sub-units of memory storage having partially pre-aggregated data to perform an aggregation on-the-fly, loading said set of sub-units of memory storage into a main memory, and performing an aggregation on-the-fly to service the query statement.

6. The OLAP system of claim 5, wherein said functionally equivalent hierarchical database structure is optimized for memory and storage access.

7. The OLAP system of claim 5, wherein said functionally equivalent hierarchical database structure is functionally equivalent to multiple hierarchies in at least one dimension.

8. The OLAP system of claim 5, wherein said functionally equivalent hierarchical database structure is chosen to minimize data storage requirements

9. The OLAP system of claim 5, wherein said OLAP system verifies the integrity of said functionally equivalent hierarchical database structure and reports errors.

10. The OLAP system of claim 5, wherein said OLAP system verifies the integrity of said functionally equivalent hierarchical database structure and fixes errors.

11. The OLAP system of claim 5, wherein said sub-units are records within a directory file.

12. The OLAP system of claim 5 wherein said sub-units are slices of multi-dimensional data.

13. The OLAP system of claim 12, wherein each slice is a N−

1 dimensional slice.

14. The OLAP system of claim 5, further comprising an OLAP server, said aggregation module serving as a complimentary accelerator to said OLAP server.

15. An on-line analytical processing (OLAP) system, comprising:

a data aggregation module for use as a complimentary accelerator with an OLAP server to improve the servicing of high dimensionality sparse data sets, said data aggregation module including;

(1) a multi-dimensional datastore,

(2) a load and indexing module for organizing said multi-dimensional datastore into records capable of being independently loaded into a main memory with each record having a size that is small compared with a maximum size of said multi-dimensional data store and each record corresponding to an autonomic segment, each autonomic segment storing base data or aggregated data with autonomic segments corresponding to partially pre-aggregated data capable of being rolled up in different sequences; and

(3) an aggregation engine performing data aggregation utilizing said autonomic segments to limit the amount of simultaneously handled data, said aggregation module having a mode of operation in which data is aggregated on-the-fly to service a given query statement by determining a rollup order of a set of records having autonomic segments capable of being rolled up to service the given query statement, loading into a main memory said set of records, and performing a data aggregation operation to service the query request.

16. The OLAP system of claim 15, further comprising an hierarchy transformation module for receiving a predetermined dimensional hierarchy defining parent-child relationships and converting said predetermined dimensional hierarchy into a functionally equivalent hierarchy optimized for aggregating large data sets, including sparse data, and wherein said aggregation module utilizes said functionally equivalent hierarchy for performing data indexing operations and data aggregation operations.

17. The OLAP system of claim 15, wherein said OLAP system verifies the integrity of said functionally equivalent hierarchy and reports errors.

18. The OLAP system of claim 15, wherein each segment corresponds to a group of related data points.

19. The OLAP system of claim 15, wherein each segment is a slice of multidimensional data.

20. The OLAP system of claim 15 wherein said records are indexed by a directory file.

21. A method for accelerating the servicing of high dimensionality sparse data sets in an on-analytical processing (OLAP) system, comprising:

receiving an original hierarchical database structure of said OLAP system; and

converting said original hierarchical database structure into a functionally equivalent hierarchical database structure optimized for performing data storage and aggregation operations on large data sets, including sparse data.

22. The method of claim 21, further comprising:

performing a segmented aggregation process based on said functionally equivalent hierarchical database structure in which data is segmented into segments of sparse data capable of being rolled up in different rollup orders.

23. The method of claim 22, further comprising:

performing a partial pre-aggregation according to said functionally equivalent hierarchical database structure to generate an initial set of segments stored as data records in a memory space of a multi-dimensional datastore;

in response to receiving a query statement requiring data that has not been pre-aggregated, determining a rollup order of autonomic segments to service the query statement on-the-fly based on said initial set of autonomic segments;

loading into a main memory records corresponding to a subset of said initial set of autonomic records required to perform said rollup; and

performing said rollup to aggregate data on-the-fly to service the query statement.

24. A method for accelerating the servicing of high dimensionality sparse data sets in an on-analytical processing (OLAP) system, comprising:

receiving an hierarchical database structure of said OLAP system;

organizing an aggregation process for autonomic segments capable of being rolled up in different rollup orders;

performing a partial pre-aggregation according to said aggregation process to generate an initial set of autonomic segments stored as data records in a memory space of a multi-dimensional datastore;

in response to receiving a query statement requiring data that has not been pre-aggregated, determining a rollup order of a subset of autonomic segments to service the query statement on-the-fly based on said initial set of autonomic segments;

loading into a main memory records corresponding to said subset of said initial set of autonomic records require to perform said rollup; and

performing said rollup to aggregate data on-the-fly to service the query statement.