Data aggregation module supporting dynamic query responsive aggregation during the servicing of database query requests provided by one or more client machines

• US 8,799,209 B2
• Filed: 11/04/2011
• Issued: 08/05/2014
• Est. Priority Date: 08/04/1999
• Status: Expired due to Term

• Current Assignee: MEC Management, LLC (Three Affiliated Tribes of the Fort Berthold Reservation, North Dakota)
• Original Assignee: Yanicklo Technology LLC (Intellectual Ventures LLC)
• Inventors: Bakalash, Reuven, Shaked, Guy, Caspi, Joseph
• Primary Examiner(s): WILSON, KIMBERLY LOVEL
• Application Number:
  US13/289,954
• Publication Number:
  US 20120089564A1
• Time in Patent Office: 1,005 Days
• Field of Search: 707/603, 707/954, 707/957, 707/966
• US Class Current: 707/603
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

Improved method of and apparatus for aggregating data elements in multidimensional databases (MDDB). In the preferred embodiment, the apparatus is realized in the form of a high-performance stand-alone (i.e. external) aggregation server which can be plugged-into conventional MOLAP systems to achieve significant improvements in system performance. In accordance with the principles of the present invention, the stand-alone aggregation server contains a scalable MDDB and a high-performance aggregation engine that are integrated into the modular architecture of the aggregation server. The stand-alone aggregation server of the present invention can uniformly distribute data elements among a plurality of processors, for balanced loading and processing, and therefore is highly scalable.

Claims

1. A system comprising:

a parallel computing platform comprising one or more processors and one or more storage volumes, wherein each processor is associated with space within said storage volumes for physically storing data elements thereof and wherein the location of each data element in a multidimensional database (MDB) is specified in MDB space by business dimensions associated with said data element;

a data loading mechanism, associated with said parallel computing platform, for loading in the business dimensions and associated data elements from a data warehouse;

an address data mapping mechanism, associated with said data loading mechanism, for mapping the business dimensions associated with each said data element to a processor identifier, and for mapping the business dimensions associated with each said data element into a data storage address within space associated with a specific processor identified by the processor identifier corresponding to said business dimensions; and

a data accessing mechanism in cooperation with said address data mapping mechanism, for accessing each said data element in said one or more storage volumes using said data storage address mapped thereto.

2. The system of claim 1, wherein said address data mapping mechanism is configured to map said business dimensions into said data storage address for each said data element using a modular arithmetic function.

3. The system of claim 2, wherein parallel data loading operations are carried out by said data loading mechanism.

4. The system of claim 2, wherein parallel data aggregation operations are carried out within said MDB using said modular arithmetic function which maps said business dimensions associated with raw or previously pre-aggregated data elements to be stored within said MDB, into data storage addresses within said storage volumes at which the pre-aggregated data elements are to be stored.

5. The system of claim 4, wherein OLAP operations are carried out within said MDB using said modular arithmetic function which maps said business dimensions associated with pre-aggregated data elements to be accessed from said MDB, into data storage addresses within said storage volumes, from which said pre-aggregated data elements are to be accessed.

6. The system of claim 1, wherein data processing tasks carried out therein are evenly distributed among said one or more processors on said parallel computing platform.

7. The system of claim 1, wherein data elements within said MDB are evenly distributed among said one or more processors on said parallel computing platform.

8. The system of claim 1, wherein each said processor on said parallel computing platform handles data elements assigned thereto during data elements address assignment operations carried out during parallel data loading operations between said data warehouse and said MDB within said system.

9. The system of claim 1, wherein each said processor on said parallel computing platform handles data elements assigned thereto during data elements address assignment operations carried out during parallel data aggregation operations within said MDB of said system.

10. The system of claim 1, wherein interprocessor communication among said one or more processors is minimized during parallel data loading operations carried out between said data warehouse and said MDB on said parallel computing platform.

11. The system of claim 1, wherein interprocessor communication among said one or more processors is minimized during parallel data aggregation operations carried out within said MDB on said parallel computing platform.

12. The system of claim 1, wherein interprocessor communication among said one or more processors is minimized during OLAP operations carried out within said MDB on said parallel computing platform.

13. The system of claim 1, wherein said data warehouse comprises a relational database management system (RDBMS).

14. The system of claim 1, wherein said address data mapping mechanism maps the business dimensions associated with each said data element into data storage addresses within data storage based upon the following formula:

Loc(D0,D1,Dn)=[D0+sizeD0*(D1+sizeD1*( . . . *Dn)))/p]

where p represents a quantity of processors to which data elements are being assigned, and D_i represents an index of a respective i_th dimension.

15. The system of claim 1, wherein said address data mapping mechanism maps the business dimensions associated with each said data element to a processor identifier using a modular arithmetic function based on how many processors are included in said parallel platform.

16. A method comprising:

(a) providing a parallel computing platform comprising one or more processors and one or more storage volumes, wherein each processor is associated with space within said storage volumes for physically storing data elements therein, and wherein the location of each data element in a multidimensional database (MDB) is specified in MDB space by business dimensions associated with said data element;

(b) loading in the business dimensions and associated data elements from a data warehouse;

(c) mapping the business dimensions associated with each said data element to a processor identifier, and mapping the business dimensions associated with each said data element into a data storage address within space associated with a specific processor identified by the processor identifier corresponding to said business dimensions; and

(d) using said data storage addresses to access said data elements from said one or more storage volumes.

17. The method of claim 16, wherein (c) comprises mapping said business dimensions for each said data element into said data storage address using a modular arithmetic function.

18. The method of claim 16, wherein (b) comprises parallel data loading operations carried out between said data warehouse and said MBD and (c) comprises using a modular-arithmetic function to map said business dimensions associated with each raw data element accessed from said data warehouse, into a data storage address within one of said storage volumes associated with said MDB.

19. The method of claim 18, wherein OLAP operations are carried out within said MDB and (c) comprises using said modular arithmetic function to map said business dimensions associated with pre-aggregated data elements to be accessed from said MBD, into data storage addresses within said storage volumes, from which said pre-aggregated data elements are to be accessed.

20. The method of claim 16, wherein parallel data aggregation operations are carried out within said MDB and (c) comprises using a modular arithmetic function to map said business dimensions associated with raw or previously pre-aggregated data elements to be stored within said MDB, into data storage addresses within said storage volumes at which the pre-aggregated data elements are to be stored.

21. The method of claim 16, wherein data processing tasks carried out therein are evenly distributed among said one or more processors on said parallel computing platform.

22. The method of claim 16, wherein data elements within said MDB are evenly distributed among said one or more processors on said parallel computing platform.

23. The method of claim 16, wherein each said processor on said parallel computing platform handles data elements assigned thereto during data elements address assignment operations carried out during parallel data loading operations between said data warehouse and said MDB within said system.

24. The method of claim 16, wherein interprocessor communication among said one or more processors is minimized during parallel data loading operations carried out in (b).

25. The method of claim 16, wherein interprocessor communication among said one or more processors is minimized during parallel data loading operations carried out between a relational database management system (RDSMS) and said MDB on said parallel computing platform.

26. The method of claim 16, wherein interprocessor communication among said one or more processors is minimized during parallel data aggregation operations carried out within said MDB on said parallel computing platform.

27. The method of claim 16, wherein interprocessor communication among said one or more processors is minimized during OLAP operations carried out within said MDB on said parallel computing platform.

28. The method of claim 16, wherein said data warehouse comprises a relational database management system (RDBMS).

29. The method of claim 16, wherein said mapping maps the business dimensions associated with each said data element into data storage addresses within said data storage based upon the following formula:

Loc(D0,D1,Dn)=[D0+sizeD0*(D1+sizeD1*( . . . *Dn)))/p]

where p represents a quantity of processors to which data elements are being assigned, and D_i represents an index of a respective i_th dimension.

30. The method of claim 16, wherein said mapping maps integer-encoded business dimensions associated with each said data element to a processor identifier using a modular arithmetic function based on how many processors are included in said parallel platform.

31. A system comprising:

a hardware platform on which a stand-alone data aggregation server resides, the stand-alone data aggregation server comprising at least;

an aggregation engine that is integrated with a multidimensional data store and an interface, the aggregation engine performing a query directed on-the-fly roll-up data aggregation operations along multiple dimensions on data loaded from a database and storing the resultant aggregated data in a multidimensional datastore,

wherein the on-the-fly roll-up data aggregation operations are performed at least partly in response to a query received while at least one aggregation operation is already being performed; and

the interface configured to receive requests communicated from any one of a plurality of different client nodes, configured to access the aggregation engine to retrieve from the multidimensional datastore aggregated data pertaining to said requests, and configured to communicate the aggregated data corresponding to said requests to said respective node that communicated said requests.

32. The system of claim 31, wherein the plurality of different client nodes comprise a plurality of different OLAP servers distributed by different vendors.

33. The system of claim 31, wherein the interface implements a standard protocol for accessing data.

34. The system of claim 33, wherein the standard protocol comprises one of OLDB, OLE-DB, ODBC, SQL, and JDBC.

35. The system of claim 31, wherein computational tasks performed by the aggregation engine is restricted to data aggregation operations.

36. The system of claim 31, wherein said interface extracts dimensions from the received requests and forwards the dimensions to a storage management module, and wherein the storage management module accesses locations of the multidimensional datastore based upon the forwarded dimensions and returns the retrieved data back to the interface for communication to the respective node that communicated said requests.

37. The system of claim 31, further comprising control logic that, upon determining that multidimensional datastore does not contain aggregated data required to service at least one given request, controls the aggregation engine to perform aggregation operations to thereby generate the required aggregated data and return the required aggregated data back to the interface for communication to said respective node that communicated said requests.

38. The system of claim 37, wherein the control logic controls a storage management module to store the required aggregation data generated by the aggregation engine in the multidimensional database.

39. The system of claim 31, wherein data stored in the multidimensional datastore is logically partitioned into N dimensions, wherein the aggregation engine performs a first stage of aggregation operations along a first dimension, and performs and second stage of aggregation operations for a given slice in the first dimension along N−

1 dimensions other than the first dimension.

40. The system of claim 39, wherein the aggregation engine stores the resultant data of aggregation operations for the given slice as a record in a data file, wherein location of the record in the data file is stored in a directory.

41. The system of claim 40, wherein the directory stores, for a given record, a start address and end address of the record and a physical address of the data file.

42. The system of claim 31, wherein time delay in responding to said requests is equivalent to accessing a local datastore of the respective node.

43. The system of claim 31, further comprising an on-line analytical processing (OLAP) system comprising:

a plurality of client machines communicating with an OLAP server over a network;

the OLAP server including OLAP analysis logic and presentation logic to enable user-directed OLAP analysis on data; and

the stand-alone aggregation server configured to operably communicate with the OLAP server to perform data aggregation operations on the data, and store and manage such data for access by the OLAP server.

44. The system of claim 43, wherein the network includes the infrastructure of the Internet.

45. The system of claim 44, wherein said client machines include a web-browser-based user interface that enables said user-directed OLAP analysis.

46. The system of claim 31, further comprising a decision support system operable within an enterprise, the decision support system comprising:

a plurality of client machines communicating with a first server over a network;

the first server including support for user-directed analysis of data; and

the stand-alone aggregation configured to operably communicate with the first server to perform data aggregation operations on the data, and store and manage such data for access by the first server.

47. The system of claim 46, wherein the network includes the infrastructure of the Internet.

48. The system of claim 47, wherein said client machines include a web-browser-based user interface that enables said user-directed analysis.

49. The system of claim 31, wherein the stand-alone data aggregation server is configured to be used with any one of a plurality of different OLAP servers that each provide an application layer and presentation layer through which multiple users perform OLAP analyses via interaction with client machines operably coupled thereto.