Dynamic data aggregation from a plurality of data sources

• US 9,292,575 B2
• Filed: 11/19/2010
• Issued: 03/22/2016
• Est. Priority Date: 11/19/2010
• Status: Active Grant

• Current Assignee: Daedalus Blue LLC
• Original Assignee: International Business Machines Corporation
• Inventors: Bose, Abhijit, Smadi, Mithkal M., Sun, Jimeng, Velpuri, Chandra Kumar
• Primary Examiner(s): Smith, Brannon W
• Application Number:
  US12/950,809
• Publication Number:
  US 20120130987A1
• Time in Patent Office: 1,950 Days
• Field of Search: 707/600, 707/605, 707/999.001, 707/632
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination

Abstract

Dynamically aggregating data is provided. A server device receives a subscriber request for a report based on a subset of metadata contained in a data dimensions catalog. The server device analyzes data aggregation requirements from a plurality of data sources for the report based on the subset of metadata defined in the subscriber request. The server device generates a data access plan for movement of data from the plurality of data sources based on the data aggregation requirements for the report. Then, the server device executes the data access plan to fetch the data from the plurality of data sources based on the data aggregation requirements for the report.

Claims

1. A method for dynamically aggregating data, the method comprising:

receiving, by a data aggregation server implemented between a plurality of remote subscribing client applications and a plurality of distributed remote databases, from each distributed remote database in the plurality of distributed remote databases via a network, tree structure metadata describing data dimensions of data stored in a respective distributed remote database and specifying aggregation requirements for the data dimensions, each of the plurality of distributed remote databases including a set of data dimensions describing a set of facts;

aggregating, by the data aggregation server, the tree structure metadata received from all of the distributed remote databases in the plurality of distributed remote databases to build a global metadata tree structure in a data dimensions catalog of available data dimensions across the plurality of distributed remote databases;

publishing, by the data aggregation server, a list of available data dimensions from which reports are generated to the plurality of remote subscribing client applications via the network;

receiving, by the data aggregation server, a subscriber request for a report based on a subset of metadata within the global metadata tree structure of the data dimensions catalog, the subscriber request being received from a remote subscribing client application using a metadata subscription application programming interface that provides a real-time data feed to and from components in the data aggregation server in order to subscribe to or search for the available data dimensions;

responsive to receiving the subscriber request, analyzing, by the data aggregation server, data aggregation requirements from a set of distributed remote databases for the report based on the subset of metadata defined by the remote subscribing client application in the subscriber request;

generating, by the data aggregation server, a data access plan based on the subset of raetadata for movement of data from the set of distributed remote databases based on the analyzing of the data aggregation requirements for the subset of metadata; and

sending, by the data aggregation server, the data access plan to the plurality of distributed remote databases along with connection information of the remote subscribing client application, the connection information providing a network connection path for the plurality of distributed remote databases to send requested data directly to the remote subscribing client application instead of to the data aggregation server.

2. The method of claim 1 further comprising:

receiving an updated set of metadata from a remote database that reflects a data dimensions update in the remote database.

3. The method of claim 1 wherein the data aggregation server provides a distributed data aggregation service to the plurality of remote subscribing client applications without utilizing a centralized multidimensional data warehouse.

4. The method of claim 1 wherein the remote subscribing subscriber client application communicates with the data aggregation server via the metadata subscription application programming interface that provides the real-time data feed between the remote subscribing client application and the data aggregation server.

5. The method of claim 1 further comprising:

authenticating, by the data aggregation server, the subscriber request from the remote subscribing client application using public and private keys.

6. The method of claim 1, wherein the plurality of distributed remote databases includes one or more relational databases and one or more object-oriented databases.

7. The method of claim 1 further comprising:

responsive to determining that one or more of the plurality of distributed remote databases are not authorized to update the data dimensions catalog with metadata, discarding, by the data aggregation server, received metadata from the one or more distributed remote databases not authorized.

8. A data aggregation server dynamically aggregating data, the data aggregation server comprising:

one or more processors, one or more computer-readable memories, and one or more computer-readable storage devices and program instructions, stored on at least one of the one or more storage devices fir execution by at least one of the one or more processors via at least one of the one or more memories, the program instructions comprising;

program instructions to receive, by the data aggregation server implemented between a plurality of remote subscribing client applications and a plurality of distributed remote databases, from each distributed remote database in the plurality of distributed remote databases via a network, tree structure metadata describing data dimensions of data stored in a respective distributed remote database and specifying aggregation requirements for the data dimensions, each of the plurality of distributed remote databases including a set of data dimensions describing a set of facts;

program instructions to aggregate, by the data aggregation server, the tree structure metadata received from all of the distributed remote databases in the plurality of distributed remote databases to build a global metadata tree structure in a data dimensions catalog of available data dimensions across the plurality of distributed remote databases;

program instructions to publish a list of available data dimensions from which reports are generated to the plurality of remote subscribing client applications via the network;

program instructions to receive a subscriber request for a report based on a subset of metadata within the global metadata tree structure of the data dimensions catalog, the subscriber request being received from a remote subscribing client application using a metadata subscription application programming interface that provides a real-time data feed to and from components in the data aggregation server in order to subscribe to or search for the available data dimensions;

program instructions to analyze data aggregation requirements from a set of distributed remote databases for the report based on the subset of metadata defined by the remote subscribing client application in the subscriber request in response to receiving the subscriber request;

program instructions to generate a data access plan based on the subset of metadata for movement of data from the set of distributed remote databases based on analyzing the data aggregation requirements for the subset of metadata; and

program instructions to send the data access plan to the plurality of distributed remote databases along with connection information of the remote subscribing client application, the connection information providing a network connection path for the plurality of distributed remote databases to send requested data directly to the remote subscribing client application instead of to the data aggregation server.

9. The data aggregation server of claim 8 further comprising:

program instructions to authenticate the subscriber request from the remote subscribing client application using public and private keys.

10. The data aggregation server of claim 8, wherein the plurality of distributed remote databases includes one or more relational databases and one or more object-oriented databases.

11. The data aggregation server of claim 8 further comprising:

program instructions to discard received metadata from one or more of the plurality of distributed remote databases not authorized to update the data dimensions catalog with metadata in response to determining that the one or more of the plurality of distributed remote databases are not authorized to update the data dimensions catalog with metadata.

12. A computer program product for dynamically aggregating data, the computer program product comprising:

one or more non-transitory computer-readable storage devices and program instructions, stored on at least one of the one or more non-transitory computer-readable storage devices, the program instructions comprising;

program instructions to receive, by a data aggregation server implemented between a plurality of remote subscribing client applications and a plurality of distributed remote databases, from each distributed remote database in the plurality of distributed remote databases via a network, structure metadata describing data dimensions of data stored in a respective distributed remote database and specifying aggregation requirements for the data dimensions, each of the plurality of distributed remote databases including a set of data dimensions describing a set of facts;

program instructions to aggregate, by the data ag)^-re ration server, the tree structure metadata received from all of the distributed remote databases in the plurality of distributed remote databases to build a global metadata tree structure in a data dimensions catalog of available data dimensions across the plurality of distributed remote databases;

program instructions to publish a list of available data dimensions from which reports are generated to the plurality of remote subscribing client applications via the network;

program instructions to receive a subscriber request for a report based on a subset of metadata within the globa1 metadata tree structure of the data dimensions catalog, the subscriber request being received from a remote subscribing client application using a metadata subscription application programming interface that provides a real-time data feed to and from components in the data aggregation server in order to subscribe to or search for the available data dimensions;

program instructions to analyze data aggregation requirements from. a set of distributed remote databases for the report based on the subset of metadata defined by the remote subscribing client application in the subscriber request in response to receiving the subscriber request;

program instructions to generate a data access plan based on the subset of metadata fbr movement of data from the set of distributed remote databases based on analyzing the data aggregation requirements for the subset of metadata; and

program instructions to send the data access plan to the plurality of distributed remote databases along with connection information of the remote subscribing client application, the connection information providing a network connection path for the plurality of distributed remote databases to send requested data directly to the remote subscribing client application instead of to the data aggregation server.

13. The computer program product of claim 12 further comprising:

program instructions to receive an updated set of metadata from a remote database that reflects a data dimensions update in the remote database.

14. The computer program product of claim 12 further comprising:

program instructions to provide a distributed data aggregation service to the plurality of remote subscribing client applications without utilizing a centralized multidimensional data warehouse.

15. The computer program product of claim 12 further comprising:

program instructions to authenticate the subscriber request from the remote subscribing client application using public and private keys.

16. The computer program product of claim 12, wherein the plurality of distributed remote databases includes one or more relational databases and one or more object-oriented databases.

17. The computer program product of claim 12 further comprising:

program instructions to discard received metadata from one or more of the plurality of distributed remote databases not authorized to update the data dimensions catalog with metadata in response to determining that the one or more of the plurality of distributed remote databases are not authorized to update the data dimensions catalog with metadata.