Systems and methods for interest-driven distributed data server systems

• US 9,940,375 B2
• Filed: 07/29/2016
• Issued: 04/10/2018
• Est. Priority Date: 03/08/2013
• Status: Active Grant

• Current Assignee: Workday Incorporated
• Original Assignee: Workday Incorporated
• Inventors: Eshleman, John Glenn, Beyer, Kevin Scott, Babcock, Brian, Kanne, Carl-Christian
• Primary Examiner(s): Mackes, Kris E
• Application Number:
  US15/223,106
• Publication Number:
  US 20170193067A1
• Time in Patent Office: 620 Days
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP

Abstract

Systems and methods for interest-driven business intelligence systems in accordance with embodiments of the invention are illustrated. In one embodiment, an interest-driven distributed data server system includes a processor, an interest-driven distributed data server application, aggregate data storage, and a data source directory, wherein the interest-driven distributed data server application configures the processor to obtain reporting data requirements, determine at least one data dimension using the reporting data requirements, identify at least one data source identified that can provide data associated with the data dimension, generate data source job data, transmit the data source job data to at least one data source, and receive job response data, store the received job response data in the aggregate data storage, build reporting data based on the reporting data requirements, and transmit the reporting data.

Claims

1. An interest-driven distributed data server system, comprising:

a processor;

a memory connected to the processor and configured to store an interest-driven distributed data server application;

aggregate data storage configured to store aggregate data, where a portion of the aggregate data is generated using job response data; and

directory storage configured to store a data source directory;

wherein the data source directory comprises;

at least one reference to at least one data source configured to store raw data; and

data source metadata describing the at least one data source, wherein the data source metadata includes data source performance data; and

wherein the interest-driven distributed data server application configures the processor to;

obtain reporting data requirements, where the reporting data requirements comprise reporting data metadata, wherein the reporting data metadata identifies at least one data source;

determine at least one data dimension using the reporting data requirements;

for each data dimension;

identify at least one data source identified using the data source metadata that can provide data associated with the data dimension;

generate data source job data for at least one of the identified at least one data source based on the data source metadata, wherein the data source job data is transmitted to a second data source not included in the at least one data source, wherein the at least one data source can provide the data associated with the data dimension, and wherein the generating of the data source job data comprises to;

request job response data based on an expected size of the job response data, an expected processing load, a time to execute the generated data source job data, or any combination thereof;

transmit the data source job data to the at least one of the identified at least one data source; and

receive the job response data from the at least one of the identified at least one data source;

update at least one piece of reporting data metadata based on the second data source;

generate data source job data for at least one of the identified at least one data source based on the data source metadata, wherein the at least one data source provides the data and the data source job data is configured to optimize the performance of the at least one data source based on the data source performance data;

store the received job response data in the aggregate data storage;

build reporting data based on the reporting data requirements; and

transmit the reporting data.

2. The interest-driven distributed data server system of claim 1, wherein the interest-driven distributed data server application configures the processor to:

identify existing data stored in the aggregate data storage that fulfills a portion of the reporting data requirements; and

build the reporting data based on the received job response data and the existing data stored in the aggregate data storage.

3. The interest-driven distributed data server system of claim 2, wherein the interest-driven distributed data server application configures the processor to:

identify additional data based on the at least one data dimension and the existing data stored in the aggregate data storage, where the additional data is associated with at least one additional data dimension in the at least one data dimension that is not associated with the existing data stored in the aggregate data storage; and

generate data source job data for the at least one data source associated with the additional data dimension based on the data source metadata, where the generated data source job data includes a request for the identified additional data.

4. The interest-driven distributed data server system of claim 2, wherein:

the aggregate data storage comprises a plurality of data silos, where at least one piece of aggregate data in the aggregate data storage is associated with at least one data silo;

the reporting data metadata comprises data silo association data identifying at least one data silo;

the data source metadata identifies at least one data silo; and

the interest-driven distributed data server application configures the processor to;

identify at least one data source identified using the data source metadata that can provide data associated with the data dimension and at least one data silo in the data silo association data matches at least one data silo in the data source metadata associated with the identified data source; and

associate the at least one data silo in the data silo association data with the received job response data.

5. The interest-driven distributed data server system of claim 4, wherein the interest-driven distributed data server application configures the processor to identify existing data stored in the aggregate data storage based on the reporting data metadata, where when aggregate data in the aggregate data storage is not associated with the at least one data silo in the data silo association data it is not identified as fulfilling a portion of the reporting data requirements.

6. The interest-driven distributed data server system of claim 1, wherein the interest-driven distributed data server application configures the processor to:

obtain an additional data source, where the additional data source comprises an additional data source identity;

determine data source capabilities associated with the additional data source;

generate additional data source metadata using the determined data source capabilities; and

add the data source identity and the generated additional data source metadata to the data source directory.

7. The interest-driven distributed data server system of claim 1, wherein the interest-driven distributed data server application configures the processor to:

generate job response metadata based on the job response data and the data source providing the job response data;

associate the job response metadata with the job response data; and

automatically request updated job response data from the data source providing the job response data using the job response metadata.

8. The interest-driven distributed data server system of claim 1, wherein the interest-driven distributed data server application configures the processor to:

generate data source job data for at least one of the identified at least one data source based on the data source metadata, where the at least one data source can provide the data and the data source job data is configured to request a portion of the data;

receive job response data from the at least one data source, where the job response data contains a partial portion of the data;

build partial reporting data using the received job response;

generate updated data source job data for the at least one of the identified at least one data source based on the data source metadata and the received job response data, where the at least one of the identified at least one data source can provide the remaining portion of the data;

transmit the updated data source job data to the at least one of the identified at least one data source;

receive updated job response data from the at least one data source, where the updated job response data contains the remaining portion of the data; and

build updated reporting data using the updated job response data, where the partial reporting data and the updated reporting data fulfill the reporting data requirements.

9. The system of claim 1, wherein the interest-driven distributed data server application configures the processor to store the generated reporting data using the aggregate data storage.

10. The system of claim 1, wherein the data source job data is selected from the group consisting of structured query language (SQL) queries, NoSQL queries, MapReduce jobs, and distributed file system queries.

11. The system of claim 1, wherein data source job data configures a data source to:

perform extract, transform, and load processes to generate aggregate data using raw data stored in the data source; and

create the job response data using the aggregate data.

12. A method for generating reporting data, comprising:

obtaining reporting data requirements using an interest-driven distributed data server system, where the reporting data requirements comprise reporting data source metadata, wherein the reporting data source metadata identifies at least one data source;

determining at least one data dimension based on the reporting data requirements using the interest-driven distributed data server system;

for each data dimension;

identifying at least one data source identified using the data source metadata that can provide data associated with the data dimension using the interest-driven distributed data server system, wherein the data source metadata includes data source performance data;

generating data source job data for at least one of the identified at least one data source based on the data source metadata using the interest-driven distributed data server system wherein the data source job data is transmitted to a second data source not included in the at least one data source, where the at least one data source can provide the data associated with the data dimension, and wherein the generating of the data source job data comprises;

requesting job response data based on an expected size of the job response data, an expected processing load, a time to execute the generated data source job data, or any combination thereof;

transmitting the data source job data to the at least one of the identified at least one data source using the interest-driven distributed data server system; and

receiving the job response data from the at least one of the identified at least one data source using the interest-driven distributed data server system;

updating at least one piece of reporting data metadata based on the second data source;

generating data source job data for at least one of the identified at least one data source based on the data source metadata, wherein the at least one data source provides the data and the data source job data is configured to optimize the performance of the at least one data source based on the data source performance data;

storing the received job response data using the interest-driven distributed data server system;

building reporting data based on the reporting data requirements using the interest-driven distributed data server system; and

transmitting the reporting data using the interest-driven distributed data server system.

13. The method of claim 12, further comprising:

identifying existing data stored in aggregate data storage that fulfills a portion of the reporting data requirements using interest-driven distributed data storage system; and

building the reporting data based on the received job response data and the existing data stored in the aggregate data storage using the interest-driven distributed data storage system.

14. The method of claim 12, further comprising:

obtaining an additional data source using interest-driven distributed data storage system, where the additional data source comprises an additional data source identity;

determining data source capabilities associated with the additional data source using the interest-driven distributed data storage system;

generating additional data source metadata based on the determined data source capabilities using the interest-driven distributed data storage system; and

adding the data source identity and the generated additional data source metadata to a data source directory contained in the interest-driven distributed data server system using the interest-driven distributed data storage system.

15. The method of claim 12, further comprising:

generating job response metadata based on the job response data and the data source providing the job response data using interest-driven distributed data storage system;

associating the job response metadata with the job response data using the interest-driven distributed data storage system; and

automatically requesting updated job response data from the data source providing the job response data based in the job response metadata using the interest-driven distributed data storage system.

16. The method of claim 12, wherein data source job data configures a data source to perform a process comprising:

performing extract, transform, and load processes to generate aggregate data based in raw data stored in the data source using the data source; and

creating the job response data based on the aggregate data using the data source.

17. An interest-driven distributed data server system, comprising:

a processor;

a memory connected to the processor and configured to store an interest-driven distributed data server application;

aggregate data storage configured to store aggregate data, where a portion of the aggregate data is generated using job response data; and

directory storage configured to store a data source directory;

wherein the data source directory comprises;

at least one reference to at least one data source configured to store raw data; and

data source metadata describing the at least one data source, wherein the data source metadata includes data source performance data; and

wherein the interest-driven distributed data server application configures the processor to;

obtain reporting data requirements, where the reporting data requirements comprise reporting data metadata, wherein the reporting data metadata identifies at least one data source;

determine at least one data dimension using the reporting data requirements;

for each data dimension;

identify at least one data source identified using the data source metadata that can provide data associated with the data dimension;

generate data source job data for at least one of the identified at least one data source based on the data source metadata, wherein the data source job data is transmitted to a second data source not included in the at least one data source, wherein the at least one data source can provide the data associated with the data dimension, and wherein the generating of the data source job data comprises to;

request job response data based on an expected size of the job response data, an expected processing load, a time to execute the generated data source job data, or any combination thereof;

transmit the data source job data to the at least one of the identified at least one data source; and

receive the job response data from the at least one of the identified at least one data source;

update at least one piece of reporting data metadata based on the second data source;

generate data source job data for at least one of the identified at least one data source based on the data source metadata, wherein the at least one data source provides the data and the data source job data is configured to optimize the performance of the at least one data source based on the data source performance data;

store the received job response data in the aggregate data storage;

build reporting data based on the reporting data requirements;

transmit the reporting data;

identify when at least one data source is unavailable;

locate an alternative data source in the identified at least one data source that is available;

generate data source job data for the alternative data source; and

transmit the data source job data to the alternative data source.