Systems and methods for interest-driven distributed data server systems

• US 9,405,811 B2
• Filed: 03/08/2013
• Issued: 08/02/2016
• Est. Priority Date: 03/08/2013
• Status: Active Grant

• Current Assignee: Workday Incorporated
• Original Assignee: Platfora Incorporated (Workday Incorporated)
• Inventors: Eshleman, John Glenn, Beyer, Kevin Scott, Babcock, Brian, Kanne, Carl-Christian
• Primary Examiner(s): MACKES, KRIS E
• Application Number:
  US13/790,624
• Publication Number:
  US 20140258209A1
• Time in Patent Office: 1,243 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP

Abstract

Systems and methods for interest-driven business intelligence systems in accordance with embodiments of the invention are illustrated. In one embodiment, an interest-driven distributed data server system includes a processor, an interest-driven distributed data server application, aggregate data storage, and a data source directory, wherein the interest-driven distributed data server application configures the processor to obtain reporting data requirements, determine at least one data dimension using the reporting data requirements, identify at least one data source identified that can provide data associated with the data dimension, generate data source job data, transmit the data source job data to at least one data source, and receive job response data, store the received job response data in the aggregate data storage, build reporting data based on the reporting data requirements, and transmit the reporting data.

Claims

1. An interest-driven distributed data server system, comprising:

a processor;

a memory connected to the processor and storing an interest-driven distributed data server application; and

a data mart comprising;

aggregate data storage storing aggregate data, where a portion of the aggregate data is generated using job response data; and

directory storage storing a data source directory;

wherein the data source directory comprises;

at least one reference to a data source storing raw data; and

data source metadata describing the at least one data source; and

wherein the interest-driven distributed data server application directs the processor to;

obtain reporting data requirements describing reporting data, where the reporting data requirements comprise reporting data metadata;

determine stored aggregate data in the aggregate data storage that fulfills the reporting data requirements;

determine at least one data dimension of the reporting data using the reporting data requirements;

for each at least one data dimension;

identify at least one data source that can provide data associated with the at least one data dimension from the data source metadata stored in the directory storage;

generate data source job data for the at least one data source to obtain the data associated with the at least one data dimension based on the data source metadata and the reporting data requirements;

transmit the data source job data to the at least one data source; and

receive job response data including the data associated with the at least one data dimension from the at least one data source;

store the received job response data in the aggregate data storage;

build the reporting data based on the reporting data requirements from the job response data received from each of the at least one data source for each of the at least one data dimensions and the identified stored aggregate data; and

transmit the reporting data.

2. The interest-driven distributed data server system of claim 1, wherein the interest-driven distributed data server application directs the processor to:

identify additional data based on the at least one data dimension and the identified stored aggregate data, where the additional data is associated with at least one additional data dimension in the at least one data dimension that is not associated with the identified stored aggregate data; and

generate data source job data for the at least one data source associated with the additional data dimension based on the data source metadata, where the generated data source job data includes a request for the identified additional data.

3. The interest-driven distributed data server system of claim 1, wherein:

the aggregate data storage comprises a plurality of data silos, where at least one piece of aggregate data in the aggregate data storage is associated with at least one of the plurality of data silos;

the reporting data metadata comprises data silo association data identifying at least one data silo;

the data source metadata identifies at least one of the plurality of data silos associated with each at least one data source; and

the interest-driven distributed data server application directs the processor to;

identify the at least one of the plurality of data silos associated with the at least one data source of each at least one data dimensions; and

associate the identified at least one of the plurality of data silos for each at least one data source of each at least one dimension with the job response data received from the at least one data source in the data silo association data.

4. The interest-driven distributed data server system of claim 3, wherein the interest-driven distributed data server application directs the processor to identify existing data stored in the identified stored aggregate data that does not fulfill at least a portion of the reporting data requirements when the existing data is not associated with the at least one of the plurality of data silos in the data silo association data based upon the reporting data metadata in the reporting data requirements.

5. The interest-driven distributed data server system of claim 1, wherein the interest-driven distributed data server application directs the processor to:

obtain an identity of an additional data source;

determine data source capabilities associated with the additional data source;

generate additional data source metadata using the determined data source capabilities; and

add the data source identity and the generated additional data source metadata to the data source directory.

6. The interest-driven distributed data server system of claim 1, wherein:

the reporting data metadata identifies at least one data source;

the data source job data is transmitted to a second data source not included in the at least one data source; and

the interest-driven distributed data server application directs the processor to update the reporting data metadata based on the second data source.

7. The interest-driven distributed data server system of claim 1, wherein the interest-driven distributed data server application directs the processor to:

identify when at least one data source of at least one data dimension is unavailable;

locate an alternative data source in the identified at least one data source for the at least one data dimension that is available;

generate data source job data to obtain the data associated with the at least one data dimension for the alternative data source; and

transmit the data source job data to the alternative data source.

8. The interest-driven distributed data server system of claim 1, wherein:

the data source metadata includes data source performance data for the at least one data source;

the interest-driven distributed data server application directs the processor to generate the data source job data for the at least one data source of at least one data dimension based on the data source metadata; and

the data source job data optimizes the performance of the at least one data source based on the data source performance data.

9. The interest-driven distributed data server system of claim 1, wherein the interest-driven distributed data server application directs the processor to:

generate job response metadata for the job response data received from at least one data source of at least one dimension based on the job response data received from at least one data source of at least one dimension and the at least one data source providing the job response data;

associate the job response metadata for the job response data received from at least one data source of at least one dimension with the job response data received from at least one data source of at least one dimension; and

automatically request updated job response data from the at least one data source providing the job response data using the job response metadata for the job response data received from at least one data source of at least one dimension.

10. The interest-driven distributed data server system of claim 1, wherein the interest-driven distributed data server application directs the processor to:

generate data source job data for the at least one data source of at least one data dimension based on the data source metadata of the at least one data source of the at least one dimension, where the at least one data source can provide the data for the at least one data dimension and the data source job data requests a portion of the data for the at least one data dimension;

receive job response data from the at least one data source, where the job response data contains a partial portion of the data for the at least one data dimension;

build partial reporting data using partial portion of data in the received job response data;

generate updated data source job data for the at least one data source of the at least one data dimension based on the data source metadata and the partial portion of data in the received job response data, where the at least one data source can provide the remaining portion of the data for the at least one data dimension;

transmit the updated data source job data to the at least one data source;

receive updated job response data from the at least one data source, where the updated job response data contains the remaining portion of the data for the at least one data dimension; and

build updated reporting data using the updated job response data, where the partial reporting data and the updated reporting data fulfill the reporting data requirements.

11. The system of claim 1, wherein the interest-driven distributed data server application directs the processor to store the generated reporting data using the aggregate data storage.

12. The system of claim 1, wherein the data source job data is selected from the group consisting of structured query language (SQL) queries, NoSQL queries, MapReduce jobs, and distributed file system queries.

13. The system of claim 1, wherein data source job data-directs a data source to:

perform extract, transform, and load processes to generate aggregate data using raw data stored in the data source; and

create job response data using the aggregate data.

14. A method for generating reporting data, comprising:

obtaining reporting data requirements using an interest-driven distributed data server system, wherein;

the reporting data requirements comprise reporting data metadata;

the interest-driven data server system comprises a processor, a memory connected to the processor and a data mart comprising;

aggregate data storage storing aggregate data, where a portion of the aggregate data is generated using job response data; and

directory storage storing a data source directory comprising;

a reference to each at least one data source storing raw data; and

data source metadata describing the at least one data source;

identifying stored aggregate data in the aggregate data storage that fulfills the reporting data requirements;

determining at least one data dimension based on the reporting data requirements using the interest-driven distributed data server system;

for each at least one data dimension;

identifying the at least one data source that can provide data associated with the at least one data dimension using the interest-driven distributed data server system;

generating data source job data for the identified at least one data source that can provide data associated with the at least one data dimension based on the data source metadata using the interest-driven distributed data server system;

transmitting the data source job data to the identified at least one data source using the interest-driven distributed data server system; and

receiving job response data from the identified at least one data source of the at least one data dimension using the interest-driven distributed data server system;

storing the received job response data for the at least one data dimension using the interest-driven distributed data server system;

building reporting data using the stored aggregate data that fulfills the reporting data requirements and job response data from each of the at least one data sources for each at least one data dimension based on the reporting data requirements using the interest-driven distributed data server system; and

transmitting the reporting data using the interest-driven distributed data server system.

15. The method of claim 14, further comprising:

obtaining an identity of an additional data source using the interest-driven distributed data storage system;

determining data source capabilities associated with the additional data source using the interest-driven distributed data storage system;

generating additional data source metadata based on the determined data source capabilities using the interest-driven distributed data storage system; and

adding the data source identity and the generated additional data source metadata to a data source directory contained in the interest-driven distributed data server system using the interest-driven distributed data storage system.

16. The method of claim 14, further comprising:

identifying when at least one data source of at least one data dimension is unavailable using the interest-driven distributed data storage system;

locating an alternative data source in the identified at least one data source of the at least one data dimension that is available using the interest-driven distributed data storage system;

generating data source job data for the alternative data source that can provide data associated with the at least one data dimension using the interest-driven distributed data storage system; and

transmitting the data source job data to the alternative data source using the interest-driven distributed data storage system.

17. The method of claim 14, further comprising:

generating job response metadata for the job response data received from the at least one data source of at least one data dimension based on the job response data received from the at least one data source of at least one data dimension and the data source providing the job response data using the interest-driven distributed data storage system;

associating the generated job response metadata with the job response data received from the at least one data source of at least one data dimension using the interest-driven distributed data storage system; and

automatically requesting updated job response data from the at least one data source providing the job response data based in the job response metadata using the interest-driven distributed data storage system.

18. The method of claim 14, wherein data source job data directs a data source to perform a process comprising:

performing extract, transform, and load processes to generate aggregate data based in raw data stored in the data source using the data source; and

creating job response data based on the aggregate data using the data source.