Interest-Driven Business Intelligence Systems and Methods of Data Analysis Using Interest-Driven Data Pipelines

• US 20130013552A1
• Filed: 02/29/2012
• Published: 01/10/2013
• Est. Priority Date: 07/07/2011
• Status: Active Grant

• Current Assignee: Workday Incorporated
• Original Assignee: Platfora Incorporated (Workday Incorporated)
• Inventors: Beyer, Kevin Scott, Zhang, Yewei, Eshleman, John Glenn, Werther, Benjamin Mark, Babcock, Brian F.
• Granted Patent: US 8,447,721 B2
• Time in Patent Office: Days
• US Class Current: 707/600
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06Q 10/00 Administration; Management
• CPC Class Codes: G06Q 10/067 Enterprise or organisation modelling

Abstract

Interest-driven Business Intelligence (BI) systems in accordance with embodiments of the invention are illustrated. In one embodiment of the invention, a data processing system includes raw data storage containing raw data, metadata storage containing metadata that describes the raw data, and an interest-driven data pipeline that is automatically compiled to generate reporting data using the raw data, wherein the interest-driven data pipeline is compiled based upon reporting data requirements automatically derived from at least one report specification defined using the metadata.

Claims

1. A data processing system, comprising:

raw data storage containing raw data;

metadata storage containing metadata that describes the raw data; and

an interest-driven data pipeline that is automatically compiled to generate reporting data using the raw data;

wherein the interest-driven data pipeline is compiled based upon reporting data requirements automatically derived from at least one report specification defined using the metadata.

2. The data processing system of claim 1, wherein the interest-driven data pipeline is automatically compiled based upon at least one change selected from the group consisting of a change to the reporting data requirements, a change to the metadata, and a change to the raw data.

3. The data processing system of claim 1, further comprising:

source data storage configured to store source data; and

wherein the interest-driven data pipeline is configured to create source data by performing an extract, transform, and load (ETL) process on raw data using a source model.

6. The data processing system of claim 1, further comprising an intermediate processing layer, wherein the intermediate processing layer comprises:

a data manager configured to store data models; and

an interest-driven data pipeline compiler.

7. The data processing system of claim 6, wherein the intermediate processing layer accesses data within the raw data storage.

8. The data processing system of claim 6, wherein the interest-driven data pipeline is automatically compiled utilizing the interest-driven data pipeline compiler.

9. The data processing system of claim 6, wherein the intermediate processing layer is configured to perform raw data intake comprising updating raw data and updating reporting data.

10. The data processing system of claim 1, wherein:

the data processing system is configured as an interest-driven business intelligence system comprising;

a business intelligence reporting engine;

wherein the business intelligence reporting engine includes reporting data storage configured to store reporting data.

11. The interest-driven business intelligence system of claim 10, wherein the business intelligence reporting engine comprises a user interface configured to display data and receive requests for data.

12. The interest-driven business intelligence system of claim 10, further comprising:

report specification storage configured to store report specifications;

wherein the interest-driven business intelligence system is configured to receive at least one report specification.

13. The interest-driven business intelligence system of claim 12, wherein the interest-driven data pipeline is configured to automatically generate reporting data requirements using at least one report specification.

14. The interest-driven business intelligence system of claim 13, wherein the business intelligence reporting engine is configured to generate a report utilizing the report specification and the reporting data.

15. The interest-driven business intelligence system of claim 14 comprising an interest-driven data pipeline compiler, wherein the interest-driven data pipeline is configured to:

convert the raw data into source data utilizing an extract, transform, and load (ETL) process;

utilize the reporting data requirements to generate aggregation processes that are applied to the source data to generate aggregate data; and

generate a data model based upon the reporting data requirements and to populate the data model using the aggregate data to generate reporting data;

wherein the interest-driven data pipeline compiler is configured to automatically compile the interest-driven data pipeline in response to the interest-driven business intelligence system receiving a report specification modifying the reporting data requirements.

16. The interest-driven business intelligence system of claim 15, wherein the interest-driven data pipeline compiler is configured to update the ETL process in response to the modifications to the reporting data requirements.

17. The interest-driven business intelligence system of claim 15, wherein the interest-driven data pipeline compiler is configured to generate new aggregations in response to the modifications to the reporting data requirements.

18. The interest-driven business intelligence system of claim 15, wherein the interest-driven data pipeline compiler is configured to generate a new data model in response to the modifications to the reporting data requirements.

19. The interest-driven business intelligence system of claim 15, wherein the report comprises at least one visualization, where the at least one visualization is configured to process and display reporting data.

4. The data processing system of claim 4, further comprising:

aggregate data storage configured to store aggregate data; and

wherein the interest-driven data pipeline is configured to generate aggregate data by performing aggregations using the source data.

5. The data processing system of claim 4, wherein the interest-driven data pipeline is configured to generate aggregations utilizing the reporting data requirements.

20. An interest-driven business intelligence system, comprising:

raw data storage system configured to contain raw data and perform extract, transform, and load (ETL) processes;

metadata storage configured to contain metadata that describes the raw data;

aggregate data storage configured to contain aggregate data generated using ETL processes from the raw data;

an intermediate processing layer, where the intermediate processing layer is configured to automatically;

generate metadata describing the raw data;

derive reporting data requirements from the at least one report specification based on the metadata; and

compile an interest-driven data pipeline based upon the reporting data requirements, where automatically compiling the interest-driven data pipeline comprises;

generating ETL processing jobs to generate aggregate data from the raw data by;

filtering the raw data using the structure of the raw data; and

applying transformations to the raw data based on the structure of the raw data; and

generating reporting data including reporting data satisfying the reporting data requirements using the aggregate data; and

a business intelligence reporting engine configured to receive metadata describing the raw data from the intermediate processing layer and to generate a user interface enabling user exploration of the metadata to define at least one report specification, where the user exploration involves selection of reporting data using the metadata;

wherein the intermediate processing layer is configured to;

automatically update the reporting data requirements based upon the additional reporting data selected using the metadata via the business intelligence reporting engine;

automatically recompile the interest-driven data pipeline in real time to generate the additional reporting data selected using the metadata in response to the changes in the updated reporting data requirements by;

automatically generating ETL jobs to generate updated aggregate data from the raw data and providing the ETL jobs to the raw data storage; and

automatically generating updated reporting data contained in the interest-driven data pipeline using the updated aggregate data, where the updated reporting data includes reporting data that satisfies the updated reporting data requirements and the additional reporting data selected using the metadata; and

wherein the business intelligence reporting engine is configured to generate at least one report based upon the at least one report specification using the updated reporting data.

21. The interest-driven business intelligence system of claim 20, wherein the raw data storage system is a data warehouse.

22. The interest-driven business intelligence system of claim 21, wherein the data warehouse is implemented utilizing a system selected from the group consisting of a distributed computing system, a database management system, and a NoSQL database.

23. The interest-driven business intelligence system of claim 21, wherein the data warehouse is a distributed computing system implemented utilizing Hadoop.

24. The interest-driven business intelligence system of claim 21, wherein the data warehouse is configured to store data generated utilizing the intermediate processing layer.

25. The interest-driven business intelligence system of claim 21, wherein the intermediate processing layer is configured to generate data warehouse requests.

26. The interest-driven business intelligence system of claim 25, wherein the data warehouse requests are requests selected from the group consisting of Hive queries and MapReduce operations.

27. The interest-driven business intelligence system of claim 20, wherein the intermediate processing layer is implemented utilizing a system selected from the group consisting of a distributed computing system, a database management system, and a NoSQL database system.

28. The interest-driven business intelligence system of claim 20, wherein the business intelligence reporting engine is configured to display an indication based upon the interactive exploration of the at least one report.

29. The interest-driven business intelligence system of claim 28, wherein the indication is an estimate of the time needed to update the reporting data requirements.

30. A method for creating a report utilizing an interest-driven business intelligence system, comprising:

storing raw data in raw data storage using an interest-driven business intelligence system;

generating metadata describing the raw data using the interest-driven business intelligence system;

receiving report definitions generated utilizing the metadata describing the raw data using the interest-driven business intelligence system;

automatically generating reporting data requirements derived from report definitions using the interest-driven business intelligence system;

automatically compiling an interest-driven data pipeline that is part of the interest-driven business intelligence system using the reporting data requirements and the raw data;

generating reporting data from the reporting data requirements using the interest-driven data pipeline;

populating reports with the reporting data using the interest-driven business intelligence system;

providing interactive exploration of the reports using the interest-driven business intelligence system;

receiving updated reporting requirements based upon the interactive exploration of the reports using the interest-driven business intelligence system, where the updated reporting requirements utilize additional reporting data selected using the metadata describing the raw data; and

automatically recompiling the interest-driven data pipeline to generate additional reporting data using the interest-driven business intelligence system, where automatically recompiling the interest-drive data pipeline comprises;

automatically generating extract, transform, and load (ETL) jobs to generate updated aggregate data from the raw data using the interest-driven business intelligence system; and

automatically generating updated reporting data using the updated aggregate data and the interest-driven business intelligence system, where the updated reporting data includes reporting data that satisfies the updated reporting data requirements and the additional reporting data selected using the metadata.