Systems and methods for interest-driven business intelligence systems including event-oriented data

• US 9,892,178 B2
• Filed: 03/05/2014
• Issued: 02/13/2018
• Est. Priority Date: 09/19/2013
• Status: Active Grant

• Current Assignee: Workday Incorporated
• Original Assignee: Workday Incorporated
• Inventors: Schuster, John, Werther, Benjamin Mark, Babcock, Brian, Beyer, Kevin Scott, Brown, Hans-Frederick
• Primary Examiner(s): Boccio, Vincent F
• Application Number:
  US14/198,039
• Publication Number:
  US 20150081618A1
• Time in Patent Office: 1,441 Days
• Field of Search: 707602
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06F 16/29 Geographical information databases

Abstract

Systems and methods for interest-driven business intelligence systems including event-oriented data in accordance with embodiments of the invention are illustrated. In one embodiment, an interest-driven business intelligence system includes raw data storage configured to contain raw data and perform ETL processes, a data mart configured to contain metadata that describes the raw data, and an intermediate processing layer, wherein the intermediate processing layer is configured to compile an interest-driven data pipeline configured to generate ETL processing jobs to generate event-oriented data from the raw data by filtering the raw data based on the metadata describing the raw data, determining ordering data based on the metadata describing the raw data, aligning the filtered raw data based on the ordering data, and generating event-oriented data based on the aligned filtered raw data, and storing the event-oriented data in the data mart.

Claims

1. An interest-driven business intelligence system for processing business data, comprising:

raw data storage comprising at least one hardware processor and at least one memory, containing raw data and capable of executing extract, transform, and load (ETL) processes; and

an intermediate processing layer comprising at least one hardware processor, at least one memory, and a data mart containing metadata that describes the raw data;

wherein the intermediate processing layer is configured to automatically;

generate metadata describing the raw data;

store the generated metadata using the data mart;

derive reporting data requirements from at least one report specification based on the metadata; and

compile an interest-driven data pipeline based upon the reporting data requirements, wherein compiling the interest-driven data pipeline comprises;

generating ETL processing jobs to generate event-oriented data from the raw data by;

obtaining raw data from a plurality of fact tables stored using the raw data storage;

filtering the raw data based on the metadata describing the raw data;

determining ordering data based on the metadata describing the raw data, wherein the ordering data comprises at least one dimension described in the plurality of fact tables;

aligning, based on the ordering data, the filtered raw data to generate event-oriented data, wherein the aligned filtered raw data comprises data taken from the plurality of fact tables and ordered based on at least one dimension described in the ordering;

data, wherein the aligning of the filtered raw data comprises;

 

mapping the ordering data to the filtered raw data, wherein the ordering data is mapped to the at least one dimension of the filtered raw data; and

 

resolving differences between the ordering data and the metadata to obtain the event-oriented data;

storing the event-oriented data in the data mart;

generating event-oriented metadata describing the event-oriented data; and

storing the event-oriented metadata in the data mart;

generating reporting data including data satisfying the reporting data requirements based on the event-oriented data; and

storing the reporting data in the data mart for exploration by an interest-driven data visualization system; and

update the event-oriented data based on a request received from the interest-driven data visualization system, wherein updating the event-oriented data comprises determining redundancies between the event-oriented data and updated event-oriented data, wherein determining the redundancies is based at least in part on the event-oriented metadata.

2. The interest-driven business intelligence system of claim 1, wherein the raw data comprises unstructured data.

3. The interest-driven business intelligence system of claim 1, wherein the raw data storage is a data warehouse.

4. The interest-driven business intelligence system of claim 3, wherein the data warehouse is implemented utilizing a system selected from the group consisting of a distributed computing system, a database management system, and a NoSQL database.

5. The interest-driven business intelligence system of claim 3, wherein the data warehouse is a distributed computing system implemented utilizing Hadoop.

6. The interest-driven business intelligence system of claim 3, wherein the data warehouse is configured to store data generated utilizing the intermediate processing layer.

7. The interest-driven business intelligence system of claim 3, wherein the intermediate processing layer is further configured to generate data warehouse requests.

8. The interest-driven business intelligence system of claim 7, wherein the data warehouse requests comprise MapReduce operations.

9. The interest-driven business intelligence system of claim 1, wherein the data mart is implemented utilizing a system selected from the group consisting of a distributed computing system, a database management system, and a NoSQL database system.

10. The interest-driven business intelligence system of claim 1, further comprising an interest-driven data visualization system;

wherein the interest-driven data visualization system is configured to;

receive metadata describing the raw data from the intermediate processing layer; and

generate a user interface enabling user exploration of the metadata to define at least one report specification, wherein the user exploration involves selection of additional reporting data based on the metadata.

11. The interest-driven business intelligence system of claim 10, wherein the interest-driven data visualization system is further configured to display an indication based upon interactive exploration of the at least one report.

12. The interest-driven business intelligence system of claim 1, wherein the data mart is configured to contain aggregate data, wherein the aggregate data comprises structured data generated using ETL processes from the raw data.

13. The interest-driven business intelligence system of claim 12, wherein compiling the interest-driven pipeline further comprises generating ETL processing jobs to generate aggregate data from the raw data by:

filtering the raw data based on the metadata describing the raw data;

applying transformations to the raw data based on the metadata describing the raw data;

generating aggregate data based on the transformed data; and

storing the aggregate data in the data mart.

14. The interest-driven business intelligence system of claim 13, wherein compiling the interest-driven pipeline further comprises generating ETL processing jobs to generate aggregate data from event-oriented data by:

identifying at least one dimension within a piece of event-oriented data;

obtaining raw data corresponding to the identified at least one dimension;

applying transformations to the obtained raw data based on the metadata describing the obtained raw data;

generating aggregate data based on the transformed data; and

storing the aggregate data in the data mart.

15. The interest-driven business intelligence system of claim 13, wherein compiling the interest-driven pipeline further comprises generating ETL processing jobs to generate event-oriented data from aggregate data by:

identifying at least one dimension within a piece of aggregate data;

obtaining raw data corresponding to the identified at least one dimension;

filtering the obtained raw data based on the metadata describing the obtained raw data;

determining ordering data based on the metadata describing the obtained raw data;

aligning the filtered obtained raw data based on the ordering data;

generating event-oriented data based on the aligned data; and

storing the event-oriented data in the data mart.

16. The interest-driven business intelligence system of claim 1, wherein the ordering data comprises at least one dimension identified in the metadata describing the raw data.

17. The interest-driven business intelligence system of claim 16, wherein aligning the filtered raw data based on the ordering data comprises sorting the event-oriented data according to the facts associated with the ordering data in each piece of event-oriented data.

18. The interest-driven business intelligence system of claim 17, wherein:

aligning the filtered raw data based on the ordering data further comprises mapping at least two dimensions within the filtered raw data to a common set of data; and

the common set of data is based on the ordering data.

19. The interest-driven business intelligence system of claim 1, wherein the ordering data comprises time data.

20. The interest-driven business intelligence system of claim 1, wherein the business data comprises data describing interactions of users with websites.

21. A method for creating a report utilizing an interest-driven business intelligence system for processing business data, comprising:

storing raw data in a raw data storage;

automatically generating metadata describing the raw data;

automatically storing the generated metadata in a data mart;

automatically deriving reporting data requirements from at least one report specification based on the metadata;

automatically compiling an interest-driven data pipeline based upon the reporting data requirements, wherein compiling the interest-driven data pipeline comprises;

generating ETL processing jobs to generate event-oriented data from the raw data by;

obtaining raw data from a plurality of fact tables stored using the raw data storage;

filtering the raw data based on the metadata describing the raw data;

determining ordering data based on the metadata describing the raw data, wherein the ordering data comprises at least one dimension described in the plurality of fact tables;

aligning, based on the ordering data, the filtered raw data to generate event-oriented data, wherein the aligned filtered raw data comprises data taken from the plurality of fact tables and ordered based on at least one dimension described in the ordering data, wherein the aligning of the filtered raw data comprises;

mapping the ordering data to the filtered raw data, wherein the ordering data is mapped to the at least one dimension of the filtered raw data; and

resolving differences between the ordering data and the metadata to obtain the event-oriented data;

storing the event-oriented data in the data mart;

generating event-oriented metadata describing the event-oriented data; and

storing the event-oriented metadata in the data mart;

generating reporting data including data satisfying the reporting data requirements based on the event-oriented data; and

storing the reporting data in the data mart for exploration by an interest-driven data visualization system; and

automatically updating the event-oriented data based on a request received from the interest-driven data visualization system, wherein updating the event-oriented data comprises determining redundancies between the event-oriented data and updated event-oriented data, wherein determining the redundancies is based at least in part on the event-oriented metadata.

22. The method of claim 21, wherein compiling the interest-driven pipeline further comprises generating ETL processing jobs to generate aggregate data from event-oriented data by:

identifying at least one dimension within a piece of event-oriented data using the interest-driven business intelligence system;

obtaining raw data corresponding to the identified at least one dimension using the interest-driven business intelligence system;

applying transformations to the obtained raw data based on the metadata describing the obtained raw data to generate aggregate data using the interest-driven business intelligence system; and

storing the aggregate data using the interest-driven business intelligence system.

23. The method of claim 22, wherein generating ETL processing jobs to generate aggregate data from the raw data further comprises:

filtering the raw data based on the metadata describing the raw data using the interest- driven business intelligence system;

applying transformations to the raw data based on the metadata describing the raw data to generate aggregate data using the interest-driven business intelligence system; and

storing the aggregate data using the interest-driven business intelligence system.

24. The method of claim 22, wherein compiling the interest-driven pipeline further comprises generating ETL processing jobs to generate event-oriented data from aggregate data by:

identifying at least one dimension within a piece of aggregate data using the interest- driven business intelligence system;

obtaining raw data corresponding to the identified at least one dimension using the interest-driven business intelligence system;

filtering the obtained raw data based on the metadata describing the obtained raw data using the interest-driven business intelligence system;

determining ordering data based on the metadata describing the obtained raw data using the interest-driven business intelligence system;

aligning the filtered obtained raw data based on the ordering data to generate event- oriented data using the interest-driven business intelligence system; and

storing the event-oriented data using the interest-driven business intelligence system.

25. The method of claim 24, wherein the ordering data comprises time data.