HADOOP OLAP ENGINE

• US 20190324975A1
• Filed: 05/22/2019
• Published: 10/24/2019
• Est. Priority Date: 04/24/2014
• Status: Active Grant

• Current Assignee: eBay Inc.
• Original Assignee: eBay Inc.
• Inventors: HAN, Luke Qing, JIANG, Xu, YI, Song, LI, Chauncey
• Granted Patent: US 11,537,635 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/185 Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof details of archiving G06F16/11
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP

Abstract

In various example embodiments, systems and methods for building data cubes to be stored in a cube store are presented. In some embodiments, a metadata engine generates the cube metadata. In further embodiments, cube data is generated by a cube build engine based on the cube metadata and source data. The cube build engine performs a multi-stage MapReduce job on the source data to produce a multi-dimensional cube lattice having multiple cuboids. In further embodiments, the cube data is provided to the cube store.

Claims

1-20. -20. (canceled)

21. A method comprising:

receiving cube metadata defining dimensions and measure information for a cube;

building the cube based on the cube metadata and source data;

storing the cube in a cube store, the cube store configured for access by a query engine;

partitioning the cube into a first data partition and a second data partition; and

refreshing the first data partition by periodically merging cube data in the second data partition with cube data in the first data partition.

22. The method of claim 21, further comprising:

accessing query history; and

automatically identifying the cube metadata based on the query history.

23. The method of claim 21, wherein the data cube serves as a cache.

24. The method of claim 21, wherein building the cube based on the cube metadata and the source data comprises transforming source Hadoop Distributed File System (HDFS) files to HFiles for storage in the cube store.

25. The method of claim 21, further comprising creating one or more HBase tables based on the cube metadata.

26. The method of claim 21, wherein the cube is an Online Analytical Processing (OLAP) cube.

27. The method of claim 21, wherein the measure information includes any one of a sum, a count, a max, or a min function.

28. The method of claim 21, wherein the cube store comprises a column-oriented database.

29. The method of claim 21, wherein the cube data is stored in a distributed database, and wherein the query engine is configured to query the cube data using Structured Query Language (SQL) queries.

30. A machine readable medium not having any transitory signals and storing instructions that, when executed by at least one processor of a machine, cause the machine to perform operations comprising:

receiving cube metadata defining dimensions and measure information for a cube;

building the cube based on the cube metadata and source data;

storing the cube in a cube store, the cube store configured for access by a query engine;

partitioning the cube into a first data partition and a second data partition; and

refreshing the first data partition by periodically merging cube data in the second data partition with cube data in the first data partition.

31. The machine readable medium of claim 30, further comprising additional instructions that, when executed by the at least one processor of the machine, cause the machine to perform operations comprising:

accessing query history; and

automatically identifying the cube metadata based on the query history.

32. The machine readable medium of claim 30, wherein building the cube based on the cube metadata and the source data comprises transforming source Hadoop Distributed File System (HDFS) files to HFiles for storage in the cube store.

33. The machine readable medium of claim 30, further comprising additional instructions that, when executed by the at least one processor of the machine, cause the machine to perform operations comprising creating one or more HBase tables based on the cube metadata.

34. The machine readable medium of claim 30, wherein the cube store comprises a column-oriented database.

35. The machine readable medium of claim 30, wherein the cube data is stored in a distributed database, and wherein the query engine is configured to query the cube data using Structured Query Language (SQL) queries.

36. A computer system, comprising:

at least one processor; and

a machine readable medium not having any transitory signals and storing instructions that, when executed by the at least one processor, cause the computer system to perform operations comprising;

receiving cube metadata defining dimensions and measure information for a cube;

building the cube based on the cube metadata and source data;

storing the cube in a cube store, the cube store configured for access by a query engine;

partitioning the cube into a first data partition and a second data partition; and

refreshing the first data partition by periodically merging cube data in the second data partition with cube data in the first data partition.

37. The computer system of claim 36, wherein the machine readable medium stores additional instructions that, when executed by the at least one processor, cause the computer system to perform operations comprising:

accessing query history; and

automatically identifying the cube metadata based on the query history.

38. The computer system of claim 36, wherein the cube is an Online Analytical Processing (OLAP) cube.

39. The computer system of claim 36, wherein building the cube based on the cube metadata and the source data comprises transforming source Hadoop Distributed File System (HDFS) files to HFiles for storage in the cube store.

40. The computer system of claim 36, wherein the measure information comprises any one of a sum, a count, a max, or a min function.