OLAP Query Processing Method Oriented to Database and HADOOP Hybrid Platform

• US 20130282650A1
• Filed: 05/16/2012
• Published: 10/24/2013
• Est. Priority Date: 04/18/2012
• Status: Active Grant

• Current Assignee: Renmin University of China
• Original Assignee: Renmin University of China
• Inventors: Zhang, Yan-Song, Wang, Shan
• Granted Patent: US 9,501,550 B2
• Time in Patent Office: Days
• US Class Current: 707/605
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP

Abstract

An OLAP query processing method oriented to a database and Hadoop hybrid platform is described. When OLAP query processing is performed, the processing is executed first on a main working copy, and a query processing result is recorded in an aggregate result table of a local database; when a working node is faulty, node information of a fault-tolerant copy corresponding to the main working copy is searched for through namenode, and a MapReduce task is invoked to complete the OLAP query processing task on the fault-tolerant copy. The database technology and the Hadoop technology are combined, and the storage performance of the database and the high expandability and high availability of the Hadoop are combined; the database query processing and the MapReduce query processing are integrated in a loosely-coupled mode, thereby ensuring the high query processing performance, and ensuring the high fault-tolerance performance.

Claims

1. An on-line analytical processing (OLAP) query processing method oriented to a database and Hadoop hybrid platform, wherein:

on the basis of a multi-copy fault-tolerance mechanism of the Hadoop, a fact table is stored in a database cluster, a main working copy and at least one fault-tolerant copy of the fact table are set, the main working copy is imported into a local database of a working node, and a table corresponding to the main working copy is named according to a unified naming rule;

the main working copy in a Hadoop distributed file system is deleted, and meta-information of the main working copy in namenode is updated into a JDBC connection of the local database and a name of the table corresponding to the main working copy;

when OLAP query processing is performed, the processing is executed first on the main working copy, and a query processing result is recorded in an aggregate result table of the local database; and

when the working node is faulty, node information of the fault-tolerant copy corresponding to the main working copy is searched for through the namenode, and a MapReduce task is invoked to complete the OLAP query processing task on the fault-tolerant copy.

2. The OLAP query processing method oriented to a database and Hadoop hybrid platform according to claim 1, wherein:

the OLAP query processing method is applied in a hybrid platform having a dual-storage engine and a dual-OLAP query processing engine, the dual-storage engine comprises a database storage engine and a Hadoop storage engine, and the dual-OLAP query processing engine comprises a database OLAP engine and a MapReduce query processing engine.

3. The OLAP query processing method oriented to a database and Hadoop hybrid platform according to claim 1, wherein:

a parallel OLAP query processing technology is adopted for the main working copy of the local database; and

a MapReduce query processing technology is adopted for the fault-tolerant copy in the Hadoop distributed file system.

4. The OLAP query processing method oriented to a database and Hadoop hybrid platform according to claim 1, wherein:

if a node is faulty during the procedure of query processing, the database cluster completes the OLAP query processing task on some datasets, searches the namenode for a storage node of the fault-tolerant copy corresponding to the main working copy in the working node according to a number of the faulty node, and invokes the MapReduce task to complete the OLAP query processing task on the fault-tolerant copy; and

an OLAP query processing result of the database cluster and an OLAP query processing result of the MapReduce task are merged.

5. The OLAP query processing method oriented to a database and Hadoop hybrid platform according to claim 1, wherein:

when the OLAP query results of the working nodes are reduced, if no node is faulty during the procedure of inserting local aggregate results to reduce nodes, after all aggregate result records are inserted to the reduce nodes, a group-by operation is executed to perform global aggregate; and

aggregate result records in the reduce nodes are loaded into a designated reduce node to perform a final aggregate result reduce operation, and an OLAP query processing result is returned.

6. The OLAP query processing method oriented to a database and Hadoop hybrid platform according to claim 1, wherein:

when the OLAP query results of the working nodes are reduced, if a node is faulty during the procedure of inserting local aggregate results to reduce nodes, after the insertion work of aggregate results of the reduce nodes is completed, the aggregate result record inserted by the faulty node is filtered out when a group-by operation is executed according to a number of the faulty node, so as to obtain an aggregate reduce result of a part of the nodes; and

after the faulty node is restored or the OLAP query processing corresponding to the faulty node copy is redone by another fault-tolerant node, the aggregate result record generated on the copy influenced by the faulty node is inserted into an aggregate result table of the reduce node, and global merge is performed on aggregate result subsets of the two OLAP query processing stages by executing the group-by operation again.

7. The OLAP query processing method oriented to a database and Hadoop hybrid platform according to claim 1, wherein:

dimension tables are stored in a central node in a centralized way, and fact tables are distributed on the working nodes according to a data distribution policy of the Hadoop distributed file system, so as to form an inverse star-schema storage structure with the dimension tables as a center.

8. The OLAP query processing method oriented to a database and Hadoop hybrid platform according to claim 7, wherein:

a predicate in an SQL statement is applied on the dimension table and a corresponding group-by attribute vector is selected, data in the group-by attribute vector is coded by bit, and a position corresponding to a dimension table record in the predicate vector and satisfying a predicate condition is filled with a code;

when the SQL statement is executed, a predicate operation is performed on the central node, a coded predicate vector is generated for each corresponding dimension table, stored in a primary vector cache, and synchronized to a secondary vector cache of each working node through a broadcasting mechanism, and each working node generates a local aggregate calculation result based on the coded predicate vector, wherein the group-by attribute is replaced with a code; and

after the global aggregate result reduce is completed, the group-by attribute code in the aggregate result is replaced with an original group-by attribute value at the central node according to a predicate result code table, and a final query processing result is returned to a user.

9. The OLAP query processing method oriented to a database and Hadoop hybrid platform according to claim 7, wherein:

when many concurrent queries and few group-by attribute updates of the dimension table exist, a query filter required by a join operation is minimized into a bitmap, and a dimension table group-by attribute required by the group-by operation is cached in a cache of the working node.

10. The OLAP query processing method oriented to a database and Hadoop hybrid platform according to claim 1, wherein:

an on-line transaction processing (OLTP) system is deployed on the central node, fact table data in the OLTP system is periodically loaded into a Hadoop OLAP system, and the OLTP system stores real-time updated data;

when executing the OLAP query processing, the Hadoop OLAP system first generates a predicate vector in the OLTP system, broadcasts the predicate vector to a Hadoop database cluster through a primary-secondary cache to perform parallel OLAP processing, at the same time, performs the OLAP processing on latest data of the OLTP system in a centralized way, reduces OLAP aggregate calculation results of the OLTP system and the Hadoop OLAP system, and returns to a user an OLAP query processing result on the real-time data.