Split processing paths for a database calculation engine

• US 8,868,594 B2
• Filed: 12/23/2011
• Issued: 10/21/2014
• Est. Priority Date: 12/23/2011
• Status: Active Grant

• Current Assignee: SAP SE
• Original Assignee: SAP AG (SAP SE)
• Inventors: Baeumges, Daniel, Bensberg, Christian, Fricke, Lars
• Primary Examiner(s): OWYANG, MICHELLE N
• Application Number:
  US13/336,935
• Publication Number:
  US 20130166589A1
• Time in Patent Office: 1,033 Days
• Field of Search: 707/713, 707736-737, 707769-770
• US Class Current: 707/769
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/24532 of parallel queries
• CPC Class Codes: G06F 16/24535 of sub-queries or views
• CPC Class Codes: G06F 16/24542 Plan optimisation
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/24554 Unary operations; Data partitioning operations
• CPC Class Codes: G06F 16/2457 with adaptation to user needs

Abstract

A dynamic split node defined within a calculation model can receive data being operated on by a calculation plan generated based on the calculation model. A partition specification can be applied to one or more reference columns in a table containing at least some of the received data. The applying can cause the table to be split such that a plurality of records in the table are partitioned according to the partition specification. A separate processing path can be set for each partition, and execution of the calculation plan can continue using the separate processing paths, each of which can be assigned to a processing node of a plurality of available processing nodes.

Claims

1. A computer program product comprising a non-transitory machine-readable storage medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:

receiving, at a dynamic split node defined within a calculation model of a database management system in a multi-node data partitioning landscape that comprises a master node and a plurality of processing nodes, data as an intermediate result of an earlier operation in a calculation plan or data from one or more tables upon which the calculation plan is currently operating in response to a query, wherein the plurality of processing nodes are being controlled by the master node, wherein the calculation plan is generated by the calculation model with sequence of execution operations tailored to the query;

applying, at the dynamic split node, a partitioning specification to one or more reference columns in a table that includes a plurality of records containing the received data;

splitting, at the dynamic split node, the table for handling by a plurality of processing paths according to the partitioning specification, wherein the partitioning specification comprises examining the one or more reference columns, quantifying a number of distinct values in the one or more reference columns, and splitting the table into a number of two or more partitions based on the quantified number of distinct values, wherein the number of two or more partitions for the table is equal to the quantified number of distinct values, wherein each distinct value has a corresponding partition, wherein the partitioning specification further comprises placing the plurality of records into respective partitions such that records having same distinct value are placed together in a partition;

setting, at the dynamic split node, a separate processing path of the plurality of processing paths for a respective partition of the table, wherein the separate processing path is assigned to a respective processing node of the plurality of processing nodes to generate one or more intermediate results used in a following execution operation of the calculation plan; and

continuing execution of the following execution operation of the calculation plan using the plurality of processing paths to union the generated one or more intermediate results from each of the plurality of processing nodes and generate a union result, wherein the union result is used as an intermediate result by additional execution operations of the calculation plan if additional operations are required by the calculation plan, and wherein the union result is returned as a final result for the query if no additional execution operations are required by the calculation plan.

2. A computer program product as in claim 1, wherein the operations further comprising:

initiating the query;

receiving, by a recipient node of the plurality of processing nodes each comprising at least one data partition of the plurality of data partitions, a data request in response to the initiated query, wherein the data request being received directly from a requesting machine without being handled by the master node;

identifying, by the recipient node, a target node of the plurality of processing nodes to handle the data request, the determining comprising the recipient node applying partitioning information to determine one partition of the plurality of partitions to which the data request should be directed and mapping information associating each data partition of the plurality of data partitions with an assigned node of the plurality of processing nodes; and

redirecting, by the recipient node, the data request to the target node, wherein the target node acts on the one partition in response to the data request to obtain at least a portion of the received data specified by the calculation plan.

3. A computer program product as in claim 2, wherein the operations further comprising:

accessing the partitioning information and the mapping information from at least one of a local storage accessible to the recipient node and a metadata repository accessible to each of the plurality of processing nodes.

4. A system comprising:

at least one programmable processor; and

a machine-readable medium storing instructions that, when executed by the at least one processor, cause the at least one programmable processor to perform operations comprising;

receiving, at a dynamic split node defined within a calculation model of a database management system in a multi-node data partitioning landscape that comprises a master node and a plurality of processing nodes, data as an intermediate result of an earlier operation in a calculation plan or data from one or more tables upon which the calculation plan is currently operating in response to a query, wherein the plurality of processing nodes are being controlled by the master node, wherein the calculation plan is generated by the calculation model with sequence of execution operations tailored to the query;

applying, at the dynamic split node, a partitioning specification to one or more reference columns in a table that includes a plurality of records containing the received data;

splitting the table for handling by a plurality of processing paths according to the partitioning specification, wherein the partitioning specification comprises examining the one or more reference columns, quantifying a number of distinct values in the one or more reference columns, and splitting the table into a number of two or more partitions based on the quantified number of distinct values, wherein the number of two or more partitions for the table is equal to the quantified number of distinct values, wherein each distinct value has a corresponding partition, wherein the partitioning specification further comprises placing the plurality of records into respective partitions such that records having same distinct value are placed together in a partition;

setting, at the dynamic split node, a separate processing path of the plurality of processing paths for a respective partition of the table, wherein the separate processing path is assigned to a respective processing node of the plurality of processing nodes to generate one or more intermediate results used in a following execution operation of the calculation plan; and

continuing execution of the following execution operation of the calculation plan using the plurality of processing paths to union the generated one or more intermediate results from each of the plurality of processing nodes and generate a union result, wherein the union result is used as an intermediate result by additional execution operations of the calculation plan if additional operations are required by the calculation plan, and wherein the union result is returned as a final result for the query if no additional execution operations are required by the calculation plan.

5. A system as in claim 4, wherein the operations further comprising:

initiating the query;

receiving, by a recipient node of the plurality of processing nodes each comprising at least one data partition of the plurality of data partitions, a data request in response to the initiated query, wherein the data request being received directly from a requesting machine without being handled by the master node;

identifying, by the recipient node, a target node of the plurality of processing nodes to handle the data request, the determining comprising the recipient node applying partitioning information to determine one partition of the plurality of partitions to which the data request should be directed and mapping information associating each data partition of the plurality of data partitions with an assigned node of the plurality of processing nodes; and

redirecting, by the recipient node, the data request to the target node, wherein the target node acts on the one partition in response to the data request to obtain at least a portion of the received data specified by the calculation plan.

6. A system as in claim 5, wherein the operations further comprising:

accessing the partitioning information and the mapping information from at least one of a local storage accessible to the recipient node and a metadata repository accessible to each of the plurality of processing nodes.

7. A computer-implemented method comprising:

receiving, at a dynamic split node defined within a calculation model of a database management system in a multi-node data partitioning landscape that comprises a master node and a plurality of processing nodes, data as an intermediate result of an earlier operation in a calculation plan or data from one or more tables upon which the calculation plan is currently operating in response to a query, wherein the plurality of processing nodes are being controlled by the master node, wherein the calculation plan is generated by the calculation model with sequence of execution operations tailored to the query;

applying, at the dynamic split node, a partitioning specification to one or more reference columns in a table that includes a plurality of records containing the received data;

splitting the table for handling by a plurality of processing paths according to the partitioning specification wherein the partitioning specification comprises examining the one or more reference columns, quantifying a number of distinct values in the one or more reference columns, and splitting the table into a number of two or more partitions based on the quantified number of distinct values, wherein the number of two or more partitions for the table is equal to the quantified number of distinct values, wherein each distinct value has a corresponding partition, wherein the partitioning specification further comprises placing the plurality of records into respective partitions such that records having same distinct value are placed together in a partition;

setting, at the dynamic split node, a separate processing path of the plurality of processing paths for a respective partition of the table, wherein the separate processing path is assigned to a respective processing node of the plurality of processing nodes to generate one or more intermediate results used in a following execution operation of the calculation plan; and

continuing execution of the following execution operation of the calculation plan using the plurality of processing paths to union the generated one or more intermediate results from each of the plurality of processing nodes and generate a union result, wherein the union result is used as an intermediate result by additional execution operations of the calculation plan if additional operations are required by the calculation plan, and wherein the union result is returned as a final result for the query if no additional execution operations are required by the calculation plan.

8. A computer-implemented method as in claim 7, wherein the operations further comprising:

initiating the query;

receiving, by a recipient node of the plurality of processing nodes each comprising at least one data partition of the plurality of data partitions, a data request in response to the initiated query, wherein the data request being received directly from a requesting machine without being handled by the master node;

identifying, by the recipient node, a target node of the plurality of processing nodes to handle the data request, the determining comprising the recipient node applying partitioning information to determine one partition of the plurality of partitions to which the data request should be directed and mapping information associating each data partition of the plurality of data partitions with an assigned node of the plurality of processing nodes; and

redirecting, by the recipient node, the data request to the target node, wherein the target node acts on the one partition in response to the data request to obtain at least a portion of the received data specified by the calculation plan.

9. A computer-implemented method as in claim 8, further comprising:

accessing the partitioning information and the mapping information from at least one of a local storage accessible to the recipient node and a metadata repository accessible to each of the plurality of processing nodes.

10. A computer-implemented method as in claim 7, wherein at least one of the receiving, the applying, the splitting, the setting, and the continuing is performed by at least one programmable processor.