Generating code for an integrated data system

• US 9,727,604 B2
• Filed: 03/10/2006
• Issued: 08/08/2017
• Est. Priority Date: 03/10/2006
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Xu, Lin, Padmanabhan, Sriram K., Jin, Qi, Liao, Hui
• Primary Examiner(s): PARK, GRACE A
• Application Number:
  US11/372,540
• Publication Number:
  US 20070214111A1
• Time in Patent Office: 4,169 Days
• Field of Search: 707802, 707809, 707810, 717104, 717106, 717144, 717147, 717155
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database

Abstract

A computer implemented method for generating code for an integrated data system. A mixed data flow is received. The mixed data flow contains mixed data flow operators, which are associated with multiple runtime environments. A graph is generated containing logical operators based on the mixed data flow in response to receiving the mixed data flow. The logical operators are independent of the plurality of runtime environments. The graph is converted to a model. The logical operators are converted to model operators associated with the multiple runtime environments. The model operators allow for analysis of operations for the mixed data flow. The model is converted into an execution plan graph. The execution plan graph is executable on different runtime environments.

Claims

1. A computer implemented method for generating code for an integrated data system, the computer-implemented method comprising:

receiving a mixed data flow containing mixed data flow operators, the mixed data flow operators collectively defining operations to be performed to complete the mixed data flow, wherein a first of a plurality of runtime environments of distinct types is selected to perform a given one of the defined operations, wherein the given operation is dependent on at least one other operation performed in a second of the plurality of runtime environments;

generating, based on the mixed data flow, a logical operator graph containing logical operators independent of the plurality of runtime environments;

converting the logical operator graph to an extended query graph model in which the logical operators are converted to model operators associated with the plurality of runtime environments;

analyzing the extended query graph model in order to pre-optimize code generation to include at least one of chunking and execution parallelism;

subsequent to analyzing the extended query graph model, converting the extended query graph model via code generation by operation of one or more computer processors into an execution plan graph executable on the plurality of different types of runtime environments; and

executing the execution plan graph by an execution engine that invokes a plurality of runtime engines, each runtime engine being of a distinct runtime engine type corresponding to a respective one of the plurality of runtime environments.

2. The computer implemented method of claim 1, wherein the plurality of runtime engines includes any of an extract, transform, load engine, a DataStage engine, and structured query language engine.

3. The computer implemented method of claim 1, wherein the graph operators are logical operator graph operators, wherein the model operators are extended query graph model operators, wherein converting the logical operator graph to the extended query graph model comprises:

mapping the logical operator graph operators to the extended query graph model operators; and

transforming relationships between respective operations of the logical operator graph operators to relationships between respective operations of the extended query graph model operators.

4. The computer implemented method of claim 1, wherein converting the logical operator graph to the extended query graph model comprises:

converting a logical operator graph operation directly to an extended query graph model quantifier.

5. The computer implemented method of claim 1, wherein converting the logical operator graph to the extended query graph model comprises:

mapping properties of a logical operator graph operation to properties of an extended query graph model entity.

6. The computer implemented method of claim 1, wherein converting the logical operator graph to the extended query graph model comprises:

mapping a logical operator graph operation to a property of an extended query graph model operator.

7. The computer implemented method of claim 1, wherein converting the logical operator graph to the extended query graph model comprises:

transforming a logical operator graph operation to any of a set of table functions, and stored procedures to invoke an executable program.

8. The computer implemented method of claim 1, wherein converting the logical operator graph to the extended query graph model comprises:

converting an expression in the logical operator graph to an expression tree in the extended query graph model.

9. The computer implemented method of claim 1, comprising:

performing analysis and optimization of the logical operator graph, the extended query graph model, and the execution plan graph, respectively.

10. The computer implemented method of claim 1, wherein the extended query graph model includes structured query language operations, executable operations, and custom operations.

11. The computer implemented method of claim 1, wherein the logical operator graph is a metadata representation of the mixed data flow.

12. The computer implemented method of claim 1, where the computer-implemented method is to generate the execution plan graph from the mixed data flow and for execution on the plurality of different types of runtime environments programmatically selected as satisfying a set of predefined criteria;

wherein the extended query graph model in analyzed in order to pre-optimize code generation to include both chunking and execution parallelism;

wherein chunking comprises breaking one subset of the mixed data flow into multiple units in order to improve execution efficiency, wherein execution parallelism comprises grouping disparate sets of operations within the mixed data flow and executing the disparate sets in parallel in order to further improve execution efficiency;

wherein the model operators allow for analysis of operations for the mixed data flow, wherein the mixed data flow is received from a user, wherein the operation is selected based on user input, wherein a processing application is programmatically selected, wherein the processing application and first runtime environment are not selected based on any user input;

wherein the plurality of different types of runtime environments are programmatically selected as satisfying the set of predefined criteria;

wherein the processing application is programmatically determined to satisfy a predefined suitability condition, wherein the suitability condition is satisfied upon identifying a matching runtime environment;

wherein the mixed data flow consists of a plurality of data flows specified in a single request from a user, wherein each data flow in a plurality of data flows is of a distinct data type;

wherein the computer-implemented method further comprises;

outputting an indication that the execution plan graph was executed.

13. The computer implemented method of claim 12, wherein the operations to be performed to complete the mixed data flow as defined by the mixed data flow operators include:

(i) a first predefined operation comprising extracting data from one or more files;

(ii) a second predefined operation comprising extracting data from one or more tables;

(iii) a third predefined operation comprising filtering said data;

(iv) a fourth predefined operation comprising joining data extracted from one or more tables with data extracted from one or more files;

(v) a fifth predefined operation comprising removing duplicate data;

(vi) a sixth predefined operation comprising saving data in a file; and

(vii) a seventh predefined operation comprising loading data onto a target.

14. The computer implemented method of claim 13, wherein converting the logical operator graph to the extended query graph model comprises:

(i) converting a logical operator graph operation directly to an extended query graph model quantifier;

(ii) mapping properties of a logical operator graph operation to properties of an extended query graph model entity;

(iii) mapping a logical operator graph operation to a property of an extended query graph model operator;

(iv) transforming a logical operator graph operation to any of a set of table functions, and stored procedures to invoke an executable program; and

(v) converting an expression in the logical operator graph to an expression tree in the extended query graph model.

15. The computer implemented method of claim 14, wherein the graph operators are logical operator graph operators, and wherein the model operators are extended query graph model operators;

wherein the one or more runtime engines is any of an extract, transform, load engine, a DataStage engine, and structured query language engine, wherein the converting the graph step further comprises;

(i) mapping the logical operator graph operators to the extended query graph model operators; and

(ii) transforming relationships between respective operations of the logical operator graph operators to relationships between respective operations of the extended query graph model operators;

wherein the computer-implemented method further comprises;

performing analysis and optimization of the logical operator graph, extended query graph model, and the execution plan graph;

wherein the extended query graph model includes structured query language operations, executable operations, and custom operations, wherein the logical operator graph is a metadata representation of the mixed data flow.

16. The computer-implemented method of claim 15, further comprising performing a first operation to dilate a sub-flow operator in a first data flow comprising the mixed data flow, the first operation comprising:

traversing the first data flow to locate sub-flow operators, wherein each sub-flow operator is associated with a corresponding sub-flow, each sub-flow itself comprises a set of one or more operators, wherein each sub-flow operator is bound to other operators in the first data flow through at least one input port of the sub-flow operator and at least one output port of the sub-flow operator;

for each located sub-flow operator, retrieving the sub-flow corresponding to the sub-flow operator;

injecting the corresponding sub-flows into the first data flow, wherein injecting each sub-flow comprises;

copying each operator in the sub-flow,

renaming each operator in the sub-flow, to form renamed operators, and

inserting the renamed operators into the first data flow, thereby replacing the sub-flow operator in the first data flow;

bridging at least the one input port and the one output port of the sub-flow operator to ports of operators in the first data flow; and

sending the first data flow for code generation, wherein the code generation comprises generating application source code representing the first data flow;

wherein traversing the first data flow to locate sub-flow operators comprises;

reading an operator;

determining if the operator is a sub-flow operator; and

repeating the reading and determining steps until all the operators in the data flow are read;

wherein the first operation further comprises;

in response to a determination that at least one sub-flow operator was encountered, repeating the traversing, retrieving, and injecting steps.

17. The computer-implemented method of claim 16, further comprising performing a second operation to process the mixed data flow, the second operation comprising:

classifying the mixed data flow operators according to operator type, wherein classifying mixed data flow operators according to operator type comprises separating the mixed data flow operators in the mixed data flow based on the respective runtime environment associated with each of the mixed data flow operators;

segregating the mixed data flow operators into regions, each region being a data flow that contains operators of the same operator type, such that each mixed data flow operator in a particular region is executable by one of the plurality of runtime environments associated with the particular region; and

sequencing the execution of the regions in a manner that ensures satisfaction of dependency between the mixed data flow operators.

18. The computer implemented method of claim 17, wherein classifying the mixed data flow operators further comprises:

creating a queue;

maintaining a table of operator type and current sequence for each mixed data flow operator in the mixed data flow;

retrieving a list of mixed data flow operators in the data flow;

responsive to determining that the mixed data flow operator has outgoing links, looking up the operator type in the table and assigning the mixed data flow operator a sequence and adding the mixed data flow operator to the queue;

responsive to mixed data flow operators being in the queue, assigning the mixed data flow operator a sequence;

wherein the second operation further comprises;

responsive to determining the mixed data flow operator does not have outgoing links, performing the assigning step;

wherein the assigning the mixed data flow operator further comprises;

setting a next operator in the queue as a parent operator;

responsive to determining the parent operator has a child operator, determining whether the operator type of the parent operator is the same as the operator type of the child operator;

responsive to determining that the operator type of the parent operator is not the same as the operator type of the child operator and responsive to the sequence of the child operator being less than the sequence of the parent operator, setting the sequence of the child operator equal to the sequence of the parent operator;

adding the child operator to the queue.

19. The computer implemented method of claim 18, further comprising:

responsive to determining that the operator type of the parent operator is not the same as the operator type of the child operator, setting the sequence of the child operator equal to the a current order for the operator type of the child operator; and

responsive to determining the sequence of the child operator is less than the sequence of the parent, retrieving a next sequence for the child;

wherein the retrieving the next sequence for the child comprises;

setting the sequence of the child operator equal to the sequence for the operator type of the child operator;

setting the sequence of the child operator equal to the sequence plus a number of operator types;

storing the sequence in the table; and

returning the sequence;

wherein segregating the mixed data flow operators comprises inserting staging terminals between the regions;

wherein segregating the mixed data flow operators further comprises, in a first instance;

setting a sequence equal to the sequence of a mixed data flow operator and setting a region equal to the region associated with the sequence;

responsive to determining the region is not associated with the sequence, creating a new region and assigning the new region to the sequence of the mixed data flow operator, setting a region type equal to an operator type corresponding to the mixed data flow operator; and

adding the mixed data flow operator to the new region.

20. The computer implemented method of claim 19, wherein sequencing the execution of the regions further comprises:

sorting the regions in descending order of a sequence number associated with each of the regions;

wherein the classifying step and segregating step are performed based on an operator type, a sequence, and dependencies, wherein the mixed data flow is any of a logical operator graph and a user data flow;

wherein segregating the mixed data flow operators further comprises, in a second instance;

setting a source operator for a link equal to a link source operator, setting a target operator for the link equal to a link target operator, setting a source region equal to the region of the source operator, and setting a target region equal to the region of the target operator; and

responsive to determining the source region is equivalent to the target region, breaking the link between the source operator and the target operator, adding a target staging terminal to the source region and linking the source operator to the target staging terminal, adding a source staging terminal to the target region and linking the source staging terminal to the target operator.

21. The computer-implemented method of claim 20, wherein the mixed data flow includes an execution dependency between a first operator and a second operator, the first operator comprising the given operator, the second operator comprising the another operator, the execution dependency specifying that the second operator is to execute only after the first operator has finished executing, wherein the execution plan graph is generated from the mixed data flow via a third operation, the third operation comprising:

generating a metadata representation of the mixed data flow in order to produce a logical operator graph, wherein generating the metadata representation comprises;

classifying the operators into groups according to the execution dependencies and the plurality of runtime engines, wherein at least one of the groups comprises a sequence of operators in the mixed data flow that is supported by a single runtime engine type, wherein the operators are classified to preclude any circular execution dependency among the groups;

determining, based on the groups and the execution dependencies, an order in which to execute the groups; and

optimizing at least one of the groups based on the runtime engine of the at least one of the groups;

generating a set of code units from the metadata representation, wherein each code unit in the set of code units is executable on the plurality of runtime engines; and

processing the set of code units in order to produce the execution plan graph.

22. The computer-implemented method of claim 21, wherein the third operation further comprises:

(i) generating the set of code units for each group; and

(ii) generating staging code for the staging terminals separating the groups;

wherein processing the set of code units further comprises aggregating the set of code units to build a particular execution plan graph for each group;

wherein the third operation further comprises processing each operator in each of the groups, which in turn comprises;

(i) resolving staging with neighboring operators;

(ii) generating a number of code units for each operator to form the set of code unit; and

(iii) placing the set of code units into the particular execution plan graph for each group;

wherein processing the set of code units further comprises;

(i) generating deployment code for preparing one or more runtime engines for execution of the execution plan graph;

(ii) generating run code for executing the execution plan graph; and

(iii) generating un-deployment code for undoing the effects of the deployment code;

wherein when executing the execution plan graph, each of the groups is executed by one of the plurality of runtime engines associated with the groups;

wherein the third operation further comprises;

(i) performing post aggregation processing;

(ii) aggregating the particular execution plan graph for each group into the execution plan graph;

(iii) adding at least one new runtime engine for executing the execution plan graph;

(iv) establishing a group definition for each group; and

(v) establishing an operator definition for each operator.

23. A non-transitory computer readable medium containing a program which, when executed, performs an operation to generate code for an integrated data system, the operation comprising:

receiving a mixed data flow containing mixed data flow operators, the mixed data flow operators collectively defining operations to be performed to complete the mixed data flow, wherein a first of a plurality of runtime environments of distinct types is selected to perform a given one of the defined operations, wherein the given operation is dependent on at least one other operation performed in a second of the plurality of runtime environments;

generating, based on the mixed data flow, a logical operator graph containing logical operators independent of the plurality of runtime environments;

converting the logical operator graph to an extended query graph model in which the logical operators are converted to model operators associated with the plurality of runtime environments;

analyzing the extended query graph model in order to pre-optimize code generation to include at least one of chunking and execution parallelism;

subsequent to analyzing the extended query graph model, converting the extended query graph model via code generation by operation of one or more computer processors when executing the program, into an execution plan graph executable on the plurality of different types of runtime environments.

24. A system to generate code for an integrated data system, the system comprising:

a computer processor;

a graphical user interface to receive input for creating a mixed data flow containing mixed data flow operators, the mixed data flow operators collectively defining operations to be performed to complete the mixed data flow, wherein a first of a plurality of runtime environments of distinct types is selected to perform a given one of the defined operations, wherein the given operation is dependent on at least one other operation performed in a second of the plurality of runtime environments; and

a code generation system operably connected to the graphical user interface and executable by the processor to;

receive the mixed data flow from the graphical user interface;

generate, based on the mixed data flow, a logical operator graph containing logical operators independent of the plurality of runtime environments,

convert the logical operator graph to an extended query graph model in which the logical operators are converted to model operators associated with the plurality of runtime environments;

analyze the extended query graph model in order to pre-optimize code generation to include at least one of chunking and execution parallelism;

subsequent to analyzing the extended query graph model, convert the extended query graph model via code generation into an execution plan graph executable on the plurality of different types of runtime environments; and

execute the execution plan graph by an execution engine that invokes a plurality of runtime engines, each runtime engine being of a distinct runtime engine type corresponding to a respective one of the plurality of runtime environments.