System and method for generating code for an integrated data system

• US 20070214111A1
• Filed: 03/10/2006
• Published: 09/13/2007
• Est. Priority Date: 03/10/2006
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Padmanabhan, Sriram K., Jin, Qi, Xu, Lin, Liao, Hui
• Granted Patent: US 9,727,604 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database

Abstract

A computer implemented method, apparatus, and computer usable program code for generating code for an integrated data system. A mixed data flow is received. The mixed data flow contains mixed data flow operators, which are associated with multiple runtime environments. A graph is generated containing logical operators based on the mixed data flow in response to receiving the mixed data flow. The logical operators are independent of the plurality of runtime environments. The graph is converted to a model. The logical operators are converted to model operators associated with the multiple runtime environments. The model operators allow for analysis of operations for the mixed data flow. The model is converted into an execution plan graph. The execution plan graph is executable on different runtime environments.

Claims

1. A computer implemented method for generating code for an integrated data system, the computer implemented method comprising:

receiving a mixed data flow, wherein the mixed data flow contains mixed data flow operators, which are associated with a plurality of runtime environments;

responsive to receiving the mixed data flow, generating a graph containing logical operators based on the mixed data flow, wherein the logical operators are independent of the plurality of runtime environments;

converting the graph to a model wherein the logical operators are converted to model operators associated with the plurality of runtime environments, wherein the model operators allow for analysis of operations for the mixed data flow; and

converting the model into an execution plan graph, wherein the execution plan graph is executable on different runtime environments.

2. The computer implemented method of claim 1, wherein the graph is a logical operator graph, wherein the graph operators are logical operator graph operators, wherein the model is a extended query graph model, and wherein the model operators are extended query graph model operators.

3. The computer implemented method of claim 1, further comprising executing the execution plan graph using an execution engine, wherein the execution engine invokes one or more runtime engines.

4. The computer implemented method of claim 2, wherein the one or more runtime engines is any of an extract, transform, load engine, a DataStage engine, and structured query language engine.

5. The computer implemented method of claim 2, wherein the converting the graph step further comprises:

mapping the logical operator graph operations to the extended query graph model operators; and

transforming logical operator graph operations relationships to extended query graph model operations relationships.

6. The computer implemented method of claim 2, wherein the converting the graph step further comprises:

converting a logical operator graph operation directly to an extended query graph model quantifier.

7. The computer implemented method of claim 2, wherein the converting the graph step further comprises:

mapping properties of a logical operator graph operation to properties of an extended query graph model entity.

8. The computer implemented method of claim 2, wherein the converting the graph step further comprises:

mapping a logical operator graph operation to a property of an extended query graph model operator.

9. The computer implemented method of claim 2, wherein the converting the graph step further comprises:

transforming a logical operator graph operation to any of a set of table functions, and stored procedures to invoke an executable program.

10. The computer implemented method of claim 2, wherein the converting the graph step further comprises:

converting an expression in the logical operator graph to an expression tree in the extended query graph model.

11. The computer implemented method of claim 2, further comprising:

performing analysis and optimization of the logical operator graph, extended query graph model, and the execution plan graph.

12. The computer implemented method of claim 2, wherein the extended query graph model includes structured query language operations, executable operations, and custom operations.

13. The computer implemented method of claim 2, wherein the logical operator graph is a metadata representation of the mixed data flow.

14. The computer implemented method of claim 1, wherein the mixed data flow is received from a user.

15. A system comprising:

a graphical user interface for allowing a user to create a mixed data flow, wherein the mixed data flow contains mixed data flow operators, which are associated with a plurality of runtime environments;

a code generation system operably connected to the graphical user interface wherein the code generation system receives the mixed data flow from the graphical user interface, generates a graph containing logical operators based on the mixed data flow wherein the logical operators are independent of the plurality of runtime environments, converts the graph to a model, wherein the logical operators are converted to model operators associated with the plurality of runtime environments, wherein the model operators allow for analysis of operations for the mixed data flow, and converts the model into an execution plan graph, wherein the execution plan graph is executable on different runtime environments.

16. The system of claim 15, wherein the, graph is a logical operator graph, wherein the model is an extended query graph model, wherein the code generation system maps the logical operator graph operations to extended query graph model operations, and wherein the functionality and performance of the mixed data flow is maintained in the execution plan graph.

17. The system of claim 15, wherein a plurality of runtime engines execute one or more regions of the execution plan graph.

18. The system of claim 17, wherein the plurality of runtime engines are any of an extract, transform, load engine, a DataStage engine, and structured query language engine.

19. A computer program product comprising a computer usable medium including computer usable program code for generating code for an integrated data system, said computer program product including:

computer usable program code for receiving a mixed data flow, wherein the mixed data flow contains mixed data flow operators, which are associated with a plurality of runtime environments;

computer usable program code responsive to receiving a mixed data flow, for generating a graph containing logical operators based on the mixed data flow, wherein the logical operators are independent of the plurality of runtime environments;

computer usable program code for converting the graph to a model wherein the logical operators are converted to model operators associated with the plurality of runtime environments, wherein the model operators allow for analysis of operations for the mixed data flow; and

computer usable program code for converting the model into an execution plan graph, wherein the execution plan graph is executable on different runtime environments.

20. The computer program product of claim 19, further comprising:

computer usable program code for mapping the graph operations to the model operators; and

computer usable program code for transforming logical operator graph operations relationships to extended query graph model operations relationships.

21. The computer program product of claim 19, comprising computer usable program code for mapping the graph operations to any of a model quantifier, a property of a model operator, a set of table functions, and a stored procedure to invoke an executable program.

22. The computer program product of claim 19, wherein the computer usable program code for converting the model into an execution plan graph further comprises:

computer usable program code for mapping properties of a graph operation to properties of an model entity.

23. The computer program product of claim 19, wherein the computer usable program code for converting the model into an execution plan graph further comprises:

computer usable program code for converting an expression in the graph to an expression tree in the model.

24. The computer program product of claim 19, further comprising:

computer usable program code for performing analysis and optimization of the logical operator graph, extended query graph model, and the execution plan graph.

25. The computer program product of claim 19, wherein the graph is a logical operator graph, wherein the graph operators are logical operator graph operators, wherein the model is an extended query graph model, and wherein the model operators are extended query graph model operators.

26. The computer program product of claim 25, wherein the logical operator graph is a metadata representation of the mixed data flow.

27. The computer program product of claim 19, wherein the mixed data flow is received from a user.

28. The computer program product of claim 19, wherein a plurality of runtime engines execute one or more regions of the execution plan graph.

29. The computer program product of claim 28, wherein the plurality of runtime engines are any of an extract, transform, load engine, a DataStage engine, and structured query language engine.