Cross-Platform Compiler for Data Transforms

• US 20140067750A1
• Filed: 03/09/2011
• Published: 03/06/2014
• Est. Priority Date: 03/09/2011
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Ranganathan, Anand, Riabov, Anton V., Udrea, Octavian
• Granted Patent: US 9,043,764 B2
• Time in Patent Office: Days
• US Class Current: 707/602
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP

Abstract

Techniques for automatically partitioning a multi-platform data transform flow graph to one or more target output platforms are provided. The techniques include performing type inference on a transform graph, wherein the transform graph comprises one or more data transforms, automatically partitioning the transform graph to one or more target output platforms based on one or more policies, performing an optimization of the partitioned transform graph, and generating code, from the partitioned transform graph, for each set of the one or more data transforms based on the one or more target output platforms.

Claims

1. A method for automatically partitioning a multi-platform data transform flow graph to one or more target output platforms, wherein the method comprises:

performing type inference on a transform graph, wherein the transform graph comprises one or more data transforms;

automatically partitioning the transform graph to one or more target output platforms based on one or more policies;

performing an optimization of the partitioned transform graph; and

generating code, from the partitioned transform graph, for each set of the one or more data transforms based on the one or more target output platforms.

2. The method of claim 1, wherein generating code comprises generating adapter code to facilitate data flow between the one or more target output platforms.

3. The method of claim 1, wherein performing an optimization of the partitioned transform graph comprises fusing one or more of the one or more data transforms if fusing of one or more data transforms is appropriate.

4. The method of claim 1, wherein performing type inference on a transform graph comprises parsing one or more entity types from at least one of a database schema and one or more virtual stream descriptions.

5. The method of claim 1, wherein performing type inference on a transform graph comprises determining each input and output schema for all one or more data transforms.

6. The method of claim 1, wherein performing type inference on a transform graph comprises verifying schema compatibility over the transform graph of one or more data transforms.

7. The method of claim 1, wherein the one or more data transforms are expressed in an Extract Transform Load (ETL) language for multi-platform transforms (ELM) grammar, and wherein data sources are expressed in SPADE or structured query language (SQL) data definition language (DDL).

8. The method of claim 1, wherein each data transform has a cost on each target platform and partitioning is performed to minimize an overall program cost.

9. The method of claim 8, wherein a data transform cost is expressed through one or more user policies and is derived from one or more target platform capabilities.

10. The method of claim 8, wherein the overall program cost comprises costs for one or more connectors between two or more platforms.

11. The method of claim 1, wherein generating code, from the partitioned transform graph, for each set of the one or more data transforms based on the one or more target output platforms comprises automatically generating bridging and mediation code between two or more output platforms.

12. The method of claim 1, wherein the one or more target output platforms comprise at least one of SPADE, SQL, and a combination thereof.

13. The method of claim 1, further comprising assigning a cost to each data transform for each target output platform.

14. The method of claim 1, further comprising determining whether each data transform is supported by each target output platform.

15. The method of claim 1, further comprising using one or more scopes to apply an operation to a set of multiple fields or to a set of multiple values.

16. The method of claim 1, further comprising providing a system, wherein the system comprises one or more distinct software modules, each of the one or more distinct software modules being embodied on a tangible computer-readable recordable storage medium, and wherein the one or more distinct software modules comprise a type inference engine module, a partitioning engine module, an optimization engine module and a code generation engine module executing on a hardware processor.

17. A computer program product comprising a tangible computer readable recordable storage medium including computer useable program code for automatically partitioning a multi-platform data transform flow graph to one or more target output platforms, the computer program product including:

computer useable program code for performing type inference on a transform graph, wherein the transform graph comprises one or more data transforms;

computer useable program code for automatically partitioning the transform graph to one or more target output platforms based on one or more policies;

computer useable program code for performing an optimization of the partitioned transform graph; and

computer useable program code for generating code, from the partitioned transform graph, for each set of the one or more data transforms based on the one or more target output platforms.

18. The computer program product of claim 17, wherein the computer useable program code for wherein generating code comprises computer useable program code for generating adapter code to facilitate data flow between the one or more target output platforms.

19. The computer program product of claim 17, wherein the computer useable program code for performing type inference on a transform graph comprises computer useable program code for determining each input and output schema for all one or more data transforms.

20. The computer program product of claim 17, wherein each data transform has a cost on each target platform and partitioning is performed to minimize an overall program cost.

21. The computer program product of claim 17, wherein the one or more data transforms are expressed in an Extract Transform Load (ETL) language for multi-platform transforms (ELM) grammar, and wherein data sources are expressed in SPADE or structured query language (SQL) data definition language (DDL).

22. A system for automatically partitioning a multi-platform data transform flow graph to one or more target output platforms, comprising:

a memory; and

at least one processor coupled to the memory and operative to;

perform type inference on a transform graph, wherein the transform graph comprises one or more data transforms;

automatically partition the transform graph to one or more target output platforms based on one or more policies;

perform an optimization of the partitioned transform graph; and

generate code, from the partitioned transform graph, for each set of the one or more data transforms based on the one or more target output platforms.

23. The system of claim 22, wherein the at least one processor coupled to the memory and operative to generate code is further operative to generate adapter code to facilitate data flow between the one or more target output platforms.

24. The system of claim 22, wherein the one or more data transforms are expressed in an Extract Transform Load (ETL) language for multi-platform transforms (ELM) grammar, and wherein data sources are expressed in SPADE or structured query language (SQL) data definition language (DDL).

25. The system of claim 22, wherein each data transform has a cost on each target platform and partitioning is performed to minimize an overall program cost.