Cross-platform compiler for data transforms

• US 9,043,764 B2
• Filed: 03/09/2011
• Issued: 05/26/2015
• Est. Priority Date: 03/09/2011
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Ranganathan, Anand, Riabov, Anton V., Udrea, Octavian
• Primary Examiner(s): Dao, Thuy
• Assistant Examiner(s): HAYIM, SAMUEL E
• Application Number:
  US13/043,952
• Publication Number:
  US 20140067750A1
• Time in Patent Office: 1,539 Days
• US Class Current: 717/137
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP

Abstract

Techniques for automatically partitioning a multi-platform data transform flow graph to one or more target output platforms are provided. The techniques include performing type inference on a transform graph, wherein the transform graph comprises one or more data transforms, automatically partitioning the transform graph to one or more target output platforms based on one or more policies, performing an optimization of the partitioned transform graph, and generating code, from the partitioned transform graph, for each set of the one or more data transforms based on the one or more target output platforms.

Claims

1. A method for automatically partitioning a multi-platform data transform flow graph to multiple target output platforms, wherein the method comprises:

performing type inference on a transform graph, wherein the transform graph comprises one or more data transforms;

automatically partitioning the transform graph into multiple sub-graphs, wherein the multiple sub-graphs correspond to at least two target output platforms, and wherein the at least two target output platforms comprise (i) a stream programming language, and (ii) a structured query language;

performing an optimization of the partitioned transform graph; and

generating multiple distinct portions of platform-specific code from the partitioned transform graph, wherein the multiple distinct portions of platform-specific code comprises platform-specific code for each set of the one or more data transforms across the multiple sub-graphs based on the at least two target output platforms associated with each of the multiple sub-graphs.

2. The method of claim 1, wherein said generating comprises generating adapter code to facilitate data flow between the multiple target output platforms.

3. The method of claim 1, wherein performing an optimization of the partitioned transform graph comprises fusing one or more of the one or more data transforms if fusing of one or more data transforms is appropriate.

4. The method of claim 1, wherein performing type inference on a transform graph comprises parsing one or more entity types from at least one of a database schema and one or more virtual stream descriptions.

5. The method of claim 1, wherein performing type inference on a transform graph comprises determining each input and output schema for all one or more data transforms.

6. The method of claim 1, wherein performing type inference on a transform graph comprises verifying schema compatibility over the transform graph of one or more data transforms.

7. The method of claim 1, wherein the one or more data transforms are expressed in an Extract Transform Load (ETL) language for multi-platform transforms (ELM) grammar, and wherein data sources are expressed in Stream Processing Application Declarative Engine (SPADE) programming language or structured query language (SQL) data definition language (DDL).

8. The method of claim 1, wherein each data transform has a cost on each target platform and partitioning is performed to minimize an overall program cost.

9. The method of claim 8, wherein a data transform cost is expressed through one or more user policies and is derived from one or more target platform capabilities.

10. The method of claim 8, wherein the overall program cost comprises costs for one or more connectors between two or more platforms.

11. The method of claim 1, wherein said generating comprises automatically generating bridging and mediation code between two or more output platforms.

12. The method of claim 1, wherein the stream programming language comprises Stream Processing Application Declarative Engine (SPADE) programming language.

13. The method of claim 1, further comprising assigning a cost to each data transform for each target output platform.

14. The method of claim 1, further comprising determining whether each data transform is supported by each target output platform.

15. The method of claim 1, further comprising using one or more scopes to apply an operation to a set of multiple fields or to a set of multiple values.

16. The method of claim 1, further comprising providing a system, wherein the system comprises one or more distinct software modules, each of the one or more distinct software modules being embodied on a tangible computer-readable recordable storage medium, and wherein the one or more distinct software modules comprise a type inference engine module, a partitioning engine module, an optimization engine module and a code generation engine module executing on a hardware processor.

17. A computer program product comprising a tangible computer readable recordable storage medium including computer useable program code for automatically partitioning a multi-platform data transform flow graph to multiple target output platforms, the computer program product including:

computer useable program code for performing type inference on a transform graph, wherein the transform graph comprises one or more data transforms;

computer useable program code for automatically partitioning the transform graph into multiple sub-graphs, wherein each of the multiple sub-graphs correspond to at least two target output platforms, and wherein the at least two target output platforms comprise (i) a stream programming language, and (ii) a structured query language;

computer useable program code for performing an optimization of the partitioned transform graph; and

computer useable program code for generating multiple distinct portions of platform-specific code from the partitioned transform graph, wherein the multiple distinct portions of platform-specific code comprises platform-specific code for each set of the one or more data transforms across the multiple sub-graphs based on the at least two target output platforms associated with each of the multiple sub-graphs.

18. The computer program product of claim 17, wherein said generating comprises computer useable program code for generating adapter code to facilitate data flow between the multiple target output platforms.

19. The computer program product of claim 17, wherein the computer useable program code for performing type inference on a transform graph comprises computer useable program code for determining each input and output schema for all one or more data transforms.

20. The computer program product of claim 17, wherein each data transform has a cost on each target platform and partitioning is performed to minimize an overall program cost.

21. The computer program product of claim 17, wherein the one or more data transforms are expressed in an Extract Transform Load (ETL) language for multi-platform transforms (ELM) grammar, and wherein data sources are expressed in Stream Processing Application Declarative Engine (SPADE) programming language or structured query language (SQL) data definition language (DDL).

22. A system for automatically partitioning a multi-platform data transform flow graph to multiple target output platforms, comprising:

a memory; and

at least one processor coupled to the memory and operative to;

perform type inference on a transform graph, wherein the transform graph comprises one or more data transforms;

automatically partition the transform graph into multiple sub-graphs, wherein the multiple sub-graphs correspond to at least two target output platforms, and wherein the at least two target output platforms comprise (i) a stream programming language, and (ii) a structured query language;

perform an optimization of the partitioned transform graph; and

generate multiple distinct portions of platform-specific code from the partitioned transform graph, wherein the multiple distinct portions of platform-specific code comprises platform-specific code for each set of the one or more data transforms across the multiple sub- graphs based on the at least two target output platforms associated with each of the multiple sub-graphs.

23. The system of claim 22, wherein the at least one processor coupled to the memory and operative to generate code is further operative to generate adapter code to facilitate data flow between the multiple target output platforms.

24. The system of claim 22, wherein the one or more data transforms are expressed in an Extract Transform Load (ETL) language for multi-platform transforms (ELM) grammar, and wherein data sources are expressed in Stream Processing Application Declarative Engine (SPADE) programming language or structured query language (SQL) data definition language (DDL).

25. The system of claim 22, wherein each data transform has a cost on each target platform and partitioning is performed to minimize an overall program cost.