System and method for code generation from a directed acyclic graph using knowledge modules

• US 10,073,867 B2
• Filed: 11/11/2013
• Issued: 09/11/2018
• Est. Priority Date: 05/17/2013
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Lau, Thomas, Allan, David
• Primary Examiner(s): Tecklu, Isaac T
• Application Number:
  US14/077,135
• Publication Number:
  US 20140344778A1
• Time in Patent Office: 1,765 Days
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 8/20 Software design
• CPC Class Codes: G06F 8/34 Graphical or visual programming
• CPC Class Codes: G06F 8/35 model driven

Abstract

In various embodiments, a data integration system is disclosed which enables users to create a logical design which is platform and technology independent. The user can create a logical design that defines, at a high level, how a user wants data to flow between sources and targets. The tool can analyze the logical design, in view of the user'"'"'s infrastructure, and create a physical design. The logical design can include a plurality of components corresponding to each source and target in the design, as well as operations such as joins or filters, and access points. Each component when transferred to the physical design generates code to perform operations on the data. Depending on the underlying technology (e.g., SQL Server, Oracle, Hadoop, etc.) and the language used (SQL, pig, etc.) the code generated by each component may be different.

Claims

1. A method of generating code for data integration based on a logical design, comprising:

generating, using a computer including a computer readable medium and processor, a physical design of a data integration process based on a logical design of the data integration process, wherein the physical design includes a plurality of execution units and the physical design is a physical implementation corresponding to physical devices;

assigning a knowledge module to each of a plurality of components in the plurality of execution units, wherein each knowledge module is specific to a type of component and a deployment'"'"'s language and technology type, and wherein the knowledge module is a code template and is configured to implement reusable transformation, and each knowledge module is customizable by a user;

generating code for each of the plurality of execution units in the physical design, based on the assigned knowledge modules, wherein the code that is generated is derived from declarative rules and metadata defined for each of the knowledge modules; and

executing the generated code in an order based on the physical design.

2. The method of claim 1 wherein each knowledge module includes one or more configurable settings.

3. The method of claim 2 wherein each knowledge module is the code template specific to a component type, a technology type, and a language type.

4. The method of claim 3 wherein each knowledge module includes one or more user configurable component, technology, or language options.

5. The method of claim 1 wherein the physical design is a directed, acyclic graph which indicates the order of execution of the generated code.

6. The method of claim 5, wherein a data structure representing code generated for each execution unit having a same language is created and passed to a next execution unit having the same language in the physical design.

7. The method of claim 1 wherein executing the generated code comprises:

simulating code generation; and

displaying the generated code.

21. The method according to claim 1, wherein the physical design comprises heterogeneous hardware platforms.

22. The method according to claim 1, wherein the plurality of execution units correspond to servers in a known IT infrastructure of the user.

23. The method according to claim 1, wherein the physical design incorporates a plurality of different infrastructures.

24. The method according to claim 1, wherein the knowledge module is configured to implement extract, load, and transform (ELT) operations across a plurality of different technologies.

25. The method according to claim 1, wherein the knowledge module comprises one of an integration knowledge module (IKM), a loading knowledge module (LKM), and a check knowledge module (CKM).

8. A non-transitory computer readable storage medium including instructions stored thereon which, when executed by a processor cause the processor to perform the steps of:

generating, using a computer including a computer readable medium and processor, a physical design of a data integration process based on a logical design of the data integration process, wherein the physical design includes a plurality of execution units and the physical design is a physical implementation corresponding to physical devices;

assigning a knowledge module to each of a plurality of components in the plurality of execution units, wherein each knowledge module is specific to a type of component and a deployment'"'"'s language and technology type, and wherein the knowledge module is a code template and is configured to implement reusable transformation, and each knowledge module is customizable by a user;

generating code for each of the plurality of execution units in the physical design, based on the assigned knowledge modules, wherein the code that is generated is derived from declarative rules and metadata defined for each of the knowledge modules; and

executing the generated code in an order based on the physical design.

9. The non-transitory computer readable storage medium of claim 8 wherein each knowledge module includes one or more configurable settings.

10. The non-transitory computer readable storage medium of claim 9, wherein each knowledge module is the code template specific to a component type, a technology type, and a language type.

11. The non-transitory computer readable storage medium of claim 10 wherein each knowledge module includes one or more user configurable component, technology, or language options.

12. The non-transitory computer readable storage medium of claim 8 wherein the physical design is a directed, acyclic graph which indicates the order of execution of the generated code.

13. The non-transitory computer readable storage medium of claim 12, wherein a data structure representing code generated for each execution unit having a same language is created and passed to a next execution unit having the same language in the physical design.

14. The non-transitory computer readable storage medium of claim 8 wherein executing the generated code comprises:

simulating code generation; and

displaying the simulated code.

15. A system of generating code for data integration based on a logical design, comprising:

a data integration system comprising one or more computing devices each including a computer readable medium and processor, wherein the data integration system is configured to

generating a physical design of a data integration process based on a logical design of the data integration process, wherein the physical design includes a plurality of execution units and the physical design is a physical implementation corresponding to physical devices;

assigning a knowledge module to each of a plurality of components in the plurality of execution units, wherein each knowledge module is specific to a type of component and a deployment'"'"'s language and technology type, and wherein the knowledge module is a code template and is configured to implement reusable transformation, and each knowledge module is customizable by a user;

generating code for each of the plurality of components in the physical design, based on the assigned knowledge modules, wherein the code that is generated is derived from declarative rules and metadata defined for each of the knowledge modules; and

executing the generated code in an order based on the physical design.

16. The system of claim 15 wherein each knowledge module includes one or more configurable settings, and wherein each knowledge module is a code template specific to a component type, technology type, and language type.

17. The system of claim 16 wherein each knowledge module includes one or more user configurable component, technology, or language options.

18. The system of claim 15 wherein the physical design is a directed, acyclic graph which indicates the order of execution of the generated code.

19. The system of claim 18 wherein a data structure representing code generated for each execution unit having a same language is created and passed to a next execution unit having the same language in the physical design.

20. The system of claim 15 wherein executing the generated code comprises:

simulating code generation; and

displaying the simulated code.