System and method for decomposition of code generation into separate physical units though execution units

• US 9,633,052 B2
• Filed: 11/11/2013
• Issued: 04/25/2017
• Est. Priority Date: 05/17/2013
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Seetharaman, Ganesh, Allan, David, Leigh, John, Lau, Thomas, Klein, Joseph F.
• Primary Examiner(s): St Leger, Geoffrey
• Application Number:
  US14/077,140
• Publication Number:
  US 20140344310A1
• Time in Patent Office: 1,261 Days
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 8/20 Software design
• CPC Class Codes: G06F 8/34 Graphical or visual programming
• CPC Class Codes: G06F 8/35 model driven

Abstract

In various embodiments, a data integration system is disclosed which enables users to create a logical design which is platform and technology independent. The user can create a logical design that defines, at a high level, how a user wants data to flow between sources and targets. The tool can analyze the logical design, in view of the user'"'"'s infrastructure, and create a physical design. The logical design can include a plurality of components corresponding to each source and target in the design, as well as operations such as joins or filters, and access points. Each component when transferred to the physical design generates code to perform operations on the data. Depending on the underlying technology (e.g., SQL Server, Oracle, Hadoop, etc.) and the language used (SQL, pig, etc.) the code generated by each component may be different.

Claims

1. A computer-implemented method of generating a physical design from a logical design for data integration, comprising:

receiving a logical design of a data integration mapping;

applying a plurality of rules to the logical design, wherein the plurality of rules are ranked and the ranking is used to resolve a conflict among the plurality of rules by applying a rule with a higher rank over a rule with a lower rank;

wherein the plurality of rules comprise transformation operations that are allowable by a plurality of execution units and the plurality of rules are pushed to the plurality of execution units;

generating a default physical design comprising the plurality of execution units based on the logical design;

grouping the plurality of execution units into one or more execution unit groups, wherein each execution unit group includes independent execution units which can execute in parallel;

receiving changes to the default physical design, wherein the changes redefine one or more of the plurality of execution units;

dynamically generating a second physical design based on the changes to the default physical design; and

associating the second physical design with the logical design, wherein the logical design can be associated with a plurality of different physical designs,

wherein a first transformation operation is pushed to an execution unit in the default physical design, and a second transformation operation that is distinctly different from the first transformation operation is pushed to an execution unit in the second physical design.

2. The method of claim 1 wherein the plurality of rules are based on one or more user preferences included in the logical design and an implementation environment, and wherein the plurality of rules identify which execution units support operations represented by logical components specified in the data integration mapping.

3. The method of claim 2 wherein the plurality of rules include at least one of:

a rule that adds components to source execution units by default;

a rule that adds components without dependencies to parallel execution units;

a rule that prevents reordering of operations; and

a rule that applies the one or more user preferences that do not conflict with other rules.

4. The method of claim 1 wherein each execution unit includes one or more components and wherein native code is generated for each component in each execution unit and executed according to the default physical design.

5. The method of claim 1 wherein the changes include selecting and moving at least one component from a first execution unit to a second execution unit.

6. The method of claim 1, wherein the default physical design is comprised of a plurality of generic components, and wherein during installation each of the plurality of generic components can be bound to physical components in the implementation environment.

7. The method of claim 1, wherein the plurality of execution units comprise a plurality of servers.

8. The method of claim 7, wherein one of the plurality of servers comprises one of an SQL server and an IT infrastructure server.

9. The method according to claim 1, wherein the transformation operations comprise one of join operations and filter operations.

10. The method according to claim 1, wherein the plurality of execution units of the default physical design correspond to components including one of a source and a target of the logical design.

11. The method according to claim 10, wherein the components comprise one of a selector type and a projector type.

12. The method according to claim 1, wherein the plurality of different physical designs correspond to a plurality of different implementation environments comprising a plurality of different infrastructure components and system resources.

13. A non-transitory computer readable storage medium including instructions stored thereon which, when executed by a processor, cause the processor to perform the steps of:

receiving a logical design of a data integration mapping;

applying a plurality of rules to the logical design, wherein the plurality of rules are ranked and the ranking is used to resolve a conflict among the plurality of rules by applying a rule with a higher rank over a rule with a lower rank;

wherein the plurality of rules comprise transformation operations that are allowable by a plurality of execution units and the plurality of rules are pushed to the plurality of execution units;

generating a default physical design comprising the plurality of execution units based on the logical design;

grouping the plurality of execution units into one or more execution unit groups, wherein each execution unit group includes independent execution units which can execute in parallel;

receiving changes to the default physical design, wherein the changes redefine one or more of the plurality of execution units;

dynamically generating a second physical design based on the changes to the default physical design; and

associating the second physical design with the logical design, wherein the logical design can be associated with a plurality of different physical designs,

wherein a first transformation operation is pushed to an execution unit in the default physical design, and a second transformation operation that is distinctly different from the first transformation operation is pushed to an execution unit in the second physical design.

14. The non-transitory computer readable storage medium of claim 13 wherein the plurality of rules are based on one or more user preferences included in the logical design and an implementation environment, and wherein the plurality of rules identify which execution units support operations specified in the data integration mapping.

15. The non-transitory computer readable storage medium of claim 14 wherein the plurality of rules include at least one of:

a rule that adds components to source execution units by default;

a rule that adds components without dependencies to parallel execution units;

a rule that prevents reordering of operations; and

a rule that applies the one or more user preferences that do not conflict with other rules.

16. The non-transitory computer readable storage medium of claim 13 wherein each execution unit includes one or more components and wherein native code is generated for each component in each execution unit and executed according to the default physical design.

17. The non-transitory computer readable storage medium of claim 13 wherein the changes include selecting and moving at least one component from a first execution unit to a second execution unit.

18. The non-transitory computer readable storage medium of claim 13, wherein the default physical design is comprised of a plurality of generic components, and wherein during installation each of the plurality of generic components can be bound to physical components in the implementation environment.

19. A system of generating a physical design from a logical design for data integration, comprising:

a data integration system comprising one or more computing devices each including a computer readable medium and processor, wherein the data integration system is configured to

receive a logical design of a data integration mapping;

apply a plurality of rules to the logical design, wherein the plurality of rules are ranked and the ranking is used to resolve a conflict among the plurality of rules by applying a rule with a higher rank over a rule with a lower rank;

wherein the plurality of rules comprise transformation operations that are allowable by a plurality of execution units and the plurality of rules are pushed to the plurality of execution units;

generate a default physical design comprising the plurality of execution units based on the logical design;

group the plurality of execution units into one or more execution unit groups, wherein each execution unit group includes independent execution units which can execute in parallel,

receive changes to the default physical design, wherein the changes redefine one or more of the plurality of execution units;

dynamically generate a second physical design based on the changes to the default physical design; and

associate the second physical design with the logical design, wherein the logical design can be associated with a plurality of different physical designs,

wherein a first transformation operation is pushed to an execution unit in the default physical design, and a second transformation operation that is distinctly different from the first transformation operation is pushed to an execution unit in the second physical design.

20. The system of claim 19 wherein the plurality of rules are based on one or more user preferences included in the logical design and an implementation environment, wherein the plurality of rules identify which execution units support operations specified in the data integration mapping, and wherein the plurality of rules include at least one of:

a rule that adds components to source execution units by default;

a rule that adds components without dependencies to parallel execution units;

a rule that prevents reordering of operations; and

a rule that applies the one or more user preferences that do not conflict with other rules.

21. The system of claim 19 wherein each execution unit includes one or more components and wherein native code is generated for each component in each execution unit and executed according to the default physical design.

22. The system of claim 19 further comprising:

a graphical user interface configured to receive the changes to the default physical design wherein the changes include selecting and moving at least one component from a first execution unit to a second execution unit.

23. The system of claim 19, wherein the default physical design is comprised of a plurality of generic components, and wherein during installation each of the plurality of generic components can be bound to physical components in the implementation environment.