Use of projector and selector component types for ETL map design

• US 9,507,838 B2
• Filed: 10/02/2013
• Issued: 11/29/2016
• Est. Priority Date: 05/17/2013
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Leigh, John, Allan, David, Lau, Kwok-hung (Thomas)
• Primary Examiner(s): Hershley, Mark E
• Application Number:
  US14/044,417
• Publication Number:
  US 20140344210A1
• Time in Patent Office: 1,154 Days
• Field of Search: 707/602
• US Class Current: 1/1
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses

Abstract

A data integration system is disclosed that incorporates one or more techniques for simplifying the design and maintenance of a mapping. As components are added or removed to an existing design, the data integration system removes the need to specify all input and output attributes. In one aspect, components types are implemented that allow assignment expressions to reference all or part of upstream components. Therefore, attributes of certain types of components can be propagated to downstream components or otherwise inherited from upstream components with minimal effort on the part of a map designer. During code generation the attributes required to be projected by any component can be derived based on the needs of the downstream components.

Claims

1. A method for facilitating generation of a data mapping, the method comprising:

receiving, at one or more computer systems, information specifying one or more components of a logical design, wherein at least one of the one or more components is of a first type, and the information specifying the one or more components of the logical design is either indicative of an operation that changes shape of the information flowing through the logical design, or indicative of an operation that controls the flow of information flowing through the logical design but does not change shape of the information flowing through the logical design;

determining, with one or more processors associated with the one or more computer systems, a set of data attributes of an object visible to downstream components in the logical design of the at least one of the one or more components that is of the first type based on upstream components in the logical design;

grouping the set of data attributes of the object that is of the first type; and

generating, with the one or more processors associated with the one or more computer systems, information indicative of the group including the set of attributes visible to the downstream components in the logical design of the at least one of the one or more components that is of the first type in order to create a data integration plan.

2. The method of claim 1 wherein determining, with the one or more processors associated with the one or more computer systems, the set of data attributes visible to the downstream components in the logical design of the at least one of the one or more components that is of the first type comprises deriving one or more attributes visible from an upstream component and exposing the one or more attributes to a downstream component.

3. The method of claim 1 wherein receiving the information specifying the one or more components of the logical design comprises receiving information indicative of a source component having one or more attributes of data stored in a source datastore.

4. The method of claim 1 wherein receiving the information specifying the one or more components of the logical design comprises receiving information indicative of a target component having one or more attributes of data to be stored in a target datastore.

5. The method of claim 1 wherein generating the information indicative of the set of attributes visible to the downstream components in the logical design of the at least one of the one or more components that is of the first type comprises exporting a list of attributes to a downstream component.

6. The method of claim 1 further comprising:

receiving, at the one or more computer systems, a change in the logical design through the introduction or removal of a component or an attribute into the logical design;

determining, with the one or more processors associated with the one or more computer systems, whether the change in the logical design affects the at least one of the one or more components that is of the first type; and

based on a determination that the change in the logical design affects the at least one of the one or more components that is of the first type, determining, with the one or more processors associated with the one or more computer systems, an updated set of data attributes visible to downstream components.

7. The method of claim 1 further comprising:

receiving, at the one or more computer systems, a change in the logical design through the introduction of a component or an attribute into the logical design;

determining, with the one or more processors associated with the one or more computer systems, whether the change in the logical design affects the at least one of the one or more components that is of the first type; and

based on a determination that the change in the logical design affects the at least one of the one or more components that is of the first type, preserving the set of data attributes visible to downstream components.

8. The method of claim 1 further comprising:

receiving, at the one or more computer systems, a change in the logical design renaming a component or an attribute;

determining, with the one or more processors associated with the one or more computer systems, whether the change in the logical design affects the at least one of the one or more components that is of the first type; and

based on a determination that the change in the logical design affects the at least one of the one or more components that is of the first type, renaming the set of data attributes visible to downstream components.

9. The method of claim 1, wherein the data integration plan is divided into a plurality of execution units that are configured to create a physical plan according to a user'"'"'s infrastructure.

10. The method of claim 1, wherein the one or more components comprise one of a source component, a target component, a set component, a table component, a join component, and a filter component.

11. The method of claim 1, wherein the logical design defines data flows between source components and target components.

12. The method of claim 1, further comprising creating the data integration plan with the information indicative of the set of attributes.

13. A non-transitory computer-readable medium storing computer-executable code for facilitating generation of a data mapping, the non-transitory computer-readable medium comprising:

code for receiving information specifying one or more components of a logical design, wherein at least one of the one or more components is of a first type, and the information specifying the one or more components of the logical design is either indicative of an operation that changes shape of the information flowing through the logical design, or indicative of an operation that controls the flow of information flowing through the logical design but does not change shape of the information flowing through the logical design;

code for determining a set of data attributes of an object visible to downstream components in the logical design of the at least one of the one or more components that is of the first type based on upstream components in the logical design;

code for grouping the set of data attributes of the object that is of the first type; and

code for generating information indicative of the group including the set of attributes visible to the downstream components in the logical design of the at least one of the one or more components that is of the first type in order to create a data integration plan.

14. The non-transitory computer-readable medium of claim 13 wherein the code for determining the set of data attributes visible to the downstream components in the logical design of the at least one of the one or more components that is of the first type comprises code for deriving one or more attributes visible from an upstream component and exposing the one or more attributes to a downstream component.

15. The non-transitory computer-readable medium of claim 13 wherein the code for receiving the information specifying the one or more components of the logical design comprises code for receiving information indicative of a source component having one or more attributes of data stored in a source datastore.

16. The non-transitory computer-readable medium of claim 13 wherein the code for receiving the information specifying the one or more components of the logical design comprises code for receiving information indicative of a target component having one or more attributes of data to be stored in a target datastore.

17. The non-transitory computer-readable medium of claim 13 wherein the code for generating the information indicative of the set of attributes visible to the downstream components in the logical design of the at least one of the one or more components that is of the first type comprises code for exporting a list of attributes to a downstream component.

18. The non-transitory computer-readable medium of claim 13 further comprising:

code for receiving a change in the logical design through the introduction or removal of a component or an attribute into the logical design;

code for determining whether the change in the logical design affects the at least one of the one or more components that is of the first type; and

based on a determination that the change in the logical design affects the at least one of the one or more components that is of the first type, code for determining an updated set of data attributes visible to downstream components.

19. The non-transitory computer-readable medium of claim 13 further comprising:

code for receiving a change in the logical design through the introduction or removal of a component or an attribute into the logical design;

code for determining whether the change in the logical design affects the at least one of the one or more components that is of the first type; and

based on a determination that the change in the logical design affects the at least one of the one or more components that is of the first type, code for preserving the set of data attributes visible to downstream components.

20. The non-transitory computer-readable medium of claim 13 further comprising:

code for receiving a change in the logical design renaming a component or an attribute;

code for determining whether the change in the logical design affects the at least one of the one or more components that is of the first type; and

based on a determination that the change in the logical design affects the at least one of the one or more components that is of the first type, code for renaming the set of data attributes visible to downstream components.

21. A system for facilitating generation of a data mapping, the system comprising:

a processor; and

a memory in communication with the processor and configured to store a set of instructions which when executed by the processor configure the processor to;

receive information specifying one or more components of a logical design, wherein at least one of the one or more components is of a first type, and the information specifying the one or more components of the logical design is either indicative of an operation that changes shape of the information flowing through the logical design, or indicative of an operation that controls the flow of information flowing through the logical design but does not change shape of the information flowing through the logical design;

determine a set of data attributes of an object visible to downstream components in the logical design of the at least one of the one or more components that is of the first type based on upstream components in the logical design;

grouping the set of data attributes of the object that is of the first type; and

generate information indicative of the group including the set of attributes visible to the downstream components in the logical design of the at least one of the one or more components that is of the first type in order to create a data integration plan.