CREATION OF CHANGE-BASED DATA INTEGRATION JOBS

• US 20150026115A1
• Filed: 07/19/2013
• Published: 01/22/2015
• Est. Priority Date: 07/19/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Gorelik, Alexander, Padmanabhan, Sriram K., Spyker, James D.
• Granted Patent: US 9,305,067 B2
• Time in Patent Office: Days
• US Class Current: 707/602
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/2358 Change logging, detection, and notification replication G06F16/27
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/5005 to service a request

Abstract

A computer software implemented method for transforming a first extract transform load (ETL) job having at least some unload transform load (UTL) portions. The method includes the following steps: (i) decomposing the first ETL job into an intermediate set of one or more jobs; and (ii) for each job of the intermediate set, transforming the job into a transactionally equivalent job to yield a final set of one or more jobs. The decomposing is performed so that each job of the intermediate jobs set is a Simple UTL job. The transforming is performed so that each job of the final set includes no UTL portions.

Claims

1. A method for transforming a first extract transform load (ETL) job having at least one unload transform load (UTL) portion, the method comprising:

decomposing the first ETL job into an intermediate set of one or more Simple UTL jobs; and

for each Simple UTL job of the intermediate set, transforming the job into a transformationally equivalent job to yield a final set of one or more jobs that are not UTL jobs;

wherein;

each Simple UTL job of the intermediate set jobs meets the following criteria;

a single input action retrieves the contents of a single table, or view on a single table, in a database,

contains no actions that operate on the entirety of data for a job,

contains no actions that rely on ordering of data, and

at least one output stage operates on a database.

2. The method of claim 1 wherein:

the first ETL job includes at least one sort type action; and

the decomposing includes removing the at least one sort type actions from the first ETL job.

3. The method of claim 1 wherein:

the first ETL job includes at least one funnel type action; and

the decomposing includes transforming the at least one funnel type action in the first ETL job into a continuous mode.

4. The method of claim 1 wherein:

the first ETL job includes at least one join type action; and

the decomposing includes transforming the at least one join type action in the first ETL job into at least one lookup action.

5. The method of claim 1 wherein:

the first ETL job includes at least one merge type action; and

the decomposing includes transforming the at least one merge type action in the first ETL job into at least one lookup action.

6. The method of claim 1 wherein the transforming includes the following (not necessarily in the following order):

replacing an input stage with a change capture stage programmed to receive change data from a database and to generate an output table including an indicator column that includes an indication of an insert type action or a delete type action;

delivering at least one update type action on a source database as a delete type action followed by an insert type action; and

performing at least one of the following;

inserting a row in a target database table or deleting a row in a target database.

7. The method of claim 1, further comprising:

determining that the first ETL job meets at least one UTL-qualifying-job characteristic;

wherein;

the UTL-qualifying-job characteristics include at least the following;

the first ETL job includes no user defined actions;

the first ETL job includes no aggregate type actions; and

the first ETL job does not utilize any reject links from any merge type actions.

8. The method of claim 1 further comprising:

running the final set of one or more jobs that are not UTL jobs on a set of at least one source database tables to yield a set of at least one target database tables.

9. A computer program product method for transforming a first extract transform load (ETL) job having at least one unload transform load (UTL) portion, the computer program product comprising software stored on a software storage device, the software comprising:

first program instructions programmed to decompose the first ETL job into an intermediate set of one or more Simple UTL jobs; and

second program instructions programmed to, for each Simple UTL job of the intermediate set, transform the job into a transformationally equivalent job to yield a final set of one or more jobs that are not UTL jobs;

wherein;

each Simple UTL job of the intermediate set jobs meets the following criteria;

a single input action retrieves the contents of a single table, or view on a single table, in a database,

contains no actions that operate on the entirety of data for a job,

contains no actions that rely on ordering of data, and

at least one output stage operates on a database; and

the software is stored on a software storage device in a manner less transitory than a signal in transit.

10. The computer program product of claim 9 wherein:

the first ETL job includes at least one sort type action; and

the first programming instructions are further programmed to remove the at least one sort type actions from the first ETL job.

11. The computer program product of claim 9 wherein:

the first ETL job includes at least one funnel type action; and

the first programming instructions are further programmed to transform the at least one funnel type action in the first ETL job into a continuous mode.

12. The computer program product of claim 9 wherein:

the first ETL job includes at least one join type action; and

the first programming instructions are further programmed to transform the at least one join type action in the first ETL job into at least one lookup action.

13. The computer program product of claim 9 wherein:

the first ETL job includes at least one merge type action; and

the first programming instructions are further programmed to transform the at least one merge type action in the first ETL job into at least one lookup action.

14. The computer program product of claim 9 wherein the second program instructions are further programmed to:

replace an input stage with a change capture stage programmed to receive change data from a database and to generate an output table including an indicator column that includes an indication of an insert type action or a delete type action;

deliver at least one update type action on a source database as a delete type action followed by an insert type action; and

perform at least one of the following;

inserting a row in a target database table or deleting a row in a target database.

15. The computer program product of claim 9, wherein the software further comprises:

third program instructions programmed to determine that the first ETL job meets at least one UTL-qualifying-job characteristic;

wherein;

the UTL-qualifying-job characteristics include at least the following;

the first ETL job includes no user defined actions;

the first ETL job includes no aggregate type actions; and

the first ETL job does not utilize any reject links from any merge type actions.

16. The computer program product of claim 9 wherein the software further comprises:

third program instructions programmed to run the final set of one or more jobs that are not UTL jobs on a set of at least one source database tables to yield a set of at least one target database tables.

17. A computer system for transforming a first extract transform load (ETL) job having at least one unload transform load (UTL) portion, the computer system comprising:

a processor(s) set; and

a software storage device;

wherein;

the processor set is structured, located, connected and/or programmed to run software stored on the software storage device; and

the software comprises;

first program instructions programmed to decompose the first ETL job into an intermediate set of one or more Simple UTL jobs, and

second program instructions programmed to, for each Simple UTL job of the intermediate set, transform the job into a transformationally equivalent job to yield a final set of one or more jobs that are not UTL jobs;

wherein;

each Simple UTL job of the intermediate set jobs meets the following criteria;

a single input action retrieves the contents of a single table, or view on a single table, in a database,

contains no actions that operate on the entirety of data for a job,

contains no actions that rely on ordering of data, and

at least one output stage operates on a database.

18. The system of claim 17 wherein the second program instructions are further programmed to:

replace an input stage with a change capture stage programmed to receive change data from a database and to generate an output table including an indicator column that includes an indication of an insert type action or a delete type action;

deliver at least one update type action on a source database as a delete type action followed by an insert type action; and

perform at least one of the following;

inserting a row in a target database table or deleting a row in a target database.

19. The system of claim 17, wherein the software further comprises:

third program instructions programmed to determine that the first ETL job meets at least one UTL-qualifying-job characteristic;

wherein;

the UTL-qualifying-job characteristics include at least the following;

the first ETL job includes no user defined actions;

the first ETL job includes no aggregate type actions; and

the first ETL job does not utilize any reject links from any merge type actions.

20. The system of claim 17 wherein the software further comprises:

third program instructions programmed to run the final set of one or more jobs that are not UTL jobson a set of at least one source database tables to yield a set of at least one target database tables.