Creation of change-based data integration jobs

• US 9,659,072 B2
• Filed: 01/29/2016
• Issued: 05/23/2017
• Est. Priority Date: 07/19/2013
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Gorelik, Alexander, Spyker, James D., Padmanabhan, Sriram K.
• Primary Examiner(s): Ly, Anh
• Application Number:
  US15/009,886
• Publication Number:
  US 20160147851A1
• Time in Patent Office: 480 Days
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/2358 Change logging, detection, and notification replication G06F16/27
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/5005 to service a request

Abstract

A computer software implemented method for transforming a first extract transform load (ETL) job having at least some unload transform load (UTL) portions. The method includes the following steps: (i) decomposing the first ETL job into an intermediate set of one or more jobs; and (ii) for each job of the intermediate set, transforming the job into a transactionally equivalent job to yield a final set of one or more jobs. The decomposing is performed so that each job of the intermediate jobs set is a Simple UTL job. The transforming is performed so that each job of the final set includes no UTL portions.

Claims

1. A method for transforming a first extract transform load (ETL) job having at least one unload transform load (UTL) portion, the method comprising:

decomposing the first ETL job into an intermediate set of one or more Simple UTL jobs;

for each Simple UTL job of the intermediate set, transforming the job into a transformationally equivalent job to yield a final set of one or more jobs that are a set of change-based transform load (CTL) job(s);

running the set of CTL job(s) on a source database to yield a target database; and

running the final set of one or more jobs that are not UTL jobs on a set of at least one source database tables to yield a set of at least one target database tables;

wherein;

each Simple UTL job of the intermediate set jobs meets the following criteria;

a single input action retrieves the contents of a single table, or view on a single table, in a database,

contains no actions that operate on the entirety of data for a job,

contains no actions that rely on ordering of data, and

at least one output stage operates on a database.

2. A computer system for transforming a first extract transform load (ETL) job having at least one unload transform load (UTL) portion, the computer system comprising:

a processor(s) set; and

a software storage device;

wherein;

the processor set is structured, located, connected or programmed to run software stored on the software storage device; and

the software comprises;

first program instructions programmed to decompose the first ETL job into an intermediate set of one or more Simple UTL jobs,

second program instructions programmed to, for each Simple UTL job of the intermediate set, transform the job into a transformationally equivalent job to yield a final set of one or more jobs that a set of change-based transform load (CTL) job(s),

third program instructions programmed to run the set of CTL job(s) on a source database to yield a target database; and

fourth program instructions programmed to run the final set of one or more jobs that are not UTL jobs on a set of at least one source database tables to yield a set of at least one target database tables;

wherein;

each Simple UTL job of the intermediate set jobs meets the following criteria;

a single input action retrieves the contents of a single table, or view on a single table, in a database,

contains no actions that operate on the entirety of data for a job,

contains no actions that rely on ordering of data, and

at least one output stage operates on a database.