Automated identification of complex transformations and generation of subscriptions for data replication

• US 10,210,233 B2
• Filed: 12/16/2015
• Issued: 02/19/2019
• Est. Priority Date: 03/20/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Jarvie, Rachel L., Marcovecchio, Nick, Miller, Gregg W.
• Primary Examiner(s): Zhen, Li B.
• Assistant Examiner(s): Paulino, Lenin
• Application Number:
  US14/970,981
• Publication Number:
  US 20160275099A1
• Time in Patent Office: 1,161 Days
• CPC Class Codes: G06F 16/172 Caching, prefetching or hoarding of files
• CPC Class Codes: G06F 16/1794 Details of file format conversion
• CPC Class Codes: G06F 16/214 Database migration support
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 8/51 Source to source
• CPC Class Codes: G06F 8/65 Updates security arrangements therefor G06F21/57
• CPC Class Codes: G06F 8/76 Adapting program code to run in a different environment; Porting
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes

Abstract

According to embodiments of the present invention, machines, systems, methods and computer program products as part of a data replication process are provided. One or more complex transformations are identified from source code files of installed software products on a target system. A subscription is created for each complex transformation, the subscription containing instructions for transforming data within the source system into a form compatible with the target system. The instructions are executed within the target system to transform source data of the source system into a form compatible with the target system.

Claims

1. A computer-implemented method for data replication comprising:

identifying one or more complex data transformations within database statements from source code files of installed software products on a target system, wherein the source code files on the target system are updated versions relative to corresponding ones of a source system, wherein the identified one or more complex data transformations are stored in an order reflecting an order of installation of the source code files on the target system, and wherein the one or more complex data transformations include one or more from a group of;

moving data from plural table columns to a common table column, moving data from plural database tables to a common database table, and changing column data using a formula;

creating a subscription for each complex data transformation, the subscription containing generated instructions executable on the target system for transforming data within the source system into a form compatible with the target system according to the complex data transformation; and

executing the instructions of the subscriptions within the target system during data replication from the source system to the target system to perform the complex data transformations in the order reflecting the order of installation of the source code files and transform source data of the source system into a form compatible with the target system.

2. The computer-implemented method of claim 1, wherein the subscription comprises one or more references to one or more executable files containing instructions for transforming data of the source system into a form compatible with the target system, the instructions comprising generated source code.

3. The computer-implemented method of claim 2, wherein the one or more executable files further comprise instructions providing fine grain access control to individual items of data to be transformed, the fine grain access control specifying instructions for controlling access to one or more particular tables of data, rows of data, columns of data or variables to be transformed.

4. The computer-implemented method of claim 3, wherein the fine grain access control includes, for a plurality of tables identified to be impacted by the complex data transformation:

determining a subset of the plurality of tables comprising data that is not to be transformed;

inactivating access to the subset of tables; and

performing the complex data transformation only on the tables that have not been inactivated.

5. The computer-implemented method of claim 2, further comprising creating the generated source code by:

selecting a source code file of the target system containing a complex data transformation;

updating source code of the source code file to transform data of the source system into a form compatible with the target system; and

saving the updated source code.

6. The computer-implemented method of claim 1, further comprising:

obtaining the order of installation of the source code files on the target system;

parsing the source code files in the order of installation to identify a corresponding order of the complex data transformations; and

generating a cache of the complex data transformations, wherein each table in the cache lists the identified complex data transformations in the corresponding order based on the respective source code file.

7. The computer-implemented method of claim 1, further comprising:

generating a report listing differences between the source system and the target system;

identifying from the report, items of data that are present within the target system; and

generating a subscription for each complex data transformation having a corresponding item of data in the target system.