Regenerating data integration functions for transfer from a data integration platform

• US 7,761,406 B2
• Filed: 03/16/2005
• Issued: 07/20/2010
• Est. Priority Date: 03/16/2004
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Harken, Richard Earl
• Primary Examiner(s): Pham; Hung Q
• Assistant Examiner(s): CHEUNG, HUBERT G
• Application Number:
  US11/083,056
• Publication Number:
  US 20050256892A1
• Time in Patent Office: 1,952 Days
• Field of Search: 707/602
• US Class Current: 707/602
• CPC Class Codes: G06F 16/214 Database migration support
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06Q 10/10 Office automation; Time management

Abstract

Methods and systems are provided for migrating a data integration facility, such as an ETL job, from a source data integration platform to a target data integration platform. Certain embodiments involve automatically interpreting at least one operation of a first data integration function adapted to operate on a first data integration platform; translating the at least one interpreted operation into an intermediate format; and regenerating the at least one operation of the first data integration function from the intermediate format to form a regenerated data integration function operation.

Claims

1. A method for migrating a data integration facility executed by a computing facility configured to perform the steps of:

interpreting at least one operation of a first data integration function configured to operate on a first data integration platform to form at least one interpreted operation, wherein the first data integration function comprises at least one of a data source identification, a data target identification, a data cleansing, a data mapping, a data extraction, a data transformation, and a loading;

translating the formed at least one interpreted operation into an intermediate format, wherein the intermediate format is a platform-independent representation of the first data integration function that preserves logic of the at least one operation of the first data integration function, wherein translating the formed at least one interpreted operation into the intermediate format further comprises;

receiving a request for real time integration of data from a pipeline containing individual requests by a real time integration agent of the computing facility, wherein each request in the pipeline is marked with a start of wave marker and an end of wave marker, wherein the start of wave marker enables the real time integration agent to recognize initiation of the received request and the end of wave marker indicates completion of a data integration job instance associated with the received request; and

responsive to existence of the start of wave marker, processing the received request by the real time integration agent of the computing facility, wherein the received request corresponds to a transaction in the pipeline, and wherein multiple transactions are in the pipeline simultaneously and are processed as each transaction is received; and

regenerating the at least one operation of the first data integration function from the translated formed at least one interpreted operation in the intermediate format to form a regenerated data integration function operation such that the formed regenerated data integration function operation is configured to operate on a second data integration platform.

2. The method of claim 1, wherein the formed regenerated data integration function operation performs substantially the same logical operation on the second data integration platform as the first data integration function on the first data integration platform.

3. The method of claim 2, wherein, before the step of translating the formed at least one interpreted operation into the intermediate format, the first data integration function is not operationally compatible with the second data integration platform.

4. The method of claim 1, wherein the step of regenerating the at least one operation into the intermediate format further comprises parsing code associated with the at least one operation.

5. The method of claim 4, wherein the parsing code associated with the at least one operation comprises parsing metadata associated with the at least one operation to the intermediate format, and wherein the intermediate format is an object-oriented representation of the first data integration function.

6. The method of claim 5, wherein the metadata associated with the at least one operation is in an eXtensible Markup Language (XML) format and wherein the parsing metadata associated with the at least one operation to the intermediate format comprises parsing the metadata associated with the at least one operation using an XML parser.

7. The method of claim 5, wherein the parsed metadata associated with the at least one operation is formed by parsing the metadata associated with the at least one operation and wherein the parsed metadata associated with the at least one operation is transformed from a first format into a second format.

8. The method of claim 5, wherein the step of regenerating the at least one operation of the first data integration function comprises translating the platform-independent representation of the first data integration function into a native code for the second data integration platform.

9. The method of claim 5, wherein the first data integration platform and the second data integration platform is semantically mapped to the intermediate format via a translation broker.

10. The method of claim 7, wherein the second format comprises at least one of a generic format, an object format, and an atomic format.

11. The method of claim 2, wherein the method further comprises a step of testing the formed regenerated data integration function operation on the second data integration platform.

12. The method of claim 11, wherein the step of testing the formed regenerated data integration function operation on the second data integration platform further comprises determining an effectiveness of regenerating the at least one operation.

13. The method of claim 1, wherein the step of translating the formed at least one interpreted operation into the intermediate format comprises analyzing a syntax of a source code associated with the at least one operation.

14. The method of claim 13, wherein the source code associated with the at least one operation is divided into a number of syntax blocks, wherein each syntax block is identified as performing a known data integration function.

15. The method of claim 14, wherein the identified syntax block is replaced by a substitute syntax block, wherein the substitute syntax block is in the intermediate format.

16. The method of claim 15, wherein the step of regenerating the at least one operation of the first data integration function comprises editing the substitute syntax block to a modified source code that is used by the second data integration platform.

17. The method of claim 1, wherein the step of interpreting the at least one operation of the first integration function, the step of translating the formed at least one interpreted operation into the intermediate format, and the step of regenerating the at least one operation of the first data integration function is performed bi-directionally between the first data integration platform and the second data integration platform.

18. A system for migrating data, the system comprising:

a first interface to a first data integration platform for interpreting at least one operation of a first data integration function configured to operate on the first data integration platform to form at least one interpreted operation, wherein the first data integration function comprises at least one of a data source identification, a data target identification, a data cleansing, a data mapping, a data extraction, a data transformation, and a loading;

a translation facility that translates the formed at least one interpreted operation into an intermediate format, wherein the intermediate format is a platform-independent representation of the first data integration function that preserves logic of the at least one operation of the first data integration function;

a regeneration facility for regenerating the at least one operation of the first data integration function from the translated formed at least on interpreted operation in the intermediate format to form a regenerated data integration function operation such that the formed regenerated data integration function operation is configured to operate on a second data integration platform;

a real time integration server configured to operate with the translation facility and the regeneration facility, wherein the real time integration server is configured to receive a request for real time integration of data from a pipeline containing individual requests by a real time integration agent of a computing facility, wherein each request in the pipeline is marked with a start of wave marker and an end of wave marker, wherein the start of wave marker enables the real time integration agent to recognize initiation of the received request and the end of wave marker indicates completion of a data integration job instance associated with the received request and process the received request by the real time integration agent of the computing facility in response to existence of the start of wave marker, wherein the received request corresponds to a transaction in the pipeline, and wherein multiple transactions are in the pipeline simultaneously and are processed as each transaction is received; and

a second interface to the second data integration platform for loading the formed regenerated data integration function operation into the second data integration platform.

19. The system of claim 18, wherein the first data integration function is not operationally compatible with the second data integration platform.

20. The system of claim 18, wherein the regeneration facility is further configured to associate code with the at least one operation during regenerating the at least one operation of the first data integration function.

21. The system of claim 20, wherein the associated code with the at least one operation comprises code for parsing metadata associated with the at least one operation, and wherein the parsed metadata associated with the at least one operation is transformed from a first format into a second format.

22. The system of claim 21, wherein the metadata associated with the at least one operation is in an eXtensible Markup Language (XML) format and the parsing is performed using an XML parser.

23. The system of claim 18 further comprising a testing facility configured to test the formed regenerated data integration function operation.

24. The system of claim 18, wherein the translation facility comprises a translation broker that semantically maps each unique data integration platform or job to an atomic, object-oriented model.