Migrating data integration processes through use of externalized metadata representations

• US 20050251533A1
• Filed: 03/16/2005
• Published: 11/10/2005
• Est. Priority Date: 03/16/2004
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: Ascential Software Corporation (International Business Machines Corporation)
• Inventors: Johnston, Robert, Harken, Richard Earl
• Application Number:
  US11/083,040
• Publication Number:
  US 20050251533A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 40/143 Markup, e.g. Standard Generalized Markup Language [SGML] or Document Type Definition [DTD]
• CPC Class Codes: G06F 40/151 Transformation

Abstract

Methods and systems are provided for migrating a data integration facility, such as an ETL job, from a source data integration platform to a target data integration platform. Certain embodiments involve externalizing a metadata representation of a source data integration job; parsing the metadata representation; importing the parsed metadata into a plurality of object representations of the source data integration job; generating an intermediate representation of the source data integration platform based on the plurality of object representations; and translating the intermediate representation to generate a target data integration job; wherein the target data integration job is adapted perform substantially the same functions as the source data integration job.

Claims

1. A method, comprising:

externalizing a metadata representation of a source data integration job;

parsing the metadata representation;

importing the parsed metadata into a plurality of object representations of the source data integration job;

generating an intermediate representation of the source data integration platform based on the plurality of object representations; and

translating the intermediate representation to generate a target data integration job;

wherein the target data integration job is adapted perform substantially the same functions as the source data integration job.

2. The method of claim 1 wherein the source data integration job has a source native format.

3. The method of claim 1 wherein the target data integration job has a target native format.

4. The method of claim 3 wherein the source native format is different than the target native format.

5. The method of claim 1 wherein the object representations comprise class/object representations.

6. The method of claim 1 wherein the object representations comprise atomic representations.

7. The method of claim 1 wherein the intermediate representation is stored in memory.

8. The method of claim 1 wherein the source data integration job comprises an ETL job.

9. The method of claim 1 wherein the metadata representations is in a format selected from the group consisting of an XML format, a Text Export format, a script format, a COBOL format, a C language format, a C++ format, and a Teradata format.

10. The method of claim 1 wherein the step of externalizing a metadata representation includes storing items to be translated in memory to facilitate the process.

11. The method of claim 1 wherein the step of generating an intermediate representation includes producing a set of objects that represent a generic meta-model for a data integration job.

12. The method of claim 11 wherein the generic meta-model comprises an atomic meta-model.

13. The method of claim 11 wherein the intermediate representation comprises a hub adapted to facilitate bi-directional translations.

14. The method of claim 1 wherein the step of generating a virtual representation creates a bi-directional translation facility.

15. The method of claim 1 wherein the source data integration job comprises a source instruction set.

16. The method of claim 1 wherein the source data integration job comprises a source data integration function.

17. The method of claim 1 wherein the source data integration job comprises a source data integration facility.

18. The method of claim 1 wherein the source data integration job is associated with a data integration platform of at least one of a banking institution, a financial services institution, a health care institution, a hospital, an educational institution, a governmental institution, a corporate environment, a non-profit institution, a law enforcement institution, a manufacturer, a professional services organization, and a research institution.

19. A method, comprising:

extracting an instruction set in a first format from a source ETL application instruction set file;

converting the instruction set into a plurality of representations in an externalized format;

parsing the plurality of representations;

transforming the plurality of representations into a generic model;

translating the generic model into the second format; and

loading the output of the translation into a destination ETL application instruction set file.

20. The method of claim 19 wherein the step of parsing the plurality of representations comprises parsing metadata associated with the plurality of representations.

21. The method of claim 20 wherein the metadata is in an XML format and the parsing is performed using an XML parser.

22. The method of claim 19 wherein the generic model comprises at least one of a generic format, an object format, and an atomic format.

23. The method of claim 19 wherein the method further comprises the step of testing the regenerated translated model.

24. The method of claim 23 wherein the step of testing further comprises determining the effectiveness of the method.

25. The method of claim 23 wherein the instruction set comprises at least one of an extract, a transform, and a load instruction set.

26. A system comprising a computer facility adapted to:

externalize a metadata representation of a source data integration job;

parse the metadata representation;

import the parsed metadata into a plurality of object representations of the source data integration job;

generate an intermediate representation of the source data integration platform based on the plurality of object representations; and

translate the intermediate representation to generate a target data integration job;

wherein the target data integration job is adapted perform substantially the same functions as the source data integration job.

27. The system of claim 26 wherein the source data integration job has a source native format.

28. The system of claim 26 wherein the target data integration job has a target native format.

29. The system of claim 28 wherein the source native format is different than the target native format.

30. The system of claim 26 wherein the object representations comprise class/object representations.

31. The system of claim 26 wherein the object representations comprise atomic representations.

32. The system of claim 26 wherein the intermediate representation is stored in memory.

33. The system of claim 26 wherein the source data integration job comprises an ETL job.

34. The system of claim 26 wherein the metadata representations is in a format selected from the group consisting of an XML format, a Text Export format, a script format, a COBOL format, a C language format, a C++ format, and a Teradata format.

35. The system of claim 26 wherein the computer facility is adapted to store items to be translated in memory.

36. The system of claim 26 wherein the computer facility is adapted to generate an intermediate representation including a set of objects that represent a generic meta-model for a data integration job.

37. The system of claim 36 wherein the generic meta-model comprises an atomic meta-model.

38. The system of claim 36 wherein the intermediate representation comprises a hub adapted to facilitate bi-directional translations.

39. The system of claim 26 wherein the computer facility is adapted to create a bi- directional translation facility.

40. The system of claim 26 wherein the source data integration job comprises a source instruction set.

41. The system of claim 26 wherein the source data integration job comprises a source data integration function.

42. The system of claim 26 wherein the source data integration job comprises a source data integration facility.

43. The system of claim 26 wherein the source data integration job is associated with a data integration platform of at least one of a banking institution, a financial services institution, a health care institution, a hospital, an educational institution, a governmental institution, a corporate environment, a non-profit institution, a law enforcement institution, a manufacturer, a professional services organization, and a research institution.

44. A system, comprising a computer facility adapted to:

extract an instruction set in a first format from a source ETL application instruction set file;

convert the instruction set into a plurality of representations in an externalized format;

parse the plurality of representations;

transform the plurality of representations into a generic model;

translate the generic model into a second format; and

load an output of the translation into a destination ETL application instruction set file.

45. The system of claim 44 wherein the computer facility is adapted to parse metadata associated with the plurality of representations.

46. The system of claim 45 wherein the metadata is in an XML format and the parsing is performed using an XML parser.

47. The system of claim 44 wherein the generic model comprises at least one of a generic format, an object format, and an atomic format.

48. The system of claim 44 wherein the computer facility is further adapted to test the regenerated translated model.

49. The system of claim 48 wherein testing includes determining an effectiveness of the output.

50. The system of claim 48 wherein the instruction set comprises at least one of an extract instruction set, a transform instruction set, and a load instruction set.