Service oriented architecture for a transformation function in a data integration platform

• US 8,060,553 B2
• Filed: 02/24/2005
• Issued: 11/15/2011
• Est. Priority Date: 08/27/2003
• Status: Active Grant

• Current Assignee: SAP SE
• Original Assignee: International Business Machines Corporation
• Inventors: Scheffler, Lee, Mamou, Jean-Claude
• Primary Examiner(s): Hamza; Faruk
• Application Number:
  US11/065,436
• Publication Number:
  US 20050262192A1
• Time in Patent Office: 2,455 Days
• Field of Search: 709/200, 709/203, 709223-224, 709/230, 709/232, 719/328
• US Class Current: 709/203
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06Q 10/10 Office automation; Time management

Abstract

A transformation function of an extract-transform-load data integration process is deployed as a service in a services oriented architecture. A method includes providing a module for a data integration function, the module being a data transformation module; providing a registry of services; providing an interface for the data transformation module; and identifying the data transformation module in the registry; wherein the data transformation module can be accessed as a service in a services oriented architecture. A system includes a data transformation module for a data integration function; a registry of services; and an interface for the data transformation module; wherein the data transformation module is identified in the registry; and wherein the data transformation module can be accessed as a service in a services oriented architecture.

Claims

1. A method executed in a facility, the method comprising:

providing a data transformation module for a data integration function in a data integration platform, wherein the data integration function provides real time data integration of data between a plurality of data sources and wherein the data transformation module transforms data in the plurality of data sources from a first form to a second form;

providing an interface for the data transformation module;

identifying, using a processor, the data transformation module as a service in a registry of services;

responsive to identifying the data transformation module in the registry of services, using the interface to access the data transformation module as the service in a services oriented architecture;

providing a smart client for the service;

identifying the smart client as a service in the registry of services;

downloading the smart client to a client;

sending a request for the service on the interface in accordance with a set of parameters for the smart client; and

responsive to a request for the service on the interface, executing the data integration function in real time, wherein the data integration function supports a batch topology, a real time topology, and a combination of the batch topology and the real time topology;

pipelining one or more data integration transactions for delivery to the data integration function; and

inserting an end-of-wave marker between one or more pipelined data integration transactions to separate processing the pipelined data integration transactions into a set of distinct transactions.

2. The method of claim 1, wherein the data integration function comprises a metadata management function and wherein the facility is a metadata facility.

3. The method of claim 2, wherein the metadata management function is a member of a metadata management system, wherein the metadata management system allows for a modification of metadata in the plurality of data sources.

4. The method of claim 3, wherein the metadata management function further comprises a metabroker, an atomic data repository, and a migration engine.

5. The method of claim 1, wherein the data transformation module cleanses a plurality of data during a transformation of the plurality of data, and wherein cleansing the plurality of data comprises automatically correcting inaccuracies in the plurality of data.

6. The method of claim 1, wherein the set of parameters cause the smart client to buffer the request for the service and send the request in a batch.

7. The method of claim 6, wherein the set of parameters further cause the smart client to select a binding to access the service from a set of available bindings on which the service is available, wherein a selection of the binding determines a coupling type used in accessing the service.

8. The method of claim 1, further comprising:

executing the data integration function in real time through a set of stages to form a data output; and

exposing the data output to a user in real time as a service using the graphical user interface.

9. The method of claim 1, wherein the step of pipelining the one or more data integration transactions for the delivery to the data integration function comprises:

delivering the one or more data integration transactions to the data integration function while the data integration function is already running.

10. The method of claim 9, wherein the data integration function is a first data integration function and further comprising:

delivering the one or more data integration transactions to a second data integration function, wherein the second data integration function is run when the first data integration function is at a maximum capacity.

11. The method of claim 10, wherein the first data integration function providing real time data integration of the data between the plurality of data sources comprising:

responsive to identifying the end of wave marker between the one or more pipelined data integration transactions, flushing a data integration transaction in the one or more pipelined data integration transaction to the first data integration function.

12. The method of claim 11, wherein the end-of-wave marker triggers immediate execution of the data integration function without waiting for a batch process.

13. The method of claim 1, wherein the set of parameters cause the smart client to buffer the request for the service and send the request in a batch.

14. The method of claim 1, wherein data in the request is cleansed by the data transformation module.

15. A system comprising:

a memory;

a data transformation module for a data integration function in a data integration platform, wherein the data integration function provides real time data integration of data between a plurality of data sources, wherein the data transformation module transforms data in the plurality of data sources from a first form to a second form, wherein the data transformation module pipelines one or more data integration transactions for delivery to the data integration function, wherein the data transformation module inserts an end-of-wave marker between one or more pipelined data integration transactions to separate processing the pipelined data integration transactions into a set of distinct transactions, wherein the data transformation module is in the memory, wherein the data integration function supports a batch topology, a real time topology, and a combination of the batch topology and the real time topology;

a registry of services, wherein the registry is in the memory;

an interface for the data transformation module, wherein the interface is in the memory;

a smart client for the service;

wherein the data transformation module is identified in the registry as a service;

wherein the data transformation module can be accessed as the service in a services oriented architecture in real time; and

wherein the smart client is downloaded to a client and sends a request to the service in accordance with a set of parameters for the smart client.

16. The system of claim 15, wherein the data integration function comprises a metadata management function.

17. The system of claim 16, wherein the metadata management function is a member of a metadata management system, wherein the metadata management system allows for a modification of metadata in the plurality of data sources.

18. The system of claim 17, wherein the metadata management function further comprises a metabroker, an atomic data repository, and a migration engine.

19. The system of claim 15, further comprising:

executing the data integration function in real time through a set of stages to form a data output; and

exposing the data output to a user in real time as a service using the graphical user interface.

20. The system of claim 15, wherein the data integration function supports a batch topology, a real time topology, and a combination of the batch topology and the real time topology.

21. The system of claim 20, further comprising:

pipelining one or more data integration transactions for delivery to the data integration function.

22. The system of claim 21, further comprising:

inserting an end-of-wave marker between one or more pipelined data integration transactions to separate processing the pipelined data integration transactions into a set of distinct transactions.

23. The system of claim 22, wherein the end-of-wave marker triggers immediate execution of the data integration function without waiting for a batch process.