User interface service for a services oriented architecture in a data integration platform

• US 7,814,142 B2
• Filed: 02/24/2005
• Issued: 10/12/2010
• Est. Priority Date: 08/27/2003
• Status: Expired due to Fees

• Current Assignee: LinkedIn Corporation (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Toum, Christophe, Mamou, Jean-Claude
• Primary Examiner(s): ISMAIL, SHAWKI SAIF
• Application Number:
  US11/065,693
• Publication Number:
  US 20050262194A1
• Time in Patent Office: 2,056 Days
• Field of Search: 709223-224, 709/203, 709/230, 709/232, 719/328
• US Class Current: 709/203
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06Q 10/10 Office automation; Time management

Abstract

A user interface, or a component of a user interface, is deployed as a service in a services oriented architecture for use, for example, in a data integration platform.

Claims

1. A computer implemented method executed in a facility, the method comprising:

providing a module for a data integration function in a data integration platform, wherein the module is stored as a service in a registry of services, wherein the data integration function provides real time data integration of data between a plurality of data sources;

providing an interface for accessing the module;

using the interface to access the module as a service in a services oriented architecture; and

executing the module for the data integration function in real time through a set of stages to dynamically load balance data integration transactions by performing the steps of;

inserting an end-of-wave marker, by a real time agent, between data integration transactions of the data integration function to separate processing the data integration transactions into distinct units;

sending the distinct units into a respective input stage of the set of stages, using the real time agent, wherein the respective input stage is an entry point for a data integration job to be processed by a server;

recognizing, by the real time agent, the end-of-wave marker as marking an end of the distinct unit to form a completed transaction; and

retrieving, by the real time agent, the completed transaction out of a respective output stage of the set of stages.

2. The method of claim 1 wherein the data integration function comprises a metadata management function.

3. The method of claim 1, further comprising:

processing a request for the data from the plurality of data sources, wherein the processing includes a discover data stage to query the plurality of data sources.

4. The method of claim 3, further comprising:

receiving the data from the plurality of data sources to form received data.

5. The method of claim 1, wherein the set of stages further comprises a preparation stage and a transformation stage.

6. The method of claim 5, wherein the preparation stage includes a cleaning process to form cleansed data.

7. The method of claim 6, wherein the transformation stage receives the cleansed data for transformation into desired formats to form transformed data and includes an aggregation process for the cleansed data and transformed data.

8. The method of claim 1, wherein the real time integration of the data supports data integration job instances, wherein a job instance is capable of supporting a batch topology, a real time topology, or a combination thereof.

9. The method of claim 8, further comprising:

pipelining a series of data integration transactions for delivery to the job instance.

10. The method of claim 1, wherein the end-of-wave marker sends a signal indicating to the module to immediately begin processing the data integration function without waiting for a batch process.

11. The method of claim 1, wherein the service is accessed through a web service protocol.

12. The method of claim 1, wherein the real time integration of the data includes communicating with at least one other data source, wherein the at least one other data source comprises a location selected from the group consisting of a first location where data is handled, a second location where data is stored, and a third location where other information is stored.

13. The method of claim 1, wherein a plurality of processing facilities processes a data request concurrently.

14. The method of claim 1, wherein the plurality of data sources comprise one of a data warehouse, a data retrieval system, or both the data warehouse and the data retrieval system.

15. A system, comprising:

a processor;

a module for a data integration function in a data integration platform, wherein the module is stored as a service in a registry of services, wherein the data integration function provides real time data integration of data between a plurality of data sources;

an interface for the module wherein the interface accesses the module, using the processor, as a service in a services oriented architecture,

a real time agent, wherein the real time agent executes the data integration module in real time using the processor through a set of stages to dynamically load balance the data integration transactions, and wherein the real time agent balances the data transactions by using the processor to insert an end-of-wave marker between data integration transactions to separate processing the data integration transactions into distinct units, send the distinct units into a respective input stage of the set of stages, recognize the end-of-wave marker as marking an end of the distinct unit to form a completed transaction, and retrieve the completed transaction out of a respective output stage of the set of stages, wherein the respective input stage is an entry point for a data integration job to be processed by a server.

16. The system of claim 15 wherein the data integration function comprises a metadata management function.

17. The system of claim 15, further comprising:

a real time server, wherein the real time server processes a request for the data from the plurality of data sources, wherein the request includes a discover data stage to query the plurality of data sources.

18. The system of claim 17, wherein the real time server receives the data from the plurality of data sources to form received data.

19. The system of claim 15, wherein the set of stages further comprises a preparation stage and a transformation stage.

20. The system of claim 19, wherein the preparation stage includes a cleaning process to form cleansed data.

21. The system of claim 20, wherein the transformation stage receives the cleansed data for transformation into desired formats to form transformed data and includes an aggregation process for the cleansed data and transformed data.

22. The system of claim 15 wherein the real time integration of the data supports data integration job instances, wherein a job instance is capable of supporting a batch topology, a real time topology, or a combination thereof.

23. The system of claim 22,

wherein the real time agent pipelines a series of data integration transactions for delivery to the job instance.

24. The system of claim 15, wherein the end-of-wave marker sends a signal indicating to the module to immediately begin processing the data integration function without waiting for a batch process.

25. The system of claim 15, wherein the service is accessed through a web service protocol.

26. The system of claim 15, wherein the real time integration of the data includes communicating with at least one other data source, wherein the at least one other data source comprises a location selected from the group consisting of a first location where data is handled, a second location where data is stored, and a third location where other information is stored.

27. The system of claim 15, wherein a plurality of processing facilities processes a data request concurrently.

28. The system of claim 15, wherein the plurality of data sources comprise one of a data warehouse, a data retrieval system, or both the data warehouse and the data retrieval system.