Methods and systems for real time integration services

• US 8,307,109 B2
• Filed: 08/24/2004
• Issued: 11/06/2012
• Est. Priority Date: 08/27/2003
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Mamou, Jean-Claude, Cherel, Thomas, Tinnel, Brian, Toum, Christophe, Santos, Cassio Dos, Meeks, David T., Scheffler, Lee
• Primary Examiner(s): Zele, Krista
• Assistant Examiner(s): NAJEE-ULLAH, TARIQ S
• Application Number:
  US10/925,897
• Publication Number:
  US 20050086360A1
• Time in Patent Office: 2,996 Days
• Field of Search: 709/232, 709/203
• US Class Current: 709/232
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses

Abstract

Methods and systems for enterprise data integration are described. The methods and systems take elements of a data integration process typically operating in a batch-mode, transform the elements in real time, and expose the results as a service that can be accessed by a business enterprise in real time, and optionally also in batch mode. The service can be accessed through a graphical user interface, providing automatic data integration in real time without additional coding. The service can also operate with mobile devices.

Claims

1. A computer-implemented method for real time integration of data residing in a plurality of data sources, the computer-implemented method comprising the steps of:

receiving a request containing individual requests for real time integration of data into a pipeline of a computing facility by a real time integration agent of the computing facility;

inserting an end of wave marker after each one of the individual requests in the pipeline to separate processing of the individual requests into distinct units;

processing the individual requests in the request for the real time integration of data by a plurality of data integration platforms of the computing facility from the plurality of data sources using a real time integration input stage until the end of wave marker is encountered, wherein the plurality of data integration platforms are waiting to process the request, wherein the request is processed when received by the plurality of data integration platforms, wherein processing the request includes a discover data stage to query the plurality of data sources to form extracted data;

receiving the extracted data, from the plurality of data sources, to form received data;

integrating the received data, in real time, using a set of stages including metadata management, a preparation stage, and a transform stage to form at least one real time integrated data output configured for at least one data target wherein the preparation stage includes a cleaning process to form cleansed data, the transform stage receives the cleansed data for transformation into a desired format selected from a plurality of formats to form transformed data and includes an aggregation process for the cleansed data and the transformed data; and

presenting the at least one integrated real time data output to a user in real time as a service.

2. The computer-implemented method of claim 1, wherein the service is accessed through a web service protocol.

3. The computer-implemented method of claim 1, wherein real time integration of the data supports data integration job instances, wherein a job instance is capable of supporting one of a batch topology, a real time topology, or the batch topology and the real time topology.

4. The computer-implemented method of claim 3, further comprising pipelining a series of data integration transactions for delivery to the job instance.

5. The computer-implemented method of claim 1, wherein real time integration of the data includes communicating with at least one other data source, wherein the at least one other data source comprises a location selected from a group consisting of a first location where data is handled, a second location where data is stored, and a third location where other information is stored.

6. The computer-implemented method of claim 1, wherein processing the request for data results in a data request, and wherein real time integration includes processing the data request through a plurality of processing facilities.

7. The computer-implemented method of claim 6, wherein the plurality of processing facilities processes the data request concurrently.

8. The computer-implemented method of claim 1, wherein receiving the extracted data from the plurality of data sources is intermittent.

9. The computer-implemented method of claim 8, wherein receiving the extracted data comprises intermittent reception, and wherein the intermittent reception occurs via a wireless communication channel.

10. The computer-implemented method of claim 1, wherein the plurality of data sources comprise one of a data warehouse, a data retrieval system, or the data warehouse and the data retrieval system.

11. The computer-implemented method of claim 1, further comprising receiving the extracted data from a facility selected from a group consisting of a mobile computing facility, a desktop computing facility, and a central computing facility.

12. The computer-implemented method of claim 6, wherein at least one of the plurality of processing facilities comprises at least one remotely located processing facility.

13. The computer-implemented method of claim 12, wherein the at least one remotely located processing facility comprises a server.

14. The computer-implemented method of claim 1, wherein the plurality of data sources contains data from different sources.

15. The computer-implemented method of claim 14, wherein at least one of the different sources comprises data that is incompatible with real time integration.

16. The computer-implemented method of claim 1, wherein presenting the at least one integrated data output as the service comprises publishing the service as a web service.

17. The computer-implemented method of claim 1 wherein the service is defined by a Web Services Description Language description.

18. The computer-implemented method of claim 1, wherein the service is published in a public registry.

19. The computer-implemented method of claim 1, wherein the service is published in a private registry.

20. The computer-implemented method of claim 1, wherein the service is bound by a multiple of program access protocols.

21. The computer-implemented method of claim 1, wherein the user is one of an enterprise device or an enterprise application.

22. The computer-implemented method of claim 21, wherein the enterprise application is a distribution process.

23. The computer-implemented method of claim 21, wherein the enterprise application is a manufacturing process.

24. The computer-implemented method of claim 21, wherein the enterprise application is a financial process.

25. The computer-implemented method of claim 1, further comprising attaching a client application to the service.

26. The computer-implemented method of claim 25, further comprising downloading the client application to a client.

27. The computer-implemented method of claim 26, wherein the client is a mobile computing device.

50. The method of claim 1, wherein the plurality of data integration platforms are a plurality of data integration job instances and further comprising:

prior to processing the individual requests in the request, passing the request and the end of wave marker to a data integration job instance in the plurality of data integration job instances.

51. The method of claim 1, further comprising:

identifying, by the real time integration agent, whether a delay in processing the request is greater than a specified amount; and

responsive to a determination that the delay in processing the request is greater than the specified amount, starting a new instance of the data integration platform without an instance start delay.

52. The method of claim 1, wherein the receiving and integrating steps are performed responsive to encountering the end of wave marker while processing the request.

53. The method of claim 1, wherein the plurality of data integration platforms waiting to process the request comprises the plurality of data integration platforms running and maintaining connections to the plurality of data sources prior to the request being received by the real time integration agent.

28. An apparatus for real time integration of data residing in a plurality of data sources, the apparatus comprising:

a computer storage facility containing computer programs stored therein for execution on a computing facility;

a computing facility connected to the computer storage facility wherein the computing facility executes the computer programs to direct the apparatus to;

receive a request containing individual requests for real time integration of data into a pipeline of the computing facility by a real time integration agent of the computing facility;

insert an end of wave marker after each one of the individual requests in the pipeline to separate processing of the individual requests into distinct units;

process the individual requests in the request for the real time integration of data by a plurality of data integration platforms of the apparatus from the plurality of data sources using a real time integration input stage until the end of wave marker is encountered, wherein the plurality of data integration platforms are waiting to process the request, wherein the request is processed when received by the plurality of data integration platforms, and wherein processing the request includes a discover data stage to query the plurality of data sources to form extracted data;

receive the extracted data, from the plurality of data sources, to form received data;

integrate the received data, in real time, using a set of stages including metadata management, a preparation stage, and a transform stage to form at least one real time integrated data output configured for at least one data target wherein the preparation stage includes a cleaning process to form cleansed data, the transform stage receives the cleansed data for transformation into a desired format selected from a plurality of formats to form transformed data and includes an aggregation process for the cleansed data and the transformed data; and

present the at least one integrated real time data output to a user in real time as a service.

29. The apparatus of claim 28, wherein the service is accessed through a web service protocol.

30. The apparatus of claim 28, wherein the apparatus supports data integration job instances, wherein a job instance is capable of supporting one of a batch topology, a real time topology, or the batch topology and the real time topology.

31. The apparatus of claim 30, further comprising a pipeline facility for managing delivery of a series of data integration transactions to a data integration platform.

32. The apparatus of claim 29, wherein the computing facility communicates with at least one other computing facility, wherein the at least one other computing facility comprises a location selected from a group consisting of a first facility where data is handled, a second facility where data is stored, and a third facility where other information is stored.

33. The apparatus of claim 28, further comprising a plurality of processing facilities, wherein the computing facility processes the request through a plurality of processing facilities.

34. The apparatus of claim 33, wherein the plurality of processing facilities processes the request concurrently.

35. The apparatus of claim 28, wherein communication between the apparatus and the plurality of data sources is intermittent.

36. The apparatus of claim 35, wherein communication comprises a wireless communication channel.

37. The apparatus of claim 28, wherein the plurality of data sources comprise one of a data warehouse, a data retrieval system, or the data warehouse and the data retrieval system.

38. The apparatus of claim 28, wherein the request is generated on a facility selected from a group comprising a mobile computing facility, a desktop computing facility, and a central computing facility.

39. The apparatus of claim 33, wherein at least one of the plurality of processing facilities comprises at least one remotely located processing facility.

40. The apparatus of claim 39, wherein the at least one remotely located processing facility comprises a server.

41. The apparatus of claim 28, wherein the plurality of data sources contains data from different sources.

42. The apparatus of claim 41, wherein at least one of the different sources comprises data that is incompatible with processing by a data integration facility.

43. The apparatus of claim 28, wherein the real time integration agent publishes the service as a web service.

44. The apparatus of claim 28 wherein the service is defined by a Web Services Description Language description.

45. The apparatus of claim 28, wherein the service is published in a public registry.

46. The apparatus of claim 28, wherein the service is published in a private registry.

47. The apparatus of claim 28, wherein the service is bound by a multiple of program access protocols.

48. The apparatus of claim 28, wherein an application is a distribution process.

49. The apparatus of claim 28, wherein an application is a manufacturing process.