System and method for flexible data transfer

• US 20060069803A1
• Filed: 07/30/2004
• Published: 03/30/2006
• Est. Priority Date: 07/30/2004
• Status: Active Grant

• Current Assignee: Sbc Knowledge Ventures, L.P.
• Original Assignee: Sbc Knowledge Ventures, L.P.
• Inventors: Woods, William S., Clark, Nicholas James
• Granted Patent: US 7,313,576 B2
• Time in Patent Office: Days
• US Class Current: 709/237
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: Y10S 707/99953 Recoverability
• CPC Class Codes: Y10S 707/99954 Version management

Abstract

A method and system for flexibly transferring data from one or more data sources to one or more data destinations within an information network where each of the one or more data sources have data in a particular source format and each of the one or more data destinations have data in the same or another particular destination format using a parameter database that includes parameters to control the transfer of data, a scheduler that initiates the transfer of data, and a data loader in communications with the parameter database and scheduler that, upon initiation by the scheduler, extracts data from the one or more data sources, manipulates the extracted source data into one or more destination formats associated with the one or more data destinations, and inserts the data into one or more data destinations according to the parameters within the parameter database.

Claims

1. A system for flexibly transferring data from one or more data sources to one or more data destinations within an information network, each of the one or more data sources having data in a particular source format, each of the one or more data destinations having data in a particular destination format, the system comprising:

a parameter database that includes parameters to control the transfer of data;

a scheduler that initiates the transfer of data; and

a data loader in communications with the parameter database and scheduler that, upon initiation by the scheduler, extracts data from the one or more data sources, manipulates the extracted source data into one or more destination formats associated with one or more data destinations, and inserts the data into one or more data destinations according to the parameters within the parameter database.

2. The system according to claim 1 wherein the parameters include any one or combination of the form of the source data, date offset, data source host name, data source locator, data source name, form of the data query, form of the destination data, destination host name, destination locator, destination name, manipulation requirements, the name and location of plug-ins to manipulate data, source drivers, destination drivers, source user, source password, destination user, destination password, scheduling and frequency of data transfers, order of data transfer, number of threads used during data transfer, batch processing procedure, batch group number, retry processing procedure, reporting location, and log location.

3. The system according to claim 1 wherein the data source is any one of a flat file, a database, a computer application, a XML data stream, web services, data warehouse, and data mart.

4. The system according to claim 1 wherein the data destination is any one of a flat file, a database, a computer application, a XML data stream, web services, data warehouse, and data mart.

5. The system according to claim 1 wherein the scheduler determines when the transfer of data is to occur based on any one of a specific time and interval of time.

6. The system according to claim 1 wherein the scheduler is a software application.

7. The system according to claim 6 wherein the scheduler is a UNIX Cron.

8. The system according to claim 1 wherein data manipulation includes any one or combination of data format conversion, conversion of source data content based on computations, merging of data from one or more other data sources, collating data from one or more data source, filtering of data from one or more data sources, derivation of new data, and transferring of data from one or more data source.

9. The system according to claim 8 wherein data manipulation is performed by one or more data manipulation plug-ins.

10. The system according to claim 9 wherein a data manipulation plug-in is co-located with the data loader.

11. The system according to claim 9 wherein a data manipulation plug-in is remote from the data loader, the data loader remotely accessing the data manipulation plug-in to manipulate source data.

12. The system according to claim 11 wherein the data loader performs any one of delivering source data to the remote data manipulation plug-in, the remote data manipulation plug-in returning the manipulated source data to the data loader, and retrieving the data manipulation plug-in from the remote location to enable source data manipulation at the data loader.

13. The system according to claim 9 wherein data manipulation plug-ins are added to and removed from the data loader to satisfy the manipulation requirements.

14. The system according to claim 1 wherein parameters are addable to and removable from the parameter database to satisfy the extraction, manipulation, and insertion requirements.

15. The system according to claim 1 wherein parameters are modifiable to satisfy the extraction, manipulation, and insertion requirements.

16. The system according to claim 1 wherein the data loader is a software application.

17. The system according to claim 1 wherein the data loader includes input and output interfaces, the input interface enabling communications with one or more data sources, the output interface enabling communications with one or more data destinations.

18. The system according to claim 17 wherein the interfaces include one or more drivers to enable communications with one or more data sources and destinations.

19. The system according to claim 18 wherein a driver may support data communications using any one of XML, point-to-point, teradata, JDBC, DB2, RDBMS, FTP, and other protocols.

20. A method of flexibly transferring data from one or more data sources to one or more data destinations within an information network, each of the one or more data sources having data in a particular source format, each of the one or more data destinations having data in a particular destination format, the method comprising:

defining parameters in a parameter database to control the transfer of data;

scheduling a transfer time when the transfer of data is to occur;

extracting data from the one or more data sources according to certain parameters within the parameter database at the transfer time;

manipulating the extracted source data into one or more destination formats associated with one or more data destinations according to certain parameters within the parameter database at the transfer time; and

inserting the destination data at one or more data destinations according to certain parameters within the parameter database at the transfer time.

21. The method according to claim 20 wherein the parameters include any one or combination of the form of the source data, date offset, data source host name, data source locator, data source name, form of the data query, form of the destination data, destination host name, destination locator, destination name, manipulation requirements, the name and location of plug-ins to manipulate data, source drivers, destination drivers, source user, source password, destination user, destination password, scheduling and frequency of data transfers, order of data transfer, number of threads used during data transfer, batch processing procedure, batch group number, retry processing procedure, reporting location, and log location.

22. The method according to claim 20 wherein the data source is any one of a flat file, a database, a computer application, a XML data stream, web services, data warehouse, and data mart.

23. The method according to claim 20 wherein the data destination is any one of a flat file, a database, a computer application, a XML data stream, web services, data warehouse, and data mart.

24. The method according to claim 20 wherein scheduling is based on any one of a specific time and interval of time.

25. The method according to claim 20 wherein scheduling is performed by a software application.

26. The method according to claim 25 wherein scheduling is performed by a UNIX Cron.

27. The method according to claim 20 wherein data manipulation includes any one or combination of data format conversion, conversion of source data content based on computations, merging of data from one or more other data sources, collating data from one or more data source, filtering of data from one or more data sources, derivation of new data, and transferring of data from one or more data source.

28. The method according to claim 27 wherein data manipulation is performed by one or more data manipulation plug-ins.

29. The method according to claim 28 wherein a data manipulation plug-in is co-located with a data loader that performs the extraction, manipulation, and insertion of data.

30. The method according to claim 28 wherein a data manipulation plug-in is remote from the data loader, the data loader remotely accessing the data manipulation plug-in to manipulate source data.

31. The method according to claim 30 wherein the data loader performs any one of delivering source data to the remote data manipulation plug-in, the remote data manipulation plug-in returning the manipulated source data to the data loader, and retrieving the data manipulation plug-in from a remote location to enable source data manipulation at the data loader.

32. The method according to claim 28 wherein data manipulation plug-ins are added to and removed from the data loader to satisfy the manipulation requirements.

33. The method according to claim 20 wherein parameters are addable to and removable from the parameter database to satisfy the extraction, manipulation, and insertion requirements.

34. The method according to claim 20 wherein parameters are modifiable to satisfy extraction, manipulation, and insertion requirements.

35. The method according to claim 20 wherein extracting, manipulating, and inserting are performed by a software application.

36. The method according to claim 20 further including using input and output interfaces to communicate with data sources and destinations, the input interface enabling communications with one or more data sources, the output interface enabling communications with one or more data destinations.

37. The system according to claim 20 wherein the interfaces include one or more drivers to enable communications with one or more data sources and destinations.

38. The system according to claim 37 wherein a driver may support data communications using any one of XML, PPP, Teradata, JDBC, DB2, RDBMS, FTP, and other protocols.

39. A computer readable medium having computer readable program codes embodied therein for causing a computer to flexibly transfer data from one or more data sources to one or more data destinations within an information network, each of the one or more data sources having data in a particular source format, each of the one or more data destinations having data in a particular destination format, the transfer of data being controlled by parameters in a parameter database, the computer readable medium program codes performing functions comprising:

scheduling a transfer time when the transfer of data is to occur;

extracting data from the one or more data sources according to certain parameters within the parameter database at the transfer time;

manipulating the extracted source data into one or more destination formats associated with one or more data destinations according to certain parameters within the parameter database at the transfer time; and

inserting the destination data at one or more data destinations according to certain parameters within the parameter database at the transfer time.

40. The computer readable medium according to claim 39 wherein the parameters include any one or combination of the form of the source data, date offset, data source host name, data source locator, data source name, form of the data query, form of the destination data, destination host name, destination locator, destination name, manipulation requirements, the name and location of plug-ins to manipulate data, source drivers, destination drivers, source user, source password, destination user, destination password, scheduling and frequency of data transfers, order of data transfer, number of threads used during data transfer, batch processing procedure, batch group number, retry processing procedure, reporting location, and log location.

41. The computer readable medium according to claim 39 wherein the data source is any one of a flat file, a database, a computer application, a XML data stream, web services, data warehouse, and data mart.

42. The computer readable medium according to claim 39 wherein the data destination is any one of a flat file, a database, a computer application, a XML data stream, web services, data warehouse, and data mart.

43. The computer readable medium according to claim 39 wherein the scheduling is based on any one of a specific time and interval of time.

44. The computer readable medium according to claim 39 wherein the scheduling is performed by a UNIX Cron.

45. The computer readable medium according to claim 39 wherein data manipulation includes any one or combination of data format conversion, conversion of source data content based on computations, merging of data from one or more other data sources, collating data from one or more data source, filtering of data from one or more data sources, derivation of new data, and transferring of data from one or more data source.

46. The computer readable medium according to claim 45 wherein data manipulation is performed by one or more data manipulation plug-ins.

47. The computer readable medium according to claim 46 wherein a data manipulation plug-in is co-located with the computer readable medium that performs the extraction, manipulation, and insertion of data.

48. The computer readable medium according to claim 46 wherein a data manipulation plug-in is remote from the computer readable medium, the computer readable medium remotely accessing the data manipulation plug-in to manipulate source data.

49. The computer readable medium according to claim 48 wherein the computer readable medium performs any one of delivering source data to the remote data manipulation plug-in, the remote data manipulation plug-in returning the manipulated source data to the computer readable medium, and retrieving the data manipulation plug-in from a remote location to enable source data manipulation at the computer readable medium.

50. The computer readable medium according to claim 46 wherein data manipulation plug-ins are added to and removed from the computer readable medium to satisfy the manipulation requirements.

51. The computer readable medium according to claim 39 wherein parameters are addable to and removable from the parameter database to satisfy the extraction, manipulation, and insertion requirements.

52. The computer readable medium according to claim 39 wherein parameters are modifiable to satisfy the extraction, manipulation, and insertion requirements.

53. The computer readable medium according to claim 39 further including the finctions of using input and output interfaces to communicate with data sources and destinations, the input interface enabling communications with one or more data sources, the output interface enabling communications with one or more data destinations.

54. The system according to claim 53 wherein the interfaces include one or more drivers to enable communications with one or more data sources and destinations.

55. The system according to claim 54 wherein a driver may support data communications using any one of XML, PPP, Teradata, JDBC, DB2, RDBMS, FTP, and other protocols.

56. A system for flexibly transferring data from one or more data sources to one or more data destinations within an information network, each of the one or more data sources having data in a particular source format, each of the one or more data destinations having data in a particular destination format, the system comprising:

means for defining parameters in a parameter database to control the transfer of data;

means for scheduling a transfer time when the transfer of data is to occur;

means for extracting data from the one or more data sources according to certain parameters within the parameter database at the transfer time;

means for manipulating the extracted source data into one or more destination formats associated with one or more data destinations according to certain parameters within the parameter database at the transfer time; and

means for inserting the destination data at one or more data destinations according to certain parameters within the parameter database at the transfer time.

57. A data structure stored in a memory, the data structure for use in a system that transfers data from one or more data sources to one or more data destinations within an information network, each of the one or more data sources having data in a particular source format, each of the one or more data destinations having data in a particular destination format, the data structure comprising:

one or more data source parameters that relate to extracting data from the one or more data sources, including defining the data source name and location;

one or more data manipulation parameters that relate to manipulating the extracted source data into one or more destination formats associated with the one or more data destinations; and

one or more data destination parameters that relate to inserting destination data at the one or more data destinations, including defining data destination name and location.

58. The data structure according to claim 57 wherein the data manipulation parameters relate to any one or combination of data format conversion, conversion of source data content based on computations, merging of data from one or more other data sources, collating data from one or more data source, filtering of data from one or more data sources, derivation of new data, and transferring of data from one or more data source.