System and method for transferring data

• US 20070067488A1
• Filed: 09/16/2005
• Published: 03/22/2007
• Est. Priority Date: 09/16/2005
• Status: Abandoned Application

• Current Assignee: eBay Inc.
• Original Assignee: eBay Inc.
• Inventors: Ramanathan, Subash, McIntire, Michael
• Application Number:
  US11/229,255
• Publication Number:
  US 20070067488A1
• Time in Patent Office: Days
• US Class Current: 709/238
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 2209/541 Client-server
• CPC Class Codes: G06F 9/54 Interprogram communication

Abstract

A system and computer-implemented method transfers data from a source system to a target system. In an embodiment, a control process instantiates and communicates with one or more source nodes and target processes on one or more nodes. After successfully instantiating the source and target processes, the control server informs the source process to transfer data to the target process. In an embodiment, the control server instantiates a multitude of server node and target node, and the transfer of data occurs simultaneously in parallel.

Claims

1. A system comprising:

a control server;

a source system coupled to said control server;

a target system coupled to said control server;

a source system module in said source system; and

a target system module in said target system;

wherein said source system module is coupled to said target system module;

wherein said control server instantiates said source system module and said target system module; and

wherein said control server instructs said source system module to transfer data to said target system module.

2. The system of claim 1, further comprising:

a master controller server coupled to said control server; and

a global process file;

wherein said master controller server has access to said global process file.

3. The system of claim 1, wherein

said source system comprises a plurality of source nodes;

and

said target system comprises a plurality of target nodes;

wherein said source nodes comprise one or more of said source system modules; and

further

wherein said target nodes comprise one or more of said target system modules.

4. The system of claim 1, further comprising:

a source database;

a first target database; and

a second target database;

wherein said source system module is to transfer a portion of said source database to said first target database;

wherein said target system module is to write said portion of said source database to said first target database;

wherein said target system module further is to validate said portion of said source database on said first target database; and

further

wherein said target system module is to write said portion of said source database to said second target database.

5. The system of claim 1, further comprising a process to receive specifications of resources for said system and to dynamically allocate said system resources.

6. The system of claim 3, wherein said control server is to instantiate and to control a plurality of said modules on said plurality of said source system nodes and a plurality of said modules on said plurality of said target system nodes.

7. The system of claim 3, wherein said plurality of modules on said plurality of source nodes and said plurality of modules on said plurality of target nodes reside on multiple images of an operating system.

8. The system of claim 1, further comprising:

one or more communication queues; and

one or more data queues;

wherein said one or more communications queue reside in said control server, said source system, and said target system; and

further

wherein said one or more data queues reside in said source system and said target system.

9. The system of claim 1, further comprising:

a second target system coupled to said control server and said source system;

wherein said control server is to instantiate said second target system; and

wherein said control server is to instruct said source system to transfer data to said second target system.

10. The system of claim 8, further comprising a control server message handler, a source system message handler, and a target system message handler;

and further wherein said one or more communication queues are multithreaded, and said one or more data queues are single threaded.

11. The system of claim 1, wherein said data is compressed and encrypted before said source system transmits said data to said target system.

12. The system of claim 2, wherein said global processing file comprises parameters for a location of said control server, a location of said source system, a location of said target system, and an allotted bandwidth on a network for said source and target systems.

13. The system of claim 3, wherein said data transfer involves said plurality of source nodes and said plurality of target nodes operating simultaneously in parallel.

14. The system according to claim 5, wherein said system resources comprise a location of said source system, a number of nodes in said source system, a location of said target system, a number of nodes in said target system, and an allotment of bandwidth for said source and target systems.

15. The system of claim 1, wherein

said control server is coupled to said source system via a first socket;

said control server is coupled to said target system via a second socket; and

said source system is coupled to said target system via a third socket.

16. A computer-implemented method comprising:

initiating a control process;

instantiating a source system and a target system with said control process;

sending a message from said control process to said source system and said target system, said message instructing said source system to transfer data to said target system; and

transmitting data from said source system to said target system.

17. The computer-implemented method of claim 16, wherein said control process instantiates one or more parent control processes, and further wherein each one of said parent control processes instantiates a source node and target node.

18. The computer-implemented method of claim 17, wherein a plurality of multiple parent control processes instantiate a plurality of source nodes and target nodes, and further wherein said plurality of source nodes transfers data to said plurality of target nodes simultaneously and in parallel.

19. The computer-implemented method of claim 16, further comprising statically setting a network bandwidth threshold for said data transfer.

20. The computer-implemented method of claim 16, further comprising dynamically setting a network bandwidth threshold for said data transfer.

21. The computer-implemented method of claim 18, wherein said control process informs a source node of the portion of a database to transfer to said target node.

22. The computer-implemented method of claim 16, further comprising dynamically determining the portion of a database that a source node transfers to a target node.

23. The computer-implemented method of claim 16, further comprising:

compressing and encrypting said data before said source system transmits said data to said target system.

24. A machine readable medium comprising instructions thereon for executing a process comprising:

initiating a control process;

instantiating a source system and a target system with said control process;

sending a message from said control process to said source system and said target system, said message instructing said source system to transfer data to said target system; and

transmitting data from said source system to said target system.

25. The machine readable medium of claim 24, wherein said control process instantiates one or more parent control processes, and further wherein each one of said parent control processes instantiates a source node and target node.

26. The machine readable medium of claim 25, wherein a plurality of multiple parent control processes instantiate a plurality of source nodes and target nodes, and further wherein said plurality of source nodes transfers data to said plurality of target nodes simultaneously and in parallel.

27. The machine readable medium of claim 24, further comprising dynamically determining an environment for the transmission of data from said source system to said target system.

28. The machine readable medium of claim 24, further comprising statically determining an environment for the transmission of data from said source system to said target system.

29. A computer-implemented method comprising:

determining data to be transferred from a first system to a second system;

generating a script to gather environmental variables and to set up instances to execute;

building runtime environments;

building directories based on said data;

generating a second script to generate subsets of information tables to inform a source system process and a target system process the functions to execute, to validate data that has been transferred to said second system, and to apply said data to said second system;

acquiring said data from a database in said first system, and transferring said data from said first system to said second system; and

writing said data to a database in said second system.

30. The computer-implemented method of claim 29, wherein

said first system comprises a plurality of nodes;

said second system comprise a plurality of nodes; and

said data is transferred simultaneously and in parallel from said first system to said second system using said plurality of first system nodes and said plurality of second system nodes.