Methods, systems and computer program products for restartable multiplexed file transfers

• US 7,159,005 B1
• Filed: 10/16/1998
• Issued: 01/02/2007
• Est. Priority Date: 10/16/1998
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Wesley, Ajamu, Wanderski, Michael C., Hind, John R., Housel, Barron C., Britton, Kathryn Heninger, Venkatapathy, Chandrasekaran, Baber, Stephen C.
• Primary Examiner(s): Jean; Frantz B.
• Application Number:
  US09/173,778
• Time in Patent Office: 3,000 Days
• Field of Search: 709/203, 709/200, 709/202, 709/217, 710/29, 707/101, 707/1, 707/103.R, 707/205, 711/4, 711/173
• US Class Current: 709/200
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]

Abstract

Methods, systems and computer program products are provided for concurrently transferring a plurality of files between a first data processing system and a second data processing system by multiplexing chunks of the plurality of files onto a connection between the first data processing system and the second data processing system and dynamically determining a chunk size for each multiplexed chunk. A “chunk” is an amount of data from a file to be transferred wherein the amount of data in the chunk is dynamically defined such that differing amounts of data are transferred in each chunk. Furthermore, the transfer of each multiplexed file is tracked so as to allow restarting of an interrupted file transfer from a point of interruption of the transfer.

Claims

1. An object oriented system for transferring files between a first data processing system and a second data processing system, comprising:

a send symbolic link object which controls the transfer of chunks of files from the first data processing system to the second data processing system so as to dynamically modify characteristics of a file transfer during transfer of the file;

a persistent queue which stores data associated with a file transfer request for populating send symbolic link objects;

a file transfer core which stores copies of files to be transferred and which populates a send symbolic link object with data associated with a file transfer request corresponding to a copy of a file stored in the file transfer core; and

a connection reader which extracts data associated with a file transfer request from the queue, repopulates a send symbol link object with the extracted data and transfers a chunk of a file associated with a file transfer request specified by the repopulated send symbol link object from the first data processing system to the second data processing system, wherein the file associated with the file transfer request comprises more than one chunk.

2. A system according to claim 1, wherein a repopulated send symbolic link object for a transfer request tracks the transfer of chunks from a file stored in the file transfer core associated with the transfer request, determines an amount of data in a chunk the connection reader transfers to the second data processing system and serializes data to repopulate the send symbolic link object for storing in the queue.

3. A system according to claim 1, wherein the data, associated with a file transfer, to repopulate a send symbolic link object is stored and extracted from the persistent queue based on a fairness policy.

4. A system according to claim 1, wherein the file transfer core copies a file to be transferred to the file transfer core when a request to transfer a file is received by the file transfer core.

5. A system according to claim 4, wherein the file transfer core determines if a file stored in the file transfer core is the same file as the file associated with the file transfer request and stores a copy of the file associated with the file transfer request in the file transfer core if the file associated with the file transfer request is different from the files stored in the file transfer core.

6. A system according to claim 4, wherein a file transfer core controls storage of the file associated with the file transfer request so as to modify file storage characteristics associated with the file when it is stored in the file transfer core.

7. A system according to claim 6, wherein the file transfer core controls the characteristics associated with the file based on an operating environment of the first data processing system.

8. A system according to claim 6, wherein the file transfer core controls the characteristics associated with the file based on a file type associated with the file.

9. A system according to claim 1, further comprising:

a receive symbolic link object which controls the receipt of files from the first data processing system to the second data processing system;

a connection writer which receives chunks associated with a file transfer request from the first computer;

a persistent queue, responsive to the connection writer, which stores data for populating receive symbolic link objects; and

a file transfer core that populates a receive symbolic link object with data associated with the received chunks and, responsive to the populated receive symbolic link object, stores received data from files to be transferred.

10. A system according to claim 1, wherein the send symbolic link object comprises:

a method for determining which data from the copy of the file stored in the file transfer core to include in a chunk to transfer to the second data processing system; and

a method for tracking which data from the copy of the file stored in the file transfer core has been sent to the second data processing system.

11. A system according to claim 1, wherein the characteristics of the file transfer dynamically modified comprises a chunk size of the transfer.

12. A method of transferring files between a first data processing system and a second data processing system, comprising:

receiving a file transfer request which requests transfer of a file from the first data processing system to the second data processing system, and which provides data for populating instances of a send symbolic link object;

generating, in response to receipt of a file transfer request, a send symbolic link object populated with data associated with the file transfer request, which controls the transfer of chunks of files from the first data processing system to the second data processing system so as to dynamically modify characteristics of a file transfer during transfer of the file;

storing a copy of the file associated with the transfer request in a transfer core;

storing in a persistent queue the data associated with the file transfer request for populating send symbolic link objects;

extracting from the persistent queue, data associated with the file transfer request to repopulate the send symbolic link object;

wherein the repopulated send symbolic link object carries out the step of determining a chunk of data from the file associated with the file transfer request to transfer to the second data processing system; and

transferring the chunk of data from the file associated with the file transfer request determined by the repopulated send symbolic link object from the first data processing system to the second data processing system.

13. A method according to claim 12, wherein the repopulated send symbolic link object for a transfer request further carries out the steps of:

tracking the transfer of chunks from the file associated with the file transfer request;

determining an amount of data in a chunk transferred to the second data processing system; and

serializing data to repopulate the send symbolic link object in the persistent queue, wherein the serialized data reflects the transfer of an amount of data from the file associated with the file transfer request corresponding to the amount of data in the chunk transferred to the second data processing system.

14. A method according to claim 13, further comprising the steps of:

determining if a file stored in the file transfer core is the same file as the file associated with the file transfer request; and

storing a copy of the file associated with the file transfer request in the file transfer core if the file associated with the file transfer request is different from the files stored in the file transfer core.

15. A method according to claim 14, further comprises the step of modifying file storage characteristics associated with the file prior to carrying out the step of storing a copy of the file.

16. A method according to claim 15, wherein said step of modifying file storage characteristics comprises the step of modifying the characteristics associated with the file based on an operating environment of the first data processing system.

17. A method according to claim 15, wherein said step of modifying file storage characteristics comprises the step of modifying the characteristics associated with the file based on a file type associated with the file.

18. A method according to claim 12, wherein the data to repopulate the send symbolic link object associated with a file transfer is stored and extracted from the queue based on a fairness policy.

19. A method according to claim 12, further comprising:

receiving chunks associated with a file transfer request from the first computer;

responsive to receiving chunks associated with a file transfer request, populating a receive symbolic link object which controls the receipt of files from the first data processing system to the second data processing system;

storing in a persistent queue data for repopulating receive symbolic link objects when subsequent chunks associated with the file transfer request are received; and

wherein a populated receive symbolic link object carries out the step of controlling storage of the received chunk received from the first data processing system.

20. A method according to claim 12, wherein the populated send symbolic link object dynamically modifies a chunk size of the transfer.

21. A computer program product for transferring files between a first data processing system and a second data processing system, comprising:

a computer-readable storage medium having computer-readable program code means embodied in said medium, said computer-readable program code means comprising;

computer-readable program code means for receiving a file transfer request which requests transfer of a file from the first data processing system to the second data processing system, and which provides data for populating instances of a send symbolic link object;

computer-readable program code means for generating, in response to receipt of a file transfer request, a send symbolic link object populated with data associated with the file transfer request, which controls the transfer of chunks of files from the first data processing system to the second data processing system so as to dynamically modify characteristics of a file transfer during transfer of the file;

computer-readable program code means for storing a copy of the file associated with the transfer request in a transfer core;

computer-readable program code means for storing in a persistent queue the data associated with the file transfer request for populating send symbolic link objects;

computer-readable program code means for extracting from the persistent queue, data associated with the file transfer request to repopulate the send symbolic link object;

wherein the repopulated send symbolic link object further comprises computer-readable program code means for determining a chunk of data from the file associated with the file transfer request to transfer to the second data processing system; and

computer-readable program code means for transferring the chunk of data from the file associated with the file transfer request determined by the repopulated send symbol link object from the first data processing system to the second data processing system.

22. A computer program product according to claim 21, wherein the repopulated send symbolic link object for a transfer request further comprises:

computer-readable program code means for tracking the transfer of chunks from the file associated with the file transfer request;

computer-readable program code means for determining an amount of data in a chunk transferred to the second data processing system; and

computer-readable program code means for serializing data to repopulate the send symbolic link object in the persistent queue, wherein the serialized data reflects the transfer of an amount of data from the file associated with the file transfer request corresponding to the amount of data in the chunk transferred to the second data processing system.

23. A computer program product according to claim 21, wherein the data to repopulate the send symbolic link object associated with a file transfer is stored and extracted from the queue based on a fairness policy.

24. A computer program product according to claim 22, further comprising:

computer-readable program code means for determining if a file stored in the file transfer core is the same file as the file associated with the file transfer request; and

computer-readable program code means for storing a copy of the file associated with the file transfer request in the file transfer core if the file associated with the file transfer request is different from the files stored in the file transfer core.

25. A computer program product according to claim 24, further comprising computer-readable program code means for modifying file storage characteristics associated with the file prior to storing a copy of the file.

26. A computer program product according to claim 25, wherein the characteristics associated with the file are modified based on an operating environment of the first data processing system.

27. A computer program product according to claim 25, wherein the characteristics associated with the file are modified based on a file type associated with the file.

28. A computer program product according to claim 21, further comprising:

computer-readable program code means for receiving chunks associated with a file transfer request from the first computer;

responsive to receiving chunks associated with a file transfer request, computer-readable program code means for populating a receive symbolic link object which controls the receipt of files from the first data processing system to the second data processing system;

computer-readable program code means for storing in a persistent queue data for repopulating receive symbolic link objects when subsequent chunks associated with the file transfer request are received; and

computer-readable program code means for storing the received chunk received from the first data processing system.

29. A computer program product according to claim 21, wherein the populated send symbolic link object dynamically modifies a chunk size of the transfer.

30. A method of concurrently transferring a plurality of files between a first data processing system and a second data processing system, the method comprising:

multiplexing chunks of the plurality of files onto a connection between the first data processing system and the second data processing system; and

dynamically determining a chunk size for each multiplexed chunk.

31. A method according to claim 30, further comprising the step of tracking the transfer of each multiplexed file so as to allow restarting of an interrupted file transfer from a point of interruption of the transfer.

32. A method according to claim 30, wherein said tracking step comprises:

storing, in a persistent queue, tracking information associated with each file transfer; and

storing a copy of each of the plurality of files in persistent storage.

33. A method according to claim 30, wherein said step of multiplexing comprises the steps of:

selecting chunks of the plurality of files based on a fairness policy; and

multiplexing the selected chunks on the connection between the first data processing system and the second data processing system.

34. A system for concurrently transferring a plurality of files between a first data processing system and a second data processing system, comprising:

means for multiplexing chunks of the plurality of files onto a connection between the first data processing system and the second data processing system; and

means for dynamically determining a chunk size for each multiplexed chunk.

35. A system according to claim 34, further comprising means for tracking the transfer of each multiplexed file so as to allow restarting of an interrupted file transfer from a point of interruption of the transfer.

36. A system according to claim 34, wherein said means for tracking comprises:

means for storing in a persistent queue, tracking information associated with each file transfer; and

means for storing a copy of each of the plurality of files in persistent storage.

37. A system according to claim 34, wherein said means for multiplexing comprises:

means for selecting chunks of the plurality of files based on a fairness policy; and

means for multiplexing the selected chunks on the connection between the first data processing system and the second data processing system.

38. A computer program product for concurrently transferring a plurality of files between a first data processing system and a second data processing system, comprising:

a computer-readable storage medium having computer-readable program code means embodied in said medium, said computer-readable program code means comprising;

computer-readable program code means for multiplexing chunks of the plurality of files onto a connection between the first data processing system and the second data processing system; and

computer-readable program code means for dynamically determining a chunk size for each multiplexed chunk.

39. A computer program product according to claim 38, further comprising computer-readable program code means for tracking the transfer of each multiplexed file so as to allow restarting of an interrupted file transfer from a point of interruption of the transfer.

40. A computer program product according to claim 38, wherein said computer-readable program code means for tracking comprises:

computer-readable program code means for storing in a persistent queue, tracking information associated with each file transfer; and

computer-readable program code means for storing a copy of each of the plurality of files in persistent storage.

41. A computer program product according to claim 38, wherein said computer-readable program code means for multiplexing comprises:

computer-readable program code means for selecting chunks of the plurality of files based on a fairness policy; and

computer-readable program code means for multiplexing the selected chunks on the connection between the first data processing system and the second data processing system.