BULK DATA TRANSFER

• US 20090006920A1
• Filed: 06/26/2007
• Published: 01/01/2009
• Est. Priority Date: 06/26/2007
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: Michelle Christine Munson, Serban Simu
• Inventors: Munson, Michelle Christine, Simu, Serban
• Granted Patent: US 8,214,707 B2
• Time in Patent Office: Days
• US Class Current: 714/748
• CPC Class Codes: H04L 1/0002 by adapting the transmission rate
• CPC Class Codes: H04L 1/0017 where the mode-switching is based on Quality of Service requirement
• CPC Class Codes: H04L 1/1848 Time-out mechanisms
• CPC Class Codes: H04L 1/1858 Transmission or retransmission of more than one copy of acknowledgement message
• CPC Class Codes: H04L 1/1887 Scheduling and prioritising arrangements

Abstract

This disclosure relates to network data communication. Some embodiments include initiating a network connection between an original source and an ultimate destination, transmitting a block of data from the original source to the ultimate destination on the network, requesting retransmission of lost blocks from the ultimate destination to the source and retransmitting the lost blocks from source to the ultimate destination. These embodiments further include measuring round-trip time of a retransmit request, the round-trip time measured from a time of transmission of a retransmit request from the ultimate destination to a time of reception at the ultimate destination after retransmission from the original source and setting the round-trip time as a minimum retransmission request time for the network connection, wherein the round-trip time includes latencies of the network connection and in data processes at the original source and at the ultimate destination.

Claims

1. A method for reliable data transfer over an unreliable network between a sender and a receiver for which the time overhead for resending lost data remains constant for arbitrarily high delay, loss, or reordering of data in transit, and for arbitrarily high transmission speed, the method comprising:

acquiring an injection rate for transmission of data blocks from the sender;

sending data blocks sequentially from the sender to the receiver at the injection rate;

receiving one or more data blocks at the receiver, each block of the one or more data blocks including an identification number;

detecting missing original data blocks based on identification numbers received;

detecting missing retransmitted data blocks based on the expected time of receipt as a function of the predicted path round-trip time;

determining a path round trip time through a predictive estimation of time to send a retransmission request from the receiver to the sender, retrieve a corresponding data block from a data source at the sender, and transmit the data block to the receiver;

storing one or more identification numbers of data blocks lost on the receiver such that a time to add or retrieve a block id from storage does not degrade as a number of lost blocks scales up;

sending retransmission requests from the receiver to the sender on a timer based on the predicted path round-trip time so that the sender can retransmit lost data as early as possible to maximize the contiguous receipt of data without re-transmitting too-early data blocks that are only delayed in transit and will eventually reach the receiver;

storing retransmission requests for missed data blocks on the sender such that the time to add or retrieve the block id for a block to retransmit does not degrade as the number of retransmission requests increases;

sending retransmitted data at the sender at a rate which is commensurate with the injection rate to minimize the storage of retransmission requests at the sender; and

sending data for all outstanding retransmission requests first before sending data that has not yet been sent to avoid compounding data loss.

2. The method of claim 1, further comprising performing a Van Jacobson method of predicting path round-trip time.

3. The method of claim 1, further comprising at the receiver storing identification numbers of blocks to be retransmitted in a linear array, where a linear index of each block travels with a request for retransmission to the sender and back with the data block itself.

4. The method of claim 1, further comprising at the sender storing a plurality of identification numbers for blocks needing retransmission in a modified Red Black Tree with substantially constant-time retrieval of block id'"'"'s sorted by number.

5. The method of claim 1, where retransmission requests travel in packets with a smallest size possible for the injection rate to minimize the time to transmit retransmission requests from the receiver to the sender.

6. The method of claim 1, further comprising using a high watermark calculated as a logarithmic function of the running average of the number of lost blocks in a disk write cache process to maximize the cache hit rate (thus minimizing random access to the disk).

7. The method of claim 1, wherein the sender transmits data blocks out of sequence.

8. The method of claim 1, further comprising the sender acquiring an injection rate for transmission of the data blocks.

9. The method of claim 1, wherein acquiring the injection rate includes acquiring a fixed injection rate.

10. The method of claim 1, wherein acquiring the injection rate includes acquiring a variable injection rate.

11. The method of claim 1, wherein the injection rate is repeatedly recalculated to prioritize the transmission speed relative to a different data transfer running concurrently over an identical network path, and wherein the injection rate is calculated as a proportion of a steady-state transmission rate for one flow of a target protocol used by the different data transfer running concurrently over the identical network path, comprising:

a) acquiring a target protocol type, a minimum injection rate, a maximum injection rate and a prioritization factor from a user application;

b) periodically re-calculating a new injection rate, comprising;

measuring an instantaneous network round-trip time before transmitting and during transfer, and recording a smallest value observed as the base round trip time brtti;

measuring the instantaneous network round-trip time during transferring and calculating a smoothed average round-trip time srtti;

detecting missed data blocks and calculating an average packet loss probability pi; and

calculating a new rate each time interval (i+1) using;

12. The method of claim 11, wherein the different data transfer uses a protocol with the rate update according to claim 11 and having a steady state rate defined by:

Rate (steady state)=α

/(srtti−

brtti), and wherein alpha is a constant multiple of the desired transfer rate requested by the user application.

13. The method of claim 11, wherein the different data transfer includes any rate or window-based TCP protocol for which its transmission rate evolves over time as a function of one or more of the following parameters:

absolute round-trip delay, change in round-trip delay, and packet-loss, and for which a steady-state transmission rate is a known function of one or more of these parameters, where alpha is derived by equating the steady-state rate for the method of claim 11 with the steady-state rate of the target TCP protocol expressed in terms of absolute round-trip delay, change in round-trip delay, or packet loss.

14. The method of claim 13, wherein an alpha value is generated by equating the steady-state rate for the method of claim 11 found by setting Rate_i=Rate_i+1=Rate (steady state):

Rate (steady state)=α

/(srtti−

brtti)

to the known steady state rate of TCP Reno;

Rate (steady state, TCP Reno)=C/(rtti*pî

0.5)

where;

C is a constant depending on the path MTU for the transfer, rtti is the smoothed round-trip time, and pi is the packet loss probability, and yielding;

α

tcp reno[bits]=(srtti−

brtti) [sec]*C[bits]/(rtti[sec]*pî

0.5)

15. The method of claim 11, wherein the protocol type, maximum and minimum rate and prioritization factor are acquired from the user or user program while the transfer is running.

16. The method of claim 15, wherein parameters are acquired by a management interface.

17. The method of claim 1, wherein the calculated injection rate is only applied when the network path between the sender and receiver is “

congested”

, indicating that other data transfers are running concurrently on the same network path, comprising;

a) measuring the base round-trip time brtti;

b) measuring the smoothed round-trip time srtti;

c) determining “

congested” and

“

not congested”

modes using a hysteresis model, by measuring the difference between the smoothed rtt and the base rtt (the network queuing delay), and a direction of change in the difference, as follows;

18. The method of claim 17, where when in the “

congested mode”

, the prioritization multiple is set to a small fraction of the targeted protocol, causing the transmission rate to equal an insignificant proportion of the steady-state rate of the target protocol.

19. The method of claim 1, further comprising establishing at least initially the injection rate using automatic detection of the total bandwidth capacity of the transfer path.

20. The method of claim 1, further comprising providing an interface that allows a user or application to select a policy for determining the injection rate, including optionally selecting a targeted protocol type, a constant factor for multiplying the steady-state rate of the targeted protocol, and a maximum or minimum rate to cap the target injection rate.

21. A data transfer system for transfer of data over a network between the sender and the receiver, comprising:

means for acquiring an injection rate for transmission of data blocks from the sender;

means for sending data blocks sequentially from the sender to the receiver at the injection rate;

means for receiving one or more data blocks at the receiver, each block of the one or more data blocks including an identification number;

means for detecting missing original data blocks based on identification numbers received;

means for detecting missing retransmitted data blocks based on the expected time of receipt as a function of the predicted path round-trip time;

means for determining a path round trip time through a predictive estimation of time to send a retransmission request from the receiver to the sender, retrieve a corresponding data block from a data source at the sender, and transmit the data block to the receiver;

means for storing one or more identification numbers of data blocks lost on the receiver such that a time to add or retrieve a block id from storage does not degrade as a number of lost blocks scales up;

means for sending retransmission requests from the receiver to the sender on a timer based on the predicted path round-trip time so that the sender can retransmit lost data as early as possible to maximize the contiguous receipt of data without re- transmitting too-early data blocks that are only delayed in transit and will eventually reach the receiver;

means for storing retransmission requests for missed data blocks on the sender such that a time to add or retrieve the block id for a block to retransmit does not degrade as the number of retransmission requests increases;

means for sending retransmitted data at the sender at a rate which is commensurate with the injection rate to minimize the storage of retransmission requests at the sender; and

means for sending data for all outstanding retransmission requests first before sending data that has not yet been sent to avoid compounding data loss.

22. The data transfer system of claim 21, further comprising a path round-trip time predictor for accurately predicting round-trip times to identify lost blocks.

23. The data transfer system of claim 22, wherein the path round-trip predictor is adapted to perform Van Jacobson prediction of predicted round-trip time.

24. The data transfer system of claim 21, wherein a modified Red Black Tree with substantially constant-time retrieval is used by the retransmission means to store sequence numbers for the retransmissions sorted by number.

25. The data transfer system of claim 21, further comprising a disk write cache mechanism to minimize random access to a disk, the mechanism using a high watermark calculated as a running average of the size of a retransmission table.

26. The data transfer system of claim 21, further comprising a management interface for reliable and efficient data transfer.

27. A method for data transfer between a sender in communication with a data source and a receiver, the method comprising:

dividing the data from the data source into one or more blocks;

associating each block of the one or more blocks with a sequential identification number;

receiving an injection rate for transmission of the data blocks;

receiving retransmission requests from the receiver identifying one or more missing blocks, the retransmission requests based on a predicted path round-trip timeout for identifying missing blocks and transmitted from the receiver commensurate with the injection rate; and

transmitting data at the injection rate, the data comprising lost blocks identified in the retransmission requests followed by remaining blocks of the one or more blocks, the transmitting allowing non-sequential transmission of data blocks.

28. The method of claim 27, wherein receiving the injection rate includes receiving a fixed injection rate.

29. The method of claim 27, wherein receiving the injection rate includes receiving a variable injection rate.

30. The method of claim 27, further comprising storing a plurality of identification numbers in a modified Red Black Tree with substantially constant-time retrieval of retransmissions sorted by number.