Pipelined high speed data transfer mechanism
First Claim
1. A communication system having an origination storage device, a destination storage device, and a data pipeline apparatus for transferring data from the origination storage device to the destination storage device, the data pipeline apparatus comprising:
- a dedicated memory associated with both the origination storage device and the destination storage device having a pool of buffers dedicated to carrying data from the origination storage device to the destination storage device;
a master control module for registering and initiating processes associated with the data transfer apparatus, the processes including;
a first stage collection process that allocates a sequence of buffers from the pool of buffers located in the dedicated memory, the sequence of buffers being locked against further allocation until freed by the process to which the sequence of buffers are assigned;
at least one restore process that stores the data resident in the sequence of buffers in the destination storage device and frees the sequence of buffers so that they may be reallocated; and
a network control module that is responsive to the master control module, the network control module initiating a plurality of network agent processes such that the dedicated memory is shared among each of the processes participating in the data transfer pipeline.
5 Assignments
0 Petitions
Accused Products
Abstract
There is disclosed in a communications system having an origination storage device and a destination storage device, a data transfer pipeline apparatus for transferring data in a sequence of N stages, where N is a positive integer greater than 1, from the origination to the destination device. The data transfer apparatus comprises dedicated memory means having a predetermined number of buffers dedicated for carrying data associated with the transfer of data from the origination storage device to the destination device; and master control means for registering and controlling processes associated with the data transfer apparatus for participation in the N stage data transfer sequence. The processes include at least a first stage process for initiating the data transfer and a last Nth stage process for completing data transfer. The first stage process is operative to allocate a buffer from the predetermined number of buffers available within the dedicated memory means for collection, processing, and sending of the data from the origination device to a next stage process. The Nth stage process is operative to receive a buffer allocated to the first stage process from the (N−1)th stage process in the data transfer sequence and to free the buffer upon processing completion and storage in the destination device to permit reallocation of the buffer. The master control means further includes means for monitoring the number of buffers from the pool of buffers allocated or assigned to particular processes in the pipeline, in order to prevent allocation of further buffers to a particular process when the number of buffers currently allocated exceeds a predetermined threshold.
1205 Citations
6 Claims
-
1. A communication system having an origination storage device, a destination storage device, and a data pipeline apparatus for transferring data from the origination storage device to the destination storage device, the data pipeline apparatus comprising:
-
a dedicated memory associated with both the origination storage device and the destination storage device having a pool of buffers dedicated to carrying data from the origination storage device to the destination storage device;
a master control module for registering and initiating processes associated with the data transfer apparatus, the processes including;
a first stage collection process that allocates a sequence of buffers from the pool of buffers located in the dedicated memory, the sequence of buffers being locked against further allocation until freed by the process to which the sequence of buffers are assigned;
at least one restore process that stores the data resident in the sequence of buffers in the destination storage device and frees the sequence of buffers so that they may be reallocated; and
a network control module that is responsive to the master control module, the network control module initiating a plurality of network agent processes such that the dedicated memory is shared among each of the processes participating in the data transfer pipeline. - View Dependent Claims (2, 3, 4)
a sequence process coupled between the first stage collection process and the multiple instantiations of the next stage compression processes for identifying each buffer received from the first stage collection process with a sequence number; and
a re-sequence process coupled between the multiple instantiations of the next stage compression process and the plurality of network agent processes and responsive to the sequence numbers for re-ordering the sequence of buffers according to the sequence numbers.
-
-
5. A method for transferring data from an origination storage device to a destination storage device with a data pipeline apparatus comprising:
-
allocating a dedicated memory associated with both the origination storage device and the destination storage device having a pool of buffers dedicated to carrying data from the origination storage device to the destination storage device;
configuring a master control module for registering and initiating processes associated with the data transfer apparatus, the processes including;
a first stage collection process that allocates a sequence of buffers from the pool of buffers located in the dedicated memory, the sequence of buffers being locked against further allocation until freed by the process to which the sequence of buffers are assigned;
at least one restore process that stores the data resident in the sequence of buffers in the destination storage device and frees the sequence of buffers so that they may be reallocated; and
providing a network control module that is responsive to the master control module, the network control module initiating a plurality of network agent processes such that the dedicated memory is shared among each of the processes participating in the data transfer pipeline. - View Dependent Claims (6)
-
Specification