Pipelined high speed data transfer mechanism

• US 6,418,478 B1
• Filed: 03/11/1998
• Issued: 07/09/2002
• Est. Priority Date: 10/30/1997
• Status: Expired due to Term

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Prahlad, Anand, Tyagarajan, Mahesh, Ignatius, Paul
• Primary Examiner(s): Rinehart, Mark H.
• Assistant Examiner(s): Kang, Paul
• Application Number:
  US09/038,440
• Time in Patent Office: 1,581 Days
• Field of Search: 709/208, 709/211, 709/201, 709/202, 709/203, 709/231, 709/232, 709/245, 709/213, 709/240, 709/104, 370/431, 712/16
• US Class Current: 709/240
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 12/02 Addressing or allocation; Relocation program address sequencing G06F9/00; arrangements for selecting an address in a digital store G11C8/00
• CPC Class Codes: G06F 3/0613 in relation to throughput
• CPC Class Codes: G06F 3/0656 Data buffering arrangements
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
• CPC Class Codes: G06F 9/5033 considering data affinity
• CPC Class Codes: G06F 9/544 Buffers; Shared memory; Pipes
• CPC Class Codes: H04L 69/04 Protocols for data compression, e.g. ROHC

Abstract

There is disclosed in a communications system having an origination storage device and a destination storage device, a data transfer pipeline apparatus for transferring data in a sequence of N stages, where N is a positive integer greater than 1, from the origination to the destination device. The data transfer apparatus comprises dedicated memory means having a predetermined number of buffers dedicated for carrying data associated with the transfer of data from the origination storage device to the destination device; and master control means for registering and controlling processes associated with the data transfer apparatus for participation in the N stage data transfer sequence. The processes include at least a first stage process for initiating the data transfer and a last Nth stage process for completing data transfer. The first stage process is operative to allocate a buffer from the predetermined number of buffers available within the dedicated memory means for collection, processing, and sending of the data from the origination device to a next stage process. The Nth stage process is operative to receive a buffer allocated to the first stage process from the (N−1)th stage process in the data transfer sequence and to free the buffer upon processing completion and storage in the destination device to permit reallocation of the buffer. The master control means further includes means for monitoring the number of buffers from the pool of buffers allocated or assigned to particular processes in the pipeline, in order to prevent allocation of further buffers to a particular process when the number of buffers currently allocated exceeds a predetermined threshold.

Claims

1. A communication system having an origination storage device, a destination storage device, and a data pipeline apparatus for transferring data from the origination storage device to the destination storage device, the data pipeline apparatus comprising:

a dedicated memory associated with both the origination storage device and the destination storage device having a pool of buffers dedicated to carrying data from the origination storage device to the destination storage device;

a master control module for registering and initiating processes associated with the data transfer apparatus, the processes including;

a first stage collection process that allocates a sequence of buffers from the pool of buffers located in the dedicated memory, the sequence of buffers being locked against further allocation until freed by the process to which the sequence of buffers are assigned;

at least one restore process that stores the data resident in the sequence of buffers in the destination storage device and frees the sequence of buffers so that they may be reallocated; and

a network control module that is responsive to the master control module, the network control module initiating a plurality of network agent processes such that the dedicated memory is shared among each of the processes participating in the data transfer pipeline.

2. The communication system of claim 1 further comprising a next stage compression process coupled between the first stage collection process and the plurality of network agent processors for receiving buffers sent from the first stage collection process and compressing the data transferred in said buffers.

3. The communication system of claim 2 wherein the next stage compression process includes multiple instantiations thereof operating in parallel to receive and process buffer data from the first stage collection process.

4. The communication system of claim 3 further comprising:

a sequence process coupled between the first stage collection process and the multiple instantiations of the next stage compression processes for identifying each buffer received from the first stage collection process with a sequence number; and

a re-sequence process coupled between the multiple instantiations of the next stage compression process and the plurality of network agent processes and responsive to the sequence numbers for re-ordering the sequence of buffers according to the sequence numbers.

5. A method for transferring data from an origination storage device to a destination storage device with a data pipeline apparatus comprising:

allocating a dedicated memory associated with both the origination storage device and the destination storage device having a pool of buffers dedicated to carrying data from the origination storage device to the destination storage device;

configuring a master control module for registering and initiating processes associated with the data transfer apparatus, the processes including;

a first stage collection process that allocates a sequence of buffers from the pool of buffers located in the dedicated memory, the sequence of buffers being locked against further allocation until freed by the process to which the sequence of buffers are assigned;

at least one restore process that stores the data resident in the sequence of buffers in the destination storage device and frees the sequence of buffers so that they may be reallocated; and

providing a network control module that is responsive to the master control module, the network control module initiating a plurality of network agent processes such that the dedicated memory is shared among each of the processes participating in the data transfer pipeline.

6. The method of claim 5 further comprising initiating network agent processes coupled between the first stage collection process and the at least one restore stage process for transferring the data buffers across a network when the first stage collection process is resident on a first computer and the at least one restore process is resident on a second computer.