Pipelined high speed data transfer mechanism

• US 7,401,154 B2
• Filed: 05/13/2002
• Issued: 07/15/2008
• Est. Priority Date: 10/30/1997
• Status: Expired due to Fees

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Prahlad, Anand, Tyagarajan, Mahesh, Ignatius, Paul
• Primary Examiner(s): Kang; Paul H
• Application Number:
  US10/144,683
• Publication Number:
  US 20020188691A1
• Time in Patent Office: 2,255 Days
• Field of Search: 709/201, 709/213, 709/224, 709/225, 709/230, 709/231, 709/232, 709/238, 709223- 5, 709229- 32, 709/226, 709/214
• US Class Current: 709/232
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 12/02 Addressing or allocation; Relocation program address sequencing G06F9/00; arrangements for selecting an address in a digital store G11C8/00
• CPC Class Codes: G06F 3/0613 in relation to throughput
• CPC Class Codes: G06F 3/0656 Data buffering arrangements
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
• CPC Class Codes: G06F 9/5033 considering data affinity
• CPC Class Codes: G06F 9/544 Buffers; Shared memory; Pipes
• CPC Class Codes: H04L 69/04 Protocols for data compression, e.g. ROHC

Abstract

There is disclosed in a communications system having an origination storage device and a destination storage device, a data transfer pipeline apparatus for transferring data in a sequence of N stages, where N is a positive integer grater than 1, from the origination to the destination device. The data transfer apparatus comprises dedicated memory means having a predetermined number of buffers dedicated for carrying data associated with the transfer of data from the origination storage device to the destination device; and master control means for registering and controlling processes associated with the data transfer apparatus for participation in the N stage data transfer sequence. The processes include at least a first stage process for initiating the data transfer and a last Nth stage process for completing data transfer. The first stage process is operative to allocate a buffer from the predetermined number of buffers available within the dedicated memory means for collection, processing, and sending of the data from the origination device to a next stage process. The Nth stage process is operative to receive a buffer allocated to the first stage process from the (N−1)th stage process in the data transfer sequence and to free the buffer upon processing completion and storage in the destination device to permit reallocation of the buffer. The master control means further includes monitor means for monitoring the number of buffers from the pool of buffers allocated or assigned to particular processes in the pipeline, in order to prevent allocation of further buffers to a particular process when the number of buffers currently allocated exceeds a predetermined threshold.

Claims

1. A data transfer system for providing reliable high speed data transfer between computers, the data transfer system comprising:

an origination storage device;

a destination storage device that receives data from the origination storage device via a data transfer pipeline;

a data transfer pipeline apparatus that transfers data in a sequence of stages, the data transfer pipeline apparatus comprising;

a data compression module for compressing data transferred through at least a portion of the data transfer pipeline apparatus,

a dedicated memory with a predetermined number of buffers, and

a master control module for registering and controlling processes associated with the data transfer pipeline apparatus for participation in the data transfer pipeline apparatus, wherein said master control module registers multiple instantiations of the same process in a given stage of said data transfer sequence; and

a network control module coupled to said master control module for initiating network agent processes to enable communications data transfer between storage devices located on different computers coupled together via a network,

and wherein said origination storage device resides on a first computer and said destination storage device resides on a second computer, wherein the dedicated memory includes a predetermined number of dedicated buffers and further comprises;

a first memory having a first pool of buffers resident on said first computer and a second memory having a second pool of buffers resident on said second computer, wherein said master control module includes a first master controller resident on said first computer, registering and controlling processes resident on said first computer, a second controller resident on said second computer for registering and controlling those processes resident on said second computer, wherein said first memory area is shared among the processes registered on said first computer and said second memory area is shared among the processes registered on said second computer.

2. The data transfer system of claim 1 wherein processes resident on said first computer are operable to allocate and free buffers only within said first dedicated shared memory and where processes resident on said second computer am operable to allocate and free buffers only within said second dedicated shared memory.

3. A data transfer system for providing reliable high speed data transfer between computers, the data transfer system comprising:

an origination storage device;

a destination storage device that receives data from the origination storage device via a data transfer pipeline;

a data transfer pipeline apparatus that transfers data in a seguence of stages, the data transfer pipeline apparatus comprising;

a data compression module for compressing data transferred through at least a portion of the data transfer pipeline apparatus,

a dedicated memory with a predetermined number of buffers, and

a master control module for registering and controlling processes associated with the data transfer pipeline apparatus for participation in the data transfer pipeline apparatus, wherein said master control module registers multiple instantiations of the same process in a given stage of said data transfer sequence; and

wherein said processes further include at least one intermediate stage process registered with said master control module and sequentially coupled between a first stage process and a last stage process, for receiving buffers of data, processing said buffers, and sending said processed buffers of data to a next stage process, wherein each said process includes an input queue for receiving or allocating a buffer and an output queue for sending or freeing a buffer,

and wherein said dedicated memory further includes a plurality of reader/writer semaphores, each said semaphore associated with a particular input/output queue for controlling access to the dedicated memory.

4. The data transfer system of claim 3 wherein said pool of buffers in each of said first and second memory areas comprise buffers of equal size.

5. A data transfer system for providing reliable high speed data transfer between computers, the data transfer system comprising:

an origination storage device;

a destination storage device that receives data from the origination storage device via a data transfer pipeline;

a data transfer pipeline apparatus that transfers data in a seguence of stages, the data transfer pipeline apparatus comprising;

a data compression module for compressing data transferred through at least a portion of the data transfer pipeline apparatus,

a dedicated memory with a predetermined number of buffers, and

a master control module for registering and controlling processes associated with the data transfer pipeline apparatus for participation in the data transfer pipeline apparatus, wherein said master control module registers multiple instantiations of the same process in a given stage of said data transfer seguence; and

wherein said processes further include at least one intermediate stage process registered with said master control module and sequentially coupled between a first stage process and a last stage process, for receiving buffers of data, processing said buffers, and sending said processed buffers of data to a next stage process, wherein each said process includes an input queue for receiving or allocating a buffer and an output gueue for sending or freeing a buffer, wherein the input queue of a process is the output queue of a preceding process,

and wherein said at least one intermediate stage process includes;

a sequence process coupled between a single instantiation of a process for a given stage, and multiple instantiations of a same process for another stage, for stamping each buffer received from said single instantiation process with a sequence number prior to sending to said multiple instantiations of processes, and

a re-sequence processor coupled to the output queue of said multiple instantiations of processes for reordering the buffer sequence received according to the sequence number to provide a correct sequencing of the buffers for further processing.

6. A data transfer system for providing reliable high speed data transfer between computers, the data transfer system comprising:

an origination storage device;

a destination storage device that receives data from the origination storage device via a data transfer pipeline; and

a data transfer pipeline apparatus that transfers data in a sequence of stages, the data transfer pipeline apparatus comprising;

a data compression module for compressing data transferred through at least a portion of the data transfer pipeline apparatus,

a dedicated memory with a predetermined number of buffers, and

a master control module for registering and controlling processes associated with the data transfer pipeline apparatus for participation in the data transfer pipeline apparatus, wherein said master control module is initiated via a request message from a requesting application process, said request message including a process identification, timestamp, and machine name where said requesting application is executing.