PIPELINE SYSTEMS AND METHOD FOR TRANSFERRING DATA IN A NETWORK ENVIRONMENT

• US 20080256173A1
• Filed: 06/26/2008
• Published: 10/16/2008
• Est. Priority Date: 10/30/1997
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Prahlad, Anand, Tyagarajan, Mahesh, Ignatius, Paul
• Granted Patent: US 7,962,642 B2
• Time in Patent Office: Days
• US Class Current: 709/202
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 12/02 Addressing or allocation; Relocation program address sequencing G06F9/00; arrangements for selecting an address in a digital store G11C8/00
• CPC Class Codes: G06F 3/0613 in relation to throughput
• CPC Class Codes: G06F 3/0656 Data buffering arrangements
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
• CPC Class Codes: G06F 9/5033 considering data affinity
• CPC Class Codes: G06F 9/544 Buffers; Shared memory; Pipes
• CPC Class Codes: H04L 69/04 Protocols for data compression, e.g. ROHC

Abstract

A communications system having a data transfer pipeline apparatus for transferring data in a sequence of N stages from an origination device to a destination device. The apparatus comprises dedicated memory having buffers dedicated for carrying data and a master control for registering and controlling processes associated with the apparatus for participation in the N stage data transfer sequence. The processes include a first stage process for initiating the data transfer and a last Nth stage process for completing data transfer. The first stage process allocates a buffer from a predetermined number of buffers available within the memory for collection, processing, and sending of the data from the origination device to a next stage process. The Nth stage process receives a buffer allocated to the first stage process from the (N−1)th stage and to free the buffer upon processing completion to permit reallocation of the buffer.

Claims

1. A pipeline system for providing data transfer between multiple computing devices, the pipeline system comprising:

a sequence of stages for transferring data from an origination computing device to a destination computing device;

a first control module resident on the origination computing device and configured to register and control a plurality of first processes resident on the origination computing device, the first control module being further configured to register multiple instantiations of the same first process in a given stage of the sequence of stages;

a second control module resident on the destination computing device and configured to register and control a plurality of second processes resident on the destination computing device;

a first dedicated memory comprising a first pool of buffers resident on the origination computing device, wherein the first dedicated memory is shared among the plurality of first processes registered on the origination computing device;

a second dedicated memory comprising a second pool of buffers resident on the destination computing device, wherein the second dedicated memory is shared among the plurality of second processes registered on the destination computing device; and

a first network control module and a second network control module coupled, respectively, to the first and second control modules and configured to initiate network agent processes to enable data transfer communications between the origination computing device and the destination device.

2. The pipeline system of claim 1, wherein the same first process comprises a data compression process.

3. The pipeline system of claim 1, wherein the plurality of first processes further includes an intermediate stage process registered with the first control module and coupled between a first stage process and a last stage process, for receiving buffers of data, processing the buffers of data, and sending the processed buffers of data to a next stage process.

4. The pipeline system of claim 3, wherein each of the plurality of first processes includes:

an input queue for receiving or allocating at least one buffer of the first pool of buffers; and

an output queue for sending or freeing the at least one buffer.

5. The pipeline system of claim 3, wherein the intermediate stage process includes a sequence process coupled between a single instantiation of the first stage process and multiple instantiations of the next stage process, for stamping each buffer received from the first stage process with a sequence number prior to sending to the multiple instantiations of the next stage process.

6. The pipeline system of claim 5, wherein the intermediate stage process further includes a re-sequence processor coupled to the output queue of the multiple instantiations of the next stage process for reordering a buffer sequence received according to the sequence number.

7. The pipeline system of claim 1, wherein each of the first dedicated memory and the second dedicated memory further includes a plurality of semaphores each associated with a particular input/output queue for controlling access to the associated dedicated memory.

8. The pipeline system of claim 1, wherein the pool of buffers in each of the first dedicated memory and the second dedicated memory comprises buffers of equal size.

9. The pipeline system of claim 1, wherein the first control module is initiated via a request message from a requesting application process, the request message including a process identification and a timestamp.

10. The pipeline system of claim 1, wherein the first pool of buffers is configured to be locked against further allocation until freed by the first process to which the first pool of buffers is assigned.

11. A method for transferring data in a pipeline system, the method comprising:

registering and initiating a plurality of pipeline processes associated with a data transfer pipeline having a plurality of stages;

initiating network agent processes to enable data transfer communications between a first computing device and a second computing device;

allocating a first dedicated memory having a pool of first buffers on the first computing device and sharing the first dedicated memory with a plurality of first processes of the pipeline processes residing on the first computing device;

allocating a second dedicated memory having a pool of second buffers on the second computing device and sharing the second dedicated memory with a plurality of second processes of the pipeline processes residing on the second computing device;

registering multiple instantiations of at least one of the first processes with a given stage of the plurality of stages in the data transfer pipeline; and

transferring data through the plurality of stages from the first computing device to the second computing device.

12. The method of claim 11, wherein said sharing the first dedicated memory comprises sharing the multiple instantiations of the at least one first process with multiple ones of the first buffers in the given stage of the data transfer pipeline.

13. The method of claim 12, additionally comprising transferring control of the multiple ones of the first buffers through the use of semaphores.

14. The method of claim 11, wherein the plurality of pipeline processes comprises a compression process and an encryption process.

15. The method of claim 11, additionally comprising storing data from the pool of second buffers to a storage device.

16. The method of claim 11, additionally comprising determining of a number of first buffers requested for allocation to one of the plurality of first processes.

17. The method of claim 16, additionally comprising terminating the one first process if the number of first buffers requested exceeds a threshold amount.

18. A pipeline system for providing data transfer between multiple computing devices, the pipeline system comprising:

means for compressing data being transferred from an origination computing device in a sequence of stages to a destination computing device;

first means for registering and controlling a plurality of first processes resident on the origination computing device, said first registering and controlling means residing on the origination computing device;

second means for registering and controlling a plurality of second processes resident on the destination computing device, said second registering and controlling means residing on the destination computing device;

first dedicated means for storing data, said first dedicated storing means residing on the origination computing device and being shared among the plurality of first processes registered on the origination computing device;

second dedicated means for storing data, said second dedicated storing means residing on the destination computing device and being shared among the plurality of second processes registered on the destination computing device; and

means for initiating data transfer communications between the origination computing device and the destination device,

and wherein at least one of said first and second registering and controlling means is further configured to register multiple instantiations of a particular process in a given stage of the sequence of stages.

19. The pipeline system of claim 18, further comprising means for allocating a sequence of buffers from said first dedicated storing means, the sequence of buffers being locked against reallocation until freed by the particular process to which the sequence of buffers is assigned.

20. The pipeline system of claim 19, further comprising means for storing data resident in the sequence of buffers in the destination computing device and for freeing the sequence of buffers for reallocation.