Pipeline systems and method for transferring data in a network environment

• US 8,326,915 B2
• Filed: 06/10/2011
• Issued: 12/04/2012
• Est. Priority Date: 10/30/1997
• Status: Expired due to Fees

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Ignatius, Paul, Prahlad, Anand, Tyagarajan, Mahesh
• Primary Examiner(s): Strange, Aaron
• Assistant Examiner(s): EDWARDS, JAMES A
• Application Number:
  US13/158,222
• Publication Number:
  US 20110238777A1
• Time in Patent Office: 543 Days
• Field of Search: 709/202
• US Class Current: 709/202
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 12/02 Addressing or allocation; Relocation program address sequencing G06F9/00; arrangements for selecting an address in a digital store G11C8/00
• CPC Class Codes: G06F 3/0613 in relation to throughput
• CPC Class Codes: G06F 3/0656 Data buffering arrangements
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
• CPC Class Codes: G06F 9/5033 considering data affinity
• CPC Class Codes: G06F 9/544 Buffers; Shared memory; Pipes
• CPC Class Codes: H04L 69/04 Protocols for data compression, e.g. ROHC

Abstract

A communications system having a data transfer pipeline apparatus for transferring data in a sequence of N stages from an origination device to a destination device. The apparatus comprises dedicated memory having buffers dedicated for carrying data and a master control for registering and controlling processes associated with the apparatus for participation in the N stage data transfer sequence. The processes include a first stage process for initiating the data transfer and a last Nth stage process for completing data transfer. The first stage process allocates a buffer from a predetermined number of buffers available within the memory for collection, processing, and sending of the data from the origination device to a next stage process. The Nth stage process receives a buffer allocated to the first stage process from the (N−1)th stage and to free the buffer upon processing completion to permit reallocation of the buffer.

Claims

1. A pipeline system for providing data transfer between multiple computing devices, the pipeline system comprising:

a datapipe that spans multiple computing devices, the datapipe comprising a sequence of stages for transferring data from an origination computing device to a destination computing device, wherein the datapipe is identified on the origination computing device and the destination computing device with a data pipeline identifier;

one or more control modules configured to control at least a first stage of the sequence of stages on the origination computing device;

a first dedicated memory comprising a first pool of buffers, wherein the one or more control modules allocate at least a first buffer from the first pool of buffers to the first stage, and wherein the first buffer is associated with the data pipeline identifier until freed by the one or more control modules;

one or more of the control modules further configured to control at least a second stage of the sequence of stages on the destination computing device; and

a second dedicated memory comprising a second pool of buffers, wherein the one or more control modules allocate at least a second buffer from the second pool of buffers to the second stage, and wherein the second buffer is associated with the data pipeline identifier until freed by the one or more control modules.

2. The pipeline system of claim 1, wherein one of the stages in the sequence of stages comprises data compression.

3. The pipeline system of claim 1, wherein the first stage comprises:

an input queue for receiving or allocating at least the first buffer of the first pool of buffers; and

an output queue for sending or freeing the first buffer.

4. The pipeline system of claim 3, comprising an intermediate stage coupled to the first stage for stamping each buffer received from the first stage process with a sequence number prior to sending to a next stage.

5. The pipeline system of claim 4, wherein a subsequent stage includes a re-sequence processor reordering a buffer sequence received according to the sequence number.

6. The pipeline system of claim 1, wherein each of the first dedicated memory and the second dedicated memory further includes a plurality of semaphores each associated with a particular input/output queue for controlling access to the associated dedicated memory.

7. The pipeline system of claim 1, wherein the pool of buffers in each of the first dedicated memory and the second dedicated memory comprises buffers of equal size.

8. The pipeline system of claim 1, wherein a first control module is initiated via a request message from a requesting application process, the request message including a process identification and a timestamp.

9. A method for transferring data in a pipeline system, the method comprising:

registering and initiating a plurality of pipeline stages associated with a data transfer pipeline that spans multiple computing devices, wherein the data transfer pipeline is identified on an origination computing device and a destination computing device with a data pipeline identifier;

controlling at least a first stage of the sequence of stages on the origination computing device;

allocating at least a first buffer from a first pool of buffers in a first dedicated memory to the first stage of the plurality of pipeline stages, wherein the first buffer is associated with the data pipeline identifier until freed;

controlling at least a second stage of the sequence of stages on the destination computing device; and

allocating at least a second buffer from a second pool of buffers in a second dedicated memory to the second stage of the plurality of pipeline stages, wherein the second buffer is associated with the data pipeline identifier until freed.

10. The method of claim 9, additionally comprising transferring control of multiple ones of the buffers through the use of semaphores.

11. The method of claim 9, wherein the plurality of pipeline stages comprises a compression process and an encryption process.

12. The method of claim 9, additionally comprising storing data from the pool of second buffers to a storage device.

13. The method of claim 9, additionally comprising determining a number of buffers in the first pool of buffers to be allocated to the first stage.

14. The method of claim 13, additionally comprising terminating the first stage when the number of buffers allocated to the first stage exceeds a threshold amount.