Pipeline systems and method for transferring data in a network environment
First Claim
1. A pipeline system for providing data transfer between multiple computing devices, the pipeline system comprising:
- a datapipe that spans multiple computing devices, the datapipe comprising a sequence of stages for transferring data from an origination computing device to a destination computing device, wherein the datapipe is identified on the origination computing device and the destination computing device with a data pipeline identifier;
a first control module resident on the origination computing device and configured to register and control a plurality of first processes resident on the origination computing device, the first control module being further configured to register multiple instantiations of the same first process in a given stage of the sequence of stages, wherein the multiple instantiations are associated with the datapipe identifier;
a second control module resident on the destination computing device and configured to register and control a plurality of second processes resident on the destination computing device, wherein at least one of the second processes on the destination computer are associated with the datapipe identifier;
a first dedicated memory comprising a first pool of buffers resident on the origination computing device, wherein the first dedicated memory is shared among the plurality of first processes registered on the origination computing device, wherein the first control module allocates a first sequence of buffers from the first pool of buffers in the origination computing device to the plurality of first processes registered on the origination computing device, and wherein the first sequence of buffers is associated with the datapipe identifier until the first sequence of buffers is freed by the first control module;
a second dedicated memory comprising a second pool of buffers resident on the destination computing device, wherein the second dedicated memory is shared among the plurality of second processes registered on the destination computing device, wherein the second control module allocates a second sequence of buffers from the second pool of buffers in the destination computing device to the plurality of second processes registered on the destination computing device, and wherein the second sequence of buffers is associated with the datapipe identifier until the second sequence of buffers is freed by the second control module; and
a first network control module and a second network control module coupled, respectively, to the first and second control modules and configured to initiate network agent processes to enable data transfer communications between the origination computing device and the destination device.
3 Assignments
0 Petitions
Accused Products
Abstract
A communications system having a data transfer pipeline apparatus for transferring data in a sequence of N stages from an origination device to a destination device. The apparatus comprises dedicated memory having buffers dedicated for carrying data and a master control for registering and controlling processes associated with the apparatus for participation in the N stage data transfer sequence. The processes include a first stage process for initiating the data transfer and a last Nth stage process for completing data transfer. The first stage process allocates a buffer from a predetermined number of buffers available within the memory for collection, processing, and sending of the data from the origination device to a next stage process. The Nth stage process receives a buffer allocated to the first stage process from the (N−1)th stage and to free the buffer upon processing completion to permit reallocation of the buffer.
-
Citations
20 Claims
-
1. A pipeline system for providing data transfer between multiple computing devices, the pipeline system comprising:
-
a datapipe that spans multiple computing devices, the datapipe comprising a sequence of stages for transferring data from an origination computing device to a destination computing device, wherein the datapipe is identified on the origination computing device and the destination computing device with a data pipeline identifier; a first control module resident on the origination computing device and configured to register and control a plurality of first processes resident on the origination computing device, the first control module being further configured to register multiple instantiations of the same first process in a given stage of the sequence of stages, wherein the multiple instantiations are associated with the datapipe identifier; a second control module resident on the destination computing device and configured to register and control a plurality of second processes resident on the destination computing device, wherein at least one of the second processes on the destination computer are associated with the datapipe identifier; a first dedicated memory comprising a first pool of buffers resident on the origination computing device, wherein the first dedicated memory is shared among the plurality of first processes registered on the origination computing device, wherein the first control module allocates a first sequence of buffers from the first pool of buffers in the origination computing device to the plurality of first processes registered on the origination computing device, and wherein the first sequence of buffers is associated with the datapipe identifier until the first sequence of buffers is freed by the first control module; a second dedicated memory comprising a second pool of buffers resident on the destination computing device, wherein the second dedicated memory is shared among the plurality of second processes registered on the destination computing device, wherein the second control module allocates a second sequence of buffers from the second pool of buffers in the destination computing device to the plurality of second processes registered on the destination computing device, and wherein the second sequence of buffers is associated with the datapipe identifier until the second sequence of buffers is freed by the second control module; and a first network control module and a second network control module coupled, respectively, to the first and second control modules and configured to initiate network agent processes to enable data transfer communications between the origination computing device and the destination device. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10)
-
-
11. A method for transferring data in a pipeline system, the method comprising:
-
registering and initiating a plurality of pipeline processes associated with a data transfer pipeline that spans multiple computing devices, the data transfer pipeline comprising a plurality of stages, wherein the data transfer pipeline is identified on the origination computing device and the destination computing device with a data pipeline identifier; initiating network agent processes to enable data transfer communications between a first computing device and a second computing device; allocating a first dedicated memory having a pool of first buffers on the first computing device and sharing the first dedicated memory with a plurality of first processes of the pipeline processes residing on the first computing device, wherein a first sequence of first buffers from the pool of first buffers are allocated to the plurality of first processes, and wherein the first sequence of first buffers are associated with the data pipeline identifier until the first sequence of first buffers are freed; allocating a second dedicated memory having a pool of second buffers on the second computing device and sharing the second dedicated memory with a plurality of second processes of the pipeline processes residing on the second computing device, wherein at least one of the second processes are associated with the data pipeline identifier, and wherein a second sequence of second buffers from the pool of second buffers are allocated to the plurality of second processes, and wherein the second sequence of second buffers are associated with the data pipeline identifier until the second sequence of second buffers are freed; registering multiple instantiations of at least one of the first processes with a given stage of the plurality of stages in the data transfer pipeline, wherein the multiple instantiations are associated with the data pipeline identifier; and transferring data through the plurality of stages from the first computing device to the second computing device. - View Dependent Claims (12, 13, 14, 15, 16, 17)
-
-
18. A pipeline system for providing data transfer between multiple computing devices, the pipeline system comprising:
-
means for compressing data being transferred from an origination computing device in a sequence of stages to a destination computing device with a data transfer pipeline that spans the origination computing device and the destination computing device, wherein the data transfer pipeline is identified on the origination computing device and the destination computing device with a data pipeline identifier; first means for registering and controlling a plurality of first processes resident on the origination computing device, said first registering and controlling means residing on the origination computing device; second means for registering and controlling a plurality of second processes resident on the destination computing device, said second registering and controlling means residing on the destination computing device; first dedicated means for storing data, said first dedicated storing means residing on the origination computing device and being shared among the plurality of first processes registered on the origination computing device, wherein the first dedicated means for storing data comprises a first sequence of first buffers from a pool of first buffers, and wherein the first sequence of first buffers is associated with the data pipeline identifier until the first sequence of first buffers is freed; second dedicated means for storing data, said second dedicated storing means residing on the destination computing device and being shared among the plurality of second processes registered on the destination computing device, wherein the second dedicated means for storing data comprises a second sequence of second buffers from a pool of second buffers, and wherein the second sequence of second buffers is associated with the data pipeline identifier until the second sequence of second buffers is freed; and means for initiating data transfer communications between the origination computing device and the destination device, and wherein at least one of said first and second registering and controlling means is further configured to register multiple instantiations of a particular process in a given stage of the sequence of stages, wherein the multiple instantiations are associated with the data pipeline identifier. - View Dependent Claims (19, 20)
-
Specification