Minimally buffered data transfers between nodes in a data communications network

• US 9,065,839 B2
• Filed: 10/02/2007
• Issued: 06/23/2015
• Est. Priority Date: 10/02/2007
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Miller, Douglas R.
• Primary Examiner(s): GEORGANDELLIS, ANDREW C
• Application Number:
  US11/865,981
• Publication Number:
  US 20090089328A1
• Time in Patent Office: 2,821 Days
• Field of Search: 709/232
• US Class Current: 1/1
• CPC Class Codes: G06F 9/546 Message passing systems or structures, e.g. queues
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Methods, apparatus, and products for minimally buffered data transfers between nodes in a data communications network are disclosed that include: receiving, by a messaging module on an origin node, a storage identifier, a origin data type, and a target data type, the storage identifier specifying application storage containing data, the origin data type describing a data subset contained in the origin application storage, the target data type describing an arrangement of the data subset in application storage on a target node; creating, by the messaging module, origin metadata describing the origin data type; selecting, by the messaging module from the origin application storage in dependence upon the origin metadata and the storage identifier, the data subset; and transmitting, by the messaging module to the target node, the selected data subset for storing in the target application storage in dependence upon the target data type without temporarily buffering the data subset.

Claims

1. A method for minimally buffered data transfers between nodes in a data communications network, the method comprising:

receiving, by an origin messaging module on an origin node from an application on the origin node, a storage identifier, a complex origin data type, and a complex target data type, the storage identifier specifying origin application storage containing data on the origin node, the complex origin data type describing a subset of the data contained in the origin application storage, the complex target data type describing an arrangement of the subset of the data in target application storage on a target node;

creating, by the origin messaging module, origin metadata describing the complex origin data type;

selecting, by the origin messaging module from the origin application storage in dependence upon the origin metadata and the storage identifier, the subset of the data for transmission to the target node; and

transmitting, by the origin messaging module to a target messaging module on the target node, the selected subset of the data for storing in the target application storage in dependence upon the complex target data type without any type of buffering the subset of the data in a temporary buffer on the target node.

2. The method of claim 1 wherein:

the complex target data type is the same as the complex origin data type; and

transmitting, by the origin messaging module to a target messaging module on the target node, the selected subset of the data for storing in the target application storage in dependence upon the complex target data type without buffering the subset of the data in a temporary buffer on the target node further comprises;

contiguously storing the selected subset of the data in a temporary buffer on the origin node; and

transmitting, from the temporary buffer on the origin node in a single data transfer operation, the subset of the data along with the origin metadata describing the complex origin data type.

3. The method of claim 1 wherein:

the method further comprises creating, by the origin messaging module, target metadata describing the complex target data type; and

transmitting, by the origin messaging module to a target messaging module on the target node, the selected subset of the data for storing in the target application storage in dependence upon the complex target data type without buffering the subset of the data in a temporary buffer on the target node further comprises;

contiguously storing the selected subset of the data in a temporary buffer on the origin node, and

transmitting, from the temporary buffer on the origin node in multiple data transfer operations, the selected subset of the data according to the target metadata.

4. The method of claim 1 wherein:

the method further comprises creating, by the origin messaging module, target metadata describing the complex target data type; and

transmitting, by the origin messaging module to a target messaging module on the target node, the selected subset of the data for storing in the target application storage in dependence upon the complex target data type without buffering the subset of the data in a temporary buffer on the target node further comprises transmitting, from the origin application storage in multiple data transfer operations, the selected subset of the data according to both the origin metadata and the target metadata.

5. The method of claim 1 wherein the origin metadata specifies, for each contiguous chunk of data described by the complex origin data type, an offset describing a location of the contiguous chunk in the origin application storage and a length describing the size of the contiguous chunk.

6. The method of claim 1 wherein the origin compute node and the target compute node are comprised in a parallel computer, the parallel computer comprising a plurality of compute nodes connected for data communications through a plurality of data communications networks, at least one of the plurality of data communications networks optimized for point to point data communications, and at least one of the plurality of data communications networks optimized for collective operations.

7. An apparatus for minimally buffered data transfers between nodes in a data communications network, the apparatus comprising a computer processor and computer memory operatively coupled to the computer processor, the computer memory having disposed within it computer program instructions capable of:

receiving, by an origin messaging module on an origin node from an application on the origin node, a storage identifier, a complex origin data type, and a complex target data type, the storage identifier specifying origin application storage containing data on the origin node, the complex origin data type describing a subset of the data contained in the origin application storage, the complex target data type describing an arrangement of the subset of the data in target application storage on a target node;

creating, by the origin messaging module, origin metadata describing the complex origin data type;

selecting, by the origin messaging module from the origin application storage in dependence upon the origin metadata and the storage identifier, the subset of the data for transmission to the target node; and

transmitting, by the origin messaging module to a target messaging module on the target node, the selected subset of the data for storing in the target application storage in dependence upon the complex target data type without any type of buffering the subset of the data in a temporary buffer on the target node.

8. The apparatus of claim 7 wherein:

the complex target data type is the same as the complex origin data type; and

transmitting, by the origin messaging module to a target messaging module on the target node, the selected subset of the data for storing in the target application storage in dependence upon the complex target data type without buffering the subset of the data in a temporary buffer on the target node further comprises;

contiguously storing the selected subset of the data in a temporary buffer on the origin node; and

transmitting, from the temporary buffer on the origin node in a single data transfer operation, the subset of the data along with the origin metadata describing the complex origin data type.

9. The apparatus of claim 7 wherein:

the computer memory has disposed within it computer program instructions capable of creating, by the origin messaging module, target metadata describing the complex target data type; and

transmitting, by the origin messaging module to a target messaging module on the target node, the selected subset of the data for storing in the target application storage according to the target metadata to select the subset of data to be transmitted without buffering the subset of the data in a temporary buffer on the target node further comprises;

contiguously storing the selected subset of the data in a temporary buffer on the origin node, and

transmitting, from the temporary buffer on the origin node in multiple data transfer operations, the selected subset of the data according to the target metadata.

10. The apparatus of claim 7 wherein:

the computer memory has disposed within it computer program instructions capable of creating, by the origin messaging module, target metadata describing the complex target data type; and

transmitting, by the origin messaging module to a target messaging module on the target node, the selected subset of the data for storing in the target application storage in dependence upon the complex target data type without buffering the subset of the data in a temporary buffer on the target node further comprises transmitting, from the origin application storage in multiple data transfer operations, the selected subset of the data according to both the origin metadata and the target metadata.

11. The apparatus of claim 7 wherein the origin metadata specifies, for each contiguous chunk of data described by the complex origin data type, an offset describing a location of the contiguous chunk in the origin application storage and a length describing the size of the contiguous chunk.

12. The apparatus of claim 7 wherein the origin compute node and the target compute node are comprised in a parallel computer, the parallel computer comprising a plurality of compute nodes connected for data communications through a plurality of data communications networks, at least one of the plurality of data communications networks optimized for point to point data communications, and at least one of the plurality of data communications networks optimized for collective operations.

13. A computer program product for minimally buffered data transfers between nodes in a data communications network, the computer program product disposed upon a computer readable, recordable medium, the computer program product comprising computer program instructions capable of:

receiving, by an origin messaging module on an origin node from an application on the origin node, a storage identifier, a complex origin data type, and a complex target data type, the storage identifier specifying origin application storage containing data on the origin node, the complex origin data type describing a subset of the data contained in the origin application storage, the complex target data type describing an arrangement of the subset of the data in target application storage on a target node;

creating, by the origin messaging module, origin metadata describing the complex origin data type;

selecting, by the origin messaging module from the origin application storage in dependence upon the origin metadata and the storage identifier, the subset of the data for transmission to the target node; and

transmitting, by the origin messaging module to a target messaging module on the target node, the selected subset of the data for storing in the target application storage in dependence upon the complex target data type without any type of buffering the subset of the data in a temporary buffer on the target node.

14. The computer program product of claim 13 wherein:

the complex target data type is the same as the complex origin data type; and

transmitting, by the origin messaging module to a target messaging module on the target node, the selected subset of the data for storing in the target application storage in dependence upon the complex target data type without buffering the subset of the data in a temporary buffer on the target node further comprises;

contiguously storing the selected subset of the data in a temporary buffer on the origin node; and

transmitting, from the temporary buffer on the origin node in a single data transfer operation, the subset of the data along with the origin metadata describing the complex origin data type.

15. The computer program product of claim 13 wherein:

the computer program product further comprises computer program instructions capable of creating, by the origin messaging module, target metadata describing the complex target data type; and

transmitting, by the origin messaging module to a target messaging module on the target node, the selected subset of the data for storing in the target application storage according to the target metadata without buffering the subset of the data in a temporary buffer on the target node further comprises;

contiguously storing the selected subset of the data in a temporary buffer on the origin node, and

transmitting, from the temporary buffer on the origin node in multiple data transfer operations, the selected subset of the data according to the target metadata.

16. The computer program product of claim 13 wherein:

the computer program product further comprises computer program instructions capable of creating, by the origin messaging module, target metadata describing the complex target data type; and

transmitting, by the origin messaging module to a target messaging module on the target node, the selected subset of the data for storing in the target application storage in dependence upon the complex target data type without buffering the subset of the data in a temporary buffer on the target node further comprises transmitting, from the origin application storage in multiple data transfer operations, the selected subset of the data according to both the origin metadata and the target metadata.

17. The computer program product of claim 13 wherein the origin metadata specifies, for each contiguous chunk of data described by the complex origin data type, an offset describing a location of the contiguous chunk in the origin application storage and a length describing the size of the contiguous chunk.

18. The computer program product of claim 13 wherein the origin compute node and the target compute node are comprised in a parallel computer, the parallel computer comprising a plurality of compute nodes connected for data communications through a plurality of data communications networks, at least one of the plurality of data communications networks optimized for point to point data communications, and at least one of the plurality of data communications networks optimized for collective operations.