Fencing direct memory access data transfers in a parallel active messaging interface of a parallel computer

• US 8,527,672 B2
• Filed: 11/05/2010
• Issued: 09/03/2013
• Est. Priority Date: 11/05/2010
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Blocksome, Michael A., Mamidala, Amith R.
• Primary Examiner(s): Alrobaye, Idriss N
• Assistant Examiner(s): Franklin, Richard B
• Application Number:
  US12/940,300
• Publication Number:
  US 20120117281A1
• Time in Patent Office: 1,033 Days
• Field of Search: None
• US Class Current: 710/22
• CPC Class Codes: G06F 13/28 using burst mode transfer, e.g. direct memory access DMA, cycle steal G06F13/32 takes precedence

Abstract

Fencing direct memory access (‘DMA’) data transfers in a parallel active messaging interface (‘PAMI’) of a parallel computer, the PAMI including data communications endpoints, each endpoint including specifications of a client, a context, and a task, the endpoints coupled for data communications through the PAMI and through DMA controllers operatively coupled to segments of shared random access memory through which the DMA controllers deliver data communications deterministically, including initiating execution through the PAMI of an ordered sequence of active DMA instructions for DMA data transfers between two endpoints, effecting deterministic DMA data transfers through a DMA controller and a segment of shared memory; and executing through the PAMI, with no FENCE accounting for DMA data transfers, an active FENCE instruction, the FENCE instruction completing execution only after completion of all DMA instructions initiated prior to execution of the FENCE instruction for DMA data transfers between the two endpoints.

Claims

1. A method of fencing direct memory access (‘

DMA’

) data transfers in a parallel active messaging interface (‘

PAMI’

) of a parallel computer, the parallel computer comprising a plurality of compute nodes that execute a parallel application, the PAMI comprising data communications endpoints, each endpoint comprising a specification of data communications parameters for a thread of execution on a compute node, including specifications of a client, a context, and a task, the compute nodes and the endpoints coupled for data communications through the PAMI and through data communications resources including DMA controllers operatively coupled to segments of shared random access memory through which the DMA controllers deliver data communications deterministically, in the same order in which the communications are transmitted, the method comprising;

initiating execution through the PAMI of an ordered sequence of active DMA instructions for DMA data transfers between two endpoints, an origin endpoint and a target endpoint, each DMA instruction effecting a deterministic DMA data transfer through a DMA controller and a segment of shared memory in which the DMA data transfers are effected according to the ordered sequence of the DMA instructions; and

executing through the PAMI, with no FENCE accounting for DMA data transfers, an active FENCE instruction, the FENCE instruction completing execution only after completion of all DMA instructions initiated prior to execution of the FENCE instruction for DMA data transfers between the two endpoints;

wherein the DMA instructions comprise DMA PUT instructions, and executing a FENCE instruction further comprises;

receiving in the origin endpoint the FENCE instruction, the FENCE instruction specifying a FENCE done callback function;

transmitting the FENCE instruction through the segment of shared memory from the origin endpoint to the target endpoint, the FENCE instruction specifying a FENCE dispatch callback function;

processing, by the target endpoint after completion of all PUT data transfers previously initiated between the two endpoints, the FENCE instruction, including executing the FENCE dispatch callback function and advising the origin endpoint of the processing of the FENCE instruction; and

executing by the origin endpoint the FENCE done callback function.

2. The method of claim 1 wherein:

each client comprises a collection of data communications resources dedicated to the exclusive use of an application-level data processing entity;

each context comprises a subset of the collection of data processing resources of a client, context functions, and a work queue of data transfer instructions to be performed by use of the subset through the context functions operated by an assigned thread of execution; and

each task represents a process of execution of the parallel application.

3. The method of claim 1 wherein each context carries out, through post and advance functions, data communications for the parallel application on data communications resources in the exclusive possession of that context.

4. The method of claim 1 wherein each context carries out data communications operations independently and in parallel with other contexts.

5. A method of fencing direct memory access (‘

DMA’

) data transfers in a parallel active messaging interface (‘

PAMI’

) of a parallel computer, the parallel computer comprising a plurality of compute nodes that execute a parallel application, the PAMI comprising data communications endpoints, each endpoint comprising a specification of data communications parameters for a thread of execution on a compute node, including specifications of a client, a context, and a task, the compute nodes and the endpoints coupled for data communications through the PAMI and through data communications resources including DMA controllers operatively coupled to segments of shared random access memory through which the DMA controllers deliver data communications deterministically, in the same order in which the communications are transmitted, the method comprising;

initiating execution through the PAMI of an ordered sequence of active DMA instructions for DMA data transfers between two endpoints, an origin endpoint and a target endpoint, each DMA instruction effecting a deterministic DMA data transfer through a DMA controller and a segment of shared memory in which the DMA data transfers are effected according to the ordered sequence of the DMA instructions; and

executing through the PAMI, with no FENCE accounting for DMA data transfers, an active FENCE instruction, the FENCE instruction completing execution only after completion of all DMA instructions initiated prior to execution of the FENCE instruction for DMA data transfers between the two endpoints;

wherein the DMA instructions comprise DMA GET instructions, and executing a FENCE instruction further comprises;

receiving in the origin endpoint the FENCE instruction, the FENCE instruction specifying a FENCE done callback function;

transmitting, after completion of all GET data transfers previously initiated between the two endpoints, the FENCE instruction through the segment of shared memory from the origin endpoint to the target endpoint, the FENCE instruction specifying a FENCE dispatch callback function;

processing the FENCE instruction by the target endpoint, including executing the FENCE dispatch callback function and advising the origin endpoint of the processing of the FENCE instruction; and

executing by the origin endpoint the FENCE done callback function.

6. A parallel computer that fences direct memory access (‘

DMA’

) data transfers in a parallel active messaging interface (‘

PAMI’

), the parallel computer comprising a plurality of compute nodes that execute a parallel application, the PAMI comprising data communications endpoints on the compute nodes of the parallel computer, each endpoint comprising a specification of data communications parameters for a thread of execution on a compute node, including specifications of a client, a context, and a task, the compute nodes and the endpoints coupled for data communications through the PAMI and through data communications resources including DMA controllers operatively coupled to segments of shared random access memory through which the DMA controllers deliver data communications deterministically, in the same order in which the communications are transmitted, the compute nodes comprising computer processors operatively coupled to computer memory having disposed within it computer program instructions that, when executed by the computer processors, cause the parallel computer to function by;

initiating execution through the PAMI of an ordered sequence of active DMA instructions for DMA data transfers between two endpoints, an origin endpoint and a target endpoint, each DMA instruction effecting a deterministic DMA data transfer through a DMA controller and a segment of shared memory in which the DMA data transfers are effected according to the ordered sequence of the DMA instructions; and

executing through the PAMI, with no FENCE accounting for DMA data transfers, an active FENCE instruction, the FENCE instruction completing execution only after completion of all DMA instructions initiated prior to execution of the FENCE instruction for DMA data transfers between the two endpoints;

the DMA instructions comprise DMA PUT instructions, and executing a FENCE instruction further comprises;

receiving in the origin endpoint the FENCE instruction, the FENCE instruction specifying a FENCE done callback function;

transmitting the FENCE instruction through the segment of shared memory from the origin endpoint to the target endpoint, the FENCE instruction specifying a FENCE dispatch callback function;

processing, by the target endpoint after completion of all PUT data transfers previously initiated between the two endpoints, the FENCE instruction, including executing the FENCE dispatch callback function and advising the origin endpoint of the processing of the FENCE instruction; and

executing by the origin endpoint the FENCE done callback function.

7. The parallel computer of claim 6 wherein:

each client comprises a collection of data communications resources dedicated to the exclusive use of an application-level data processing entity;

each context comprises a subset of the collection of data processing resources of a client, context functions, and a work queue of data transfer instructions to be performed by use of the subset through the context functions operated by an assigned thread of execution; and

each task represents a process of execution of the parallel application.

8. The parallel computer of claim 6 wherein each context carries out, through post and advance functions, data communications for the parallel application on data communications resources in the exclusive possession of that context.

9. The parallel computer of claim 6 wherein each context carries out data communications operations independently and in parallel with other contexts.

10. A parallel computer that fences direct memory access (‘

DMA’

) data transfers in a parallel active messaging interface (‘

PAMI’

), the parallel computer comprising a plurality of compute nodes that execute a parallel application, the PAMI comprising data communications endpoints on the compute nodes of the parallel computer, each endpoint comprising a specification of data communications parameters for a thread of execution on a compute node, including specifications of a client, a context, and a task, the compute nodes and the endpoints coupled for data communications through the PAMI and through data communications resources including DMA controllers operatively coupled to segments of shared random access memory through which the DMA controllers deliver data communications deterministically, in the same order in which the communications are transmitted, the compute nodes comprising computer processors operatively coupled to computer memory having disposed within it computer program instructions that, when executed by the computer processors, cause the parallel computer to function by;

initiating execution through the PAMI of an ordered sequence of active DMA instructions for DMA data transfers between two endpoints, an origin endpoint and a target endpoint, each DMA instruction effecting a deterministic DMA data transfer through a DMA controller and a segment of shared memory in which the DMA data transfers are effected according to the ordered sequence of the DMA instructions; and

executing through the PAMI, with no FENCE accounting for DMA data transfers, an active FENCE instruction, the FENCE instruction completing execution only after completion of all DMA instructions initiated prior to execution of the FENCE instruction for DMA data transfers between the two endpoints;

wherein the DMA instructions comprise DMA GET instructions, and executing a FENCE instruction further comprises;

receiving in the origin endpoint the FENCE instruction, the FENCE instruction specifying a FENCE done callback function;

transmitting, after completion of all GET data transfers previously initiated between the two endpoints, the FENCE instruction through the segment of shared memory from the origin endpoint to the target endpoint, the FENCE instruction specifying a FENCE dispatch callback function;

processing the FENCE instruction by the target endpoint, including executing the FENCE dispatch callback function and advising the origin endpoint of the processing of the FENCE instruction; and

executing by the origin endpoint the FENCE done callback function.

11. A computer program product for fencing direct memory access (‘

DMA’

) data transfers in a parallel active messaging interface (‘

PAMI’

) of a parallel computer, the parallel computer comprising a plurality of compute nodes that execute a parallel application, the PAMI comprising data communications endpoints on the compute nodes of the parallel computer, each endpoint comprising a specification of data communications parameters for a thread of execution on a compute node, including specifications of a client, a context, and a task, the compute nodes and the endpoints coupled for data communications through the PAMI and through data communications resources including DMA controllers operatively coupled to segments of shared random access memory through which the DMA controllers deliver data communications deterministically, in the same order in which the communications are transmitted, the computer program product disposed upon a non-transitory computer readable storage medium, the computer program product comprising computer program instructions that, when installed and executed, cause the parallel computer to function by;

initiating execution through the PAMI of an ordered sequence of active DMA instructions for DMA data transfers between two endpoints, an origin endpoint and a target endpoint, each DMA instruction effecting a deterministic DMA data transfer through a DMA controller and a segment of shared memory in which the DMA data transfers are effected according to the ordered sequence of the DMA instructions; and

executing through the PAMI, with no FENCE accounting for DMA data transfers, an active FENCE instruction, the FENCE instruction completing execution only after completion of all DMA instructions initiated prior to execution of the FENCE instruction for DMA data transfers between the two endpoints;

wherein the DMA instructions comprise DMA PUT instructions, and executing a FENCE instruction further comprises;

receiving in the origin endpoint the FENCE instruction, the FENCE instruction specifying a FENCE done callback function;

transmitting the FENCE instruction through the segment of shared memory from the origin endpoint to the target endpoint, the FENCE instruction specifying a FENCE dispatch callback function;

processing, by the target endpoint after completion of all PUT data transfers previously initiated between the two endpoints, the FENCE instruction, including executing the FENCE dispatch callback function and advising the origin endpoint of the processing of the FENCE instruction; and

executing by the origin endpoint the FENCE done callback function.

12. The computer program product of claim 11 wherein:

each client comprises a collection of data communications resources dedicated to the exclusive use of an application-level data processing entity;

each context comprises a subset of the collection of data processing resources of a client, context functions, and a work queue of data transfer instructions to be performed by use of the subset through the context functions operated by an assigned thread of execution; and

each task represents a process of execution of the parallel application.

13. The computer program product of claim 11 wherein each context carries out, through post and advance functions, data communications for the parallel application on data communications resources in the exclusive possession of that context.

14. The computer program product of claim 11 wherein each context carries out data communications operations independently and in parallel with other contexts.

15. A computer program product for fencing direct memory access (‘

DMA’

) data transfers in a parallel active messaging interface (‘

PAMI’

) of a parallel computer, the parallel computer comprising a plurality of compute nodes that execute a parallel application, the PAMI comprising data communications endpoints on the compute nodes of the parallel computer, each endpoint comprising a specification of data communications parameters for a thread of execution on a compute node, including specifications of a client, a context, and a task, the compute nodes and the endpoints coupled for data communications through the PAMI and through data communications resources including DMA controllers operatively coupled to segments of shared random access memory through which the DMA controllers deliver data communications deterministically, in the same order in which the communications are transmitted, the computer program product disposed upon a non-transitory computer readable storage medium, the computer program product comprising computer program instructions that, when installed and executed, cause the parallel computer to function by;

initiating execution through the PAMI of an ordered sequence of active DMA instructions for DMA data transfers between two endpoints, an origin endpoint and a target endpoint, each DMA instruction effecting a deterministic DMA data transfer through a DMA controller and a segment of shared memory in which the DMA data transfers are effected according to the ordered sequence of the DMA instructions; and

executing through the PAMI, with no FENCE accounting for DMA data transfers, an active FENCE instruction, the FENCE instruction completing execution only after completion of all DMA instructions initiated prior to execution of the FENCE instruction for DMA data transfers between the two endpoints

wherein the DMA instructions comprise DMA GET instructions, and executing a FENCE instruction further comprises;

receiving in the origin endpoint the FENCE instruction, the FENCE instruction specifying a FENCE done callback function;

transmitting, after completion of all GET data transfers previously initiated between the two endpoints, the FENCE instruction through the segment of shared memory from the origin endpoint to the target endpoint, the FENCE instruction specifying a FENCE dispatch callback function;

processing the FENCE instruction by the target endpoint, including executing the FENCE dispatch callback function and advising the origin endpoint of the processing of the FENCE instruction; and

executing by the origin endpoint the FENCE done callback function.