SHARED ADDRESS COLLECTIVES USING COUNTER MECHANISMS

• US 20110078249A1
• Filed: 09/28/2009
• Published: 03/31/2011
• Est. Priority Date: 09/28/2009
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Kumar, Sameer, Blocksome, Michael, Miller, Douglas, Mamidala, Amith R., Dozsa, Gabor, Gooding, Thomas M., Heidelberger, Philip
• Granted Patent: US 8,655,962 B2
• Time in Patent Office: Days
• US Class Current: 709/206
• CPC Class Codes: G06F 9/544 Buffers; Shared memory; Pipes

Abstract

A shared address space on a compute node stores data received from a network and data to transmit to the network. The shared address space includes an application buffer that can be directly operated upon by a plurality of processes, for instance, running on different cores on the compute node. A shared counter is used for one or more of signaling arrival of the data across the plurality of processes running on the compute node, signaling completion of an operation performed by one or more of the plurality of processes, obtaining reservation slots by one or more of the plurality of processes, or combinations thereof.

Claims

1. A device for communication in message passing interface applications with multiple processes running on a compute node connected to a global network of compute nodes, comprising:

a shared address space on a compute node, operable to store data coming in from a network and data to be written out to the network, wherein the shared address space includes an application buffer that can be directly operated upon by a plurality of processes; and

a shared counter operable to be used for one or more of signaling arrival of the data across the plurality of processes running on the compute node, signaling completion of an operation performed by one or more of the plurality of processes, obtaining reservation slots by one or more of the plurality of processes, or combinations thereof.

2. The device of claim 1, wherein each of the plurality of processes run on a separate core on the compute node.

3. The device of claim 1, wherein the shared counter is operable for signaling arrival of the data across the plurality of processes running on the compute node and signaling completion of an operation.

4. The device of claim 3, wherein a process running on the compute node in response to receiving data from the network writes to the shared address space and increments a byte count in the shared counter, and wherein one or more other processes running on the compute node poll the shared counter and in response to detecting an increment in the byte count copy the data.

5. The device of claim 4, wherein the one or more other processes running on the compute node signal completion of copy operation in the shared counter.

6. The device of claim 1, wherein the shared counter includes FIFO data structure, wherein one or more of the plurality of processes obtain a reservation slot in the FIFO.

7. The device of claim 6, wherein the shared counter further includes an atomic counter for indicating whether one or more of the plurality of processes read the FIFO, wherein broadcast operations may be performed between one of the plurality of processes and rest of the plurality of processes using the FIFO and the atomic counter.

8. The device of claim 1, wherein the shared address space includes send and receive buffers used for allreduce MPI operation.

9. The device of claim 1, wherein injection and reception of the data to and from a global network is handled exclusively by two separate dedicated cores, and local broadcast is handled by using another two cores to copy the data from an application buffer of a core dedicated to receiving the data from the global network, the application buffer being in the shared address space, and wherein the shared counter is used to synchronized the data.

10. The device of claim 1, where in the device include one counter for each connection to a global network.

11. The device of claim 1, wherein the shared address space and the shared counter are used to pipeline across different stages of operations performed by the plurality of processes.

12. A collective communication method for message passing interface applications with multiple processes running on a compute node, comprising:

receiving data from a global network for performing a collective operation;

writing the data directly into an application buffer of a core on a compute node receiving the data, the application buffer being in shared address space;

signaling using a shared counter of the received data; and

in response to the signaling, copying the data directly from the application buffer to a plurality of cores on the compute node different from the core receiving the data.

13. The method of claim 12, wherein the shared counter includes at least information associated with base address of the application buffer and total bytes written into the application buffer.

14. The method of claim 13, wherein the shared counter further includes an atomic completion counter.

15. The method of claim 13, wherein the signaling using the shared counter includes incrementing the shared counter.

16. The method of claim 12, wherein the shared counter includes a FIFO data structure.

17. The method of claim 15, wherein an element in the FIFO data structure is reserved for one of the plurality of processes running on the compute node using an atomic counter.

18. The method of claim 12, further including:

dedicating each core on the compute node to a different task.

19. The method of claim 18, wherein injection and reception of the data to and from a global network is handled exclusively by two separate dedicated cores, and local broadcast is handled by using another two cores to copy the data from an application buffer of a core dedicated to receiving the data from the global network, the application buffer being in the shared address space, and wherein the shared counter is used to synchronized the data.

20. The method of claim 18, wherein an allreduce MPI operation is performed by the compute node by having one core in the compute node perform local sum and network reduction and rest of the cores on the compute node perform local data reduction, wherein the shared address space is used for data access and the shared counter is used for synchronization among the one core and the rest of the cores.

21. A computer readable storage medium storing a program of instructions executable by a machine to perform a collective communication method for message passing interface applications with multiple processes running on a compute node, comprising:

receiving data from a global network for performing a collective operation;

writing the data directly into an application buffer of a core on a compute node receiving the data, the application buffer being in shared address space;

signaling using a shared counter of the received data; and

in response to the signaling, copying the data directly from the application buffer to a plurality of cores on the compute node different from the core receiving the data.

22. The computer readable storage medium of claim 21, wherein the shared counter includes at least information associated with base address of the application buffer and total bytes written into the application buffer.

23. The computer readable storage medium of claim 22, wherein the signaling using the shared counter includes incrementing the shared counter.

24. The computer readable storage medium of claim 21, wherein the shared counter includes a FIFO data structure.

25. The computer readable storage medium of claim 21, injection and reception of the data to and from a global network is handled exclusively by two separate dedicated cores, and local broadcast is handled by using another two cores to copy the data from an application buffer of a core dedicated to receiving the data from the global network, the application buffer being in the shared address space, and wherein the shared counter is used to synchronized the data.