Optimized collectives using a DMA on a parallel computer

• US 7,886,084 B2
• Filed: 06/26/2007
• Issued: 02/08/2011
• Est. Priority Date: 06/26/2007
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Gabor, Dozsa, Chen, Dong, Heidelberger, Phillip, Giampapa, Mark E.
• Primary Examiner(s): Tsai; Henry W
• Assistant Examiner(s): RHU, KRIS M
• Application Number:
  US11/768,645
• Publication Number:
  US 20090006662A1
• Time in Patent Office: 1,323 Days
• Field of Search: None
• US Class Current: 710/22
• CPC Class Codes: G06F 13/28 using burst mode transfer, e.g. direct memory access DMA, cycle steal G06F13/32 takes precedence

Abstract

Optimizing collective operations using direct memory access controller on a parallel computer, in one aspect, may comprise establishing a byte counter associated with a direct memory access controller for each submessage in a message. The byte counter includes at least a base address of memory and a byte count associated with a submessage. A byte counter associated with a submessage is monitored to determine whether at least a block of data of the submessage has been received. The block of data has a predetermined size, for example, a number of bytes. The block is processed when the block has been fully received, for example, when the byte count indicates all bytes of the block have been received. The monitoring and processing may continue for all blocks in all submessages in the message.

Claims

1. A method for optimizing collective operations using direct memory access controller on a parallel computer, comprising:

establishing a byte counter associated with direct memory access controller for each submessage in a message, the byte counter storing at least a base address of memory and a byte count associated with a submessage, the byte counter including at least a reception counter and an injection counter;

monitoring the byte counter associated with a submessage to determine whether at least a block of data of the submessage has been received, the block of data having a predetermined size;

processing the block when said block has been fully received; and

continuing the monitoring and processing step until all blocks in all submessages in the message have been processed.

2. The method of claim 1, wherein the step of processing includes:

sending the block when said block has been fully received.

3. The method of claim 1, wherein the step of processing includes:

performing a reduction operation on the block.

4. The method of claim 3, wherein the step of processing further includes:

sending result of the reduction operation to a next node on a parallel computer.

5. The method of claim 1, wherein the step of monitoring includes:

reading a byte counter value stored in the byte counter and updated by a direct memory access controller to determine whether the block of data of the submessage has been received.

6. The method of claim 1, wherein the step of processing includes:

generating a message descriptor describing at least a counter identifier and offset in memory associated with the block of data; and

resending the block of data to a next node in a parallel computer.

7. The method of claim 6, wherein the step of processing further includes:

injecting the message descriptor into a fifo in memory associated with the block of data.

8. The method of claim 7, wherein the step of processing further includes:

modifying fifo metadata associated with the block in a direct memory access controller, wherein the direct memory access controller reads the fifo metadata and corresponding message descriptor in the fifo in memory to trigger resending of the data block described by the message descriptor.

9. The method of claim 1, wherein all nodes in the parallel computer agree on counter identifiers used for said each submessage in the message.

10. The method of claim 1, wherein the message is a broadcast message.

11. The method of claim 1, wherein the message is a reduction operation message.

12. A system for optimizing collective operations using direct memory access controller on a parallel computer, comprising:

one or more processors in a node;

memory in the node, the memory including at least an injection fifo and a receive buffer;

direct memory access controller in the node, the direct memory access controller including at least a byte counter for each submessage of a message, the byte counter including at least a base address in memory for storing associated submessage and a counter value, the byte counter including at least a reception counter and an injection counter, the direct memory access controller operable to update the counter value as a result of receiving one or more bytes of the associated submessage into the node;

said one or more processors operable to monitor the counter value and when a predetermined number of bytes of the submessage is received, the one or more processors further operable to process a block of data comprising the received predetermined number of bytes of the submessage.

13. The system of claim 12, wherein said one or more processors are operable to resend the block of data by updating a message descriptor in the injection fifo with information about the block of data.

14. The system of claim 13, wherein the information about the block of data includes at least a counter identifier associated with the submessage, the counter identifier including at least a base address in memory, and an offset from the base address in memory where the block of data is stored.

15. The system of claim 12, wherein the block of data is stored in the receive buffer.

16. The system of claim 12, wherein said one or more processors are operable to read the byte counter to determine a number of bytes received for the submessage.

17. The system of claim 12, wherein the message is a collective operation message in a parallel computer.

18. The system of claim 12, wherein said one or more processors are operable to perform a reduction operation on the block of data.

19. The system of claim 18, wherein said one or more processors are operable to send a result of the reduction operation on the block of data.

20. A method for optimizing collective operations using direct memory access controller on a parallel computer, comprising:

allocating a reception buffer on a node for receiving a message, the reception buffer having different slots for storing data received from different nodes on a parallel computer;

defining a counter identifier associated with a counter on the node, the counter identifier being common among all nodes on the parallel computer and the counter comprising at least a base address of the reception buffer for placing a message and a counter value indicating a count of received data;

receiving at a node a message from all the nodes;

polling the counter value until the count of received data equals a predetermined size of data that is expected to be received; and

performing a short reduction on the received message.