INCREASING AVAILABLE FIFO SPACE TO PREVENT MESSAGING QUEUE DEADLOCKS IN A DMA ENVIRONMENT

• US 20100082848A1
• Filed: 09/30/2008
• Published: 04/01/2010
• Est. Priority Date: 09/30/2008
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Chen, Dong, Blocksome, Michael A., Heidelberger, Philip, Gooding, Thomas, Parker, Jeff
• Granted Patent: US 8,112,559 B2
• Time in Patent Office: Days
• US Class Current: 710/22
• CPC Class Codes: G06F 13/28 using burst mode transfer, e.g. direct memory access DMA, cycle steal G06F13/32 takes precedence

Abstract

Embodiments of the invention may be used to manage message queues in a parallel computing environment to prevent message queue deadlock. A direct memory access controller of a compute node may determine when a messaging queue is full. In response, the DMA may generate an interrupt. An interrupt handler may stop the DMA and swap all descriptors from the full messaging queue into a larger queue (or enlarge the original queue). The interrupt handler then restarts the DMA. Alternatively, the interrupt handler stops the DMA, allocates a memory block to hold queue data, and then moves descriptors from the full messaging queue into the allocated memory block. The interrupt handler then restarts the DMA. During a normal messaging advance cycle, a messaging manager attempts to inject the descriptors in the memory block into other messaging queues until the descriptors have all been processed.

Claims

1. A method for managing message queues in a parallel computing system having a plurality of compute nodes, comprising:

determining that a first queue, on a first compute node, storing a set of message descriptors has become full, wherein a direct memory access controller (DMA) is configured to inject message descriptors into the first queue; and

generating an interrupt delivered to an interrupt handler, wherein the interrupt handler is configured to perform the steps of;

stopping the DMA controller;

allocating a region of memory, wherein the memory region is large enough to store the set of messaging descriptors from the first queue;

moving the stored descriptors in the first queue into a second queue local to a messaging manager

notifying the messaging manager about the memory region, and

restarting the DMA controller.

2. The method of claim 1, wherein the messaging manager is configured to attempt to inject the stored descriptors from the second queue back into the first queue during a messaging advance cycle.

3. The method of claim 1, wherein the message descriptors are sent to the first compute node from a second compute node over a network connection connecting the first compute node and the second compute node.

4. The method of claim 1, wherein the message descriptors are generated in response to a remote get packet sent to the first compute node by a second compute node.

5. The method of claim 4, wherein the message descriptors identify a location and a size of data requested to be sent from the first compute node to the second compute node.

6. The method of claim 5, wherein the DMA controller is configured to perform the steps of:

retrieving a message descriptor from the second queue;

retrieving the data at the location identified in the retrieved message descriptor; and

transmitting the data to the second compute node.

7. The method of claim 4, wherein the remote get packet identifies a message descriptor to be injected into the first queue.

8. The method of claim 1, further comprising, after restarting the DMA controller:

receiving, from a second compute node, a remote get packet which includes a message descriptor to be injected into the first queue; and

injecting the message descriptor into the first queue.

9. A computer-readable storage-medium containing a program which, when executed, performs an operation for managing message queues in a parallel computing system having a plurality of compute nodes, the operation comprising:

determining that a first queue, on a first compute node, storing a set of message descriptors has become full, wherein a direct memory access controller (DMA) is configured to inject message descriptors into the first queue; and

generating an interrupt delivered to an interrupt handler, wherein the interrupt handler is configured to perform the steps of;

stopping the DMA controller;

allocating a region of memory, wherein the memory region is large enough to store the set of messaging descriptors from the first queue;

moving the stored descriptors in the first queue into a second queue local to a messaging manager

notifying the messaging manager about the memory region, and

restarting the DMA controller.

10. The computer-readable storage-medium of claim 9, wherein the messaging manager is configured to attempt to inject the stored descriptors from the second queue back into the first queue during a messaging advance cycle.

11. The computer-readable storage-medium of claim 9, wherein the message descriptors are sent to the first compute node from a second compute node over a network connection connecting the first compute node and the second compute node.

12. The computer-readable storage-medium of claim 9, wherein the message descriptors are generated in response to a remote get packet sent to the first compute node by a second compute node.

13. The computer-readable storage-medium of claim 12, wherein the message descriptors identify a location and a size of data requested to be sent from the first compute node to the second compute node.

14. The computer-readable storage-medium of claim 13, wherein the DMA controller is configured to perform the steps of:

retrieving a message descriptor from the second queue;

retrieving the data at the location identified in the retrieved message descriptor; and

transmitting the data to the second compute node.

15. The computer-readable storage-medium of claim 12, wherein the remote get packet identifies a message descriptor to be injected into the first queue.

16. The computer-readable storage-medium of claim 9, wherein the operation further comprises, after restarting the DMA controller:

receiving, from a second compute node, a remote get packet which includes a message descriptor to be injected into the first queue; and

injecting the message descriptor into the first queue.

17. A parallel computing system, comprising:

a plurality of compute nodes, each having at least a processor, a memory and a direct memory access controller (DMA), wherein the plurality of compute nodes are configured to move messages between two compute nodes of the plurality, and wherein the DMA on a first compute node is configured to;

determine that a first queue, on a first compute node, storing a set of message descriptors has become full, wherein a direct memory access controller (DMA) is configured to inject message descriptors into the first queue; and

generate an interrupt delivered to an interrupt handler, wherein the interrupt handler is configured to perform the steps of;

stopping the DMA controller,

allocating a region of memory, wherein the memory region is large enough to store the set of messaging descriptors from the first queue;

moving the stored descriptors in the first queue into a second queue local to a messaging manager

notifying the messaging manager about the memory region, and

restarting the DMA controller.

18. The parallel computing system of claim 17, wherein the messaging manager is configured to attempt to inject the stored descriptors from the second queue back into the first queue during a messaging advance cycle.

20. The parallel computing system of claim 17, wherein the message descriptors are generated in response to a remote get packet sent to the first compute node by a second compute node.

21. The parallel computing system of claim 17, wherein the DMA is further configured to, after restarting the DMA controller:

receive, from a second compute node, a remote get packet which includes a message descriptor to be injected into the first queue; and

inject the message descriptor into the first queue.

19. The parallel computing system of claim 117, wherein the message descriptors are sent to the first compute node from a second compute node over a network connection connecting the first compute node and the second compute node.