Message Flow Control in a Multi-Node Computer System

• US 20090319621A1
• Filed: 06/24/2008
• Published: 12/24/2009
• Est. Priority Date: 06/24/2008
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Darrington, David L., Barsness, Eric L., Santosuosso, John M., Peters, Amanda
• Granted Patent: US 9,514,023 B2
• Time in Patent Office: Days
• US Class Current: 709/206
• CPC Class Codes: G06F 11/0724 in a multiprocessor or a multi-core unit multiprocessors per se G06F15/80
• CPC Class Codes: G06F 11/0781 Error filtering or prioritizing based on a policy defined by the user or on a policy defined by a hardware/software module, e.g. according to a severity level
• CPC Class Codes: G06F 11/3433 for load management allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029
• CPC Class Codes: G06F 11/3471 Address tracing
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: G06F 2201/88 Monitoring involving counting

Abstract

Embodiments of the invention provide for controlling message flow across a parallel computer system having multiple compute nodes by selectively grouping compute nodes of such a system into node pools and assigning message flow control policies to nodes in the node pools. The message flow control policies specify logging and/or tracing activities to be performed by instances of applications running on nodes assigned to the node pools. As the application is executed, logging and/or tracing messages are generated on the compute nodes according to message flow control policies assigned to the nodes. Optionally, the message flow is analyzed, the message flow control policies are adjusted, and duplicate messages are eliminated.

Claims

1. A computer-implemented method for controlling message flow in a parallel computing system having a plurality of compute nodes, the method comprising:

assigning a first set of compute nodes to a first node pool, wherein a first message flow control policy is assigned to each compute node of the first node pool, and wherein the message flow control policy specifies at least one of logging and/or tracing activity to be performed by an instance of an application running on at least a first compute node assigned to the first node pool;

initiating execution of the application on each of the compute nodes in the first node pool; and

while executing the application on at least the first compute node, generating one or more logging or tracing messages according to the first message flow control policy.

2. The method of claim 1, further comprising:

transmitting, by first compute node, the one or more generated messages to a service node; and

storing, by the service node, each generated message.

3. The method of claim 1, wherein the message flow control policy specifies tracing messages to be generated by the instance of the application running on the first compute node assigned to the first node pool.

4. The method of claim 1, wherein the message flow control policy specifies logging activity to be performed by a plurality of compute nodes assigned to the first node pool, and wherein a logging activity level is assigned to each instance of the application.

5. The method of claim 4, wherein the logging activity level specifies a level of verbosity for generating log messages while executing the instance of the application on each respective compute node.

6. The method of claim 4, wherein the message flow control policy specifies logging activity to be performed by a plurality of compute nodes assigned to the first node pool, and wherein multiple sets of compute nodes of the first node pool are each assigned a respective logging activity level.

7. The method of claim 1, wherein the message flow control policy specifies logging activity to be performed by the instance of the application running on the compute nodes assigned to the first node pool, and wherein a first logging activity level is assigned to a first instance of the application and a second logging activity level is assigned to at least a second instance of the application, wherein the first and second logging activity levels are different from one another.

8. The method of claim 1, wherein the message flow control policy specifies tracing activity to be performed by a plurality of compute nodes assigned to the first node pool and specifies that the first instance of the application executing on the first compute node should generate tracing messages relative to a first portion of the application and further specifies that a second instance of the application should generate tracing messages relative to a second portion of the application executing on a second compute node.

9. The method of claim 1, wherein the message flow control policy specifies logging activity to be performed by a plurality of compute nodes assigned to the first node pool and specifies that the first instance of the application executing on the first compute node should generate log messages relative to a first type of event of the application and a second instance of the application should generate log messages relative to a second a second type of event.

10. The method of claim 1, wherein the message flow control policy specifies to generate a logging message for a first occurrence of a specified event and to not generate further logging messages for additional occurs of the specified event.

11. The method of claim 1, further comprising:

assigning a second set of compute nodes to a second node pool, wherein a second message flow control policy is assigned to each compute node of the second node pool, and wherein the message flow control policy specifies at least one of logging and or tracing activity to be performed by an instance of the application running on at least a first compute node assigned to the second node pool.

12. A computer-readable storage medium containing a program which, when executed by a processor, performs an operation for controlling message flow in a parallel computing system having a plurality of compute nodes, the operation comprising:

assigning a first set of compute nodes to a first node pool, wherein a first message flow control policy is assigned to each compute node of the first node pool, and wherein the message flow control policy specifies at least one of logging and or tracing activity to be performed by an instance of an application running on at least a first compute node assigned to the first node pool;

initiating execution of the application on each of the compute nodes in the first node pool; and

while executing the application on at least the first compute node, generating one or more logging or tracing messages according to the first message flow control policy.

13. The computer-readable storage medium of claim 12, wherein the operations further comprises:

transmitting, by first compute node, the one or more generated messages to a service node; and

storing, by the service node, each generated message.

14. The computer-readable storage medium of claim 12, wherein the message flow control policy specifies tracing messages to be generated by the instance of the application running on the first compute node assigned to the first node pool.

15. The computer-readable storage medium of claim 12, wherein the message flow control policy specifies logging activity to be performed by a plurality of compute nodes assigned to the first node pool, and wherein a logging activity level is assigned to each instance of the application.

16. The computer-readable storage medium of claim 15, wherein the logging activity level specifies a level of verbosity for generating log messages while executing the instance of the application on each respective compute node.

17. The computer-readable storage medium of claim 15, wherein the message flow control policy specifies logging activity to be performed by a plurality of compute nodes assigned to the first node pool, and wherein multiple sets of compute nodes of the first node pool are each assigned a respective logging activity level.

18. The computer-readable storage medium of claim 12, wherein the message flow control policy specifies logging activity to be performed by the instance of the application running on the compute nodes assigned to the first node pool, and wherein a first logging activity level is assigned to a first instance of the application and a second logging activity level is assigned to at least a second instance of the application, wherein the first and second logging activity levels are different from one another.

19. The computer-readable storage medium of claim 12, wherein the message flow control policy specifies tracing activity to be performed by a plurality of compute nodes assigned to the first node pool and specifies that the first instance of the application executing on the first compute node should generate tracing messages relative to a first portion of the application and further specifies that a second instance of the application should generate tracing messages relative to a second portion of the application executing on a second compute node.

20. The computer-readable storage medium of claim 12, wherein the message flow control policy specifies logging activity to be performed by a plurality of compute nodes assigned to the first node pool and specifies that the first instance of the application executing on the first compute node should generate log messages relative to a first type of event of the application and a second instance of the application should generate log messages relative to a second a second type of event.

21. The computer-readable storage medium of claim 12, wherein the operation further comprises:

assigning a second set of compute nodes to a second node pool, wherein a second message flow control policy is assigned to each compute node of the second node pool, and wherein the message flow control policy specifies at least one of logging and or tracing activity to be performed by an instance of the application running on at least a first compute node assigned to the second node pool.

22. A parallel computing system, comprising:

a plurality of compute nodes, each having at least a processor and a memory, wherein the plurality of compute nodes is configured to execute a parallel computing task; and

a service node having at least a processor and a memory and a tracing-logging control program for controlling message flow in the parallel computing system, wherein the tracing-logging control program is configured to;

assign a first set of compute nodes to a first node pool, wherein a first message flow control policy is assigned to each compute node of the first node pool, and wherein the message flow control policy specifies at least one of logging and/or tracing activity to be performed by an instance of an application running on at least a first compute node assigned to the first node pool; and

initiate execution of the application on each of the compute nodes in the first node pool, wherein at least the first compute node is configured to generate, while executing the application, one or more logging or tracing messages according to the first message flow control policy.

23. The parallel computing system of claim 22, wherein the message flow control policy specifies tracing messages to be generated by the instance of the application running on the first compute node assigned to the first node pool.

24. The parallel computing system of claim 22, wherein the message flow control policy specifies logging activity to be performed by a plurality of compute nodes assigned to the first node pool, and wherein multiple sets of compute nodes of the first node pool are each assigned a respective logging activity level.

25. The parallel computing system of claim 22, wherein the tracing-logging control program is further configured to:

assign a second set of compute nodes to a second node pool, wherein a second message flow control policy is assigned to each compute node of the second node pool, and wherein the message flow control policy specifies at least one of logging and or tracing activity to be performed by an instance of the application running on at least a first compute node assigned to the second node pool.