Monitoring operating parameters in a distributed computing system with active messages

• US 8,436,720 B2
• Filed: 04/29/2010
• Issued: 05/07/2013
• Est. Priority Date: 04/29/2010
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Archer, Charles J., Carey, James E., Markland, Matthew W., Sanders, Philip J.
• Primary Examiner(s): TRIEU, VAN THANH
• Application Number:
  US12/770,187
• Publication Number:
  US 20110267197A1
• Time in Patent Office: 1,104 Days
• Field of Search: 340/539.1, 340/539.16, 340/539.17, 340/539.19, 340/539.22, 340/539.26, 340/286.01, 340/286.07, 340/286.02, 370/222, 370/236, 370/248, 370/258, 702/57, 702/117, 702/182, 702/188, 714/11, 714/47.1, 709/224
• US Class Current: 340/286.02
• CPC Class Codes: G06F 1/206 comprising thermal management
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3058 Monitoring arrangements for monitoring environmental properties or parameters of the computing system or of the computing system component, e.g. monitoring of power, currents, temperature, humidity, position, vibrations thermal management in cooling arrangements of a computing system G06F1/206
• CPC Class Codes: G06F 11/3082 the data filtering being achieved by aggregating or compressing the monitored data
• CPC Class Codes: G06F 11/3404 for parallel or distributed programming
• CPC Class Codes: G06F 9/5094 where the allocation takes into account power or heat criteria power management in computers in general G06F1/3203; thermal management in computers in general G06F1/206
• CPC Class Codes: G06F 9/54 Interprogram communication
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

In a distributed computing system including a nodes organized for collective operations: initiating, by a root node through an active message to all other nodes, a collective operation, the active message including an instruction to each node to store operating parameter data in each node'"'"'s send buffer; and, responsive to the active message: storing, by each node, the node'"'"'s operating parameter data in the node'"'"'s send buffer and returning, by the node, the operating parameter data as a result of the collective operation.

Claims

1. A method of monitoring operating parameters in a distributed computing system with active messages, the distributed computing system comprising a plurality of nodes organized for collective operations, the method comprising:

initiating, by a root node through an active message to all other nodes, a collective operation, the active message comprising an instruction to each node to store operating parameter data in each node'"'"'s send buffer; and

responsive to the active message;

storing, by each node, the node'"'"'s operating parameter data in the node'"'"'s send buffer and returning, by the node, the operating parameter data as a result of the collective operation.

2. The method of claim 1 wherein storing, by each node, the node'"'"'s operating parameter data in the node'"'"'s send buffer further comprises storing environmental condition data produced by one or more environmental sensors in the node'"'"'s send buffer.

3. The method of claim 2 wherein initiating the collective operation further comprises initiating the collective operation periodically at a predefined time interval, storing environmental condition data further comprises storing system component temperature data, and the method further comprises:

calculating an average system component temperature of the plurality of nodes including averaging the results of the collective operation; and

determining, in dependence upon the average system component temperature and predefined system component temperature criteria, whether to throttle a system component of one or more nodes.

4. The method of claim 2 wherein storing environmental condition data further comprises storing system component temperature data, initiating the collective operation further comprises initiating the collective operation responsive to determining, by the root node, that the root node'"'"'s system component temperature exceeds a predetermined threshold, and the method further comprises:

determining, by the root node in dependence upon the results of the collective operation, whether to trigger a failure alert; and

if the root node does not determine to trigger a failure alert, determining, by the root node, whether to throttle the root node'"'"'s system component having the system component temperature that exceeds the predetermined threshold.

5. The method of claim 1 wherein storing the node'"'"'s operating parameter data further comprises storing application performance metrics and the method further comprises:

determining, in dependence upon the results of the reduction, whether the application performance metrics of the plurality of nodes meet predefined performance criteria; and

if the application performance metrics do not meet the predefined performance criteria, notifying a user of the application performance metrics.

6. The method of claim 1 wherein the active message further comprises an instruction to perform a diagnostic analysis and storing the node'"'"'s operating parameter data further comprises performing the diagnostic analysis and producing diagnostic analysis data.

7. The method of claim 1 wherein the collective operation further comprises one of:

a ‘

reduce’

operation;

an ‘

allreduce’

operation;

or a ‘

gather’

operation.

8. The method of claim 1, wherein the distributed computing system further comprises a parallel computer that includes:

a plurality of compute nodes;

a first data communications network coupling the compute nodes for data communications and optimized for point to point data communications; and

a second data communications network that includes data communications links coupling the compute nodes so as to organize the compute nodes as a tree, each compute node having a separate arithmetic logic unit (‘

ALU’

) dedicated to parallel operations.

9. The method of claim 1, wherein the distributed computing system further comprises a hybrid computing environment, the hybrid computing environment comprising:

a host computer having a host computer architecture; and

an accelerator having an accelerator architecture, the accelerator architecture optimized, with respect to the host computer architecture, for speed of execution of a particular class of computing functions, the host computer and the accelerator adapted to one another for data communications by a system level message passing module.

10. An apparatus for monitoring operating parameters in a distributed computing system with active messages, the distributed computing system comprising a plurality of nodes organized for collective operations, the apparatus comprising a computer processor and a computer memory operatively coupled to the computer processor, the computer memory having disposed within it computer program instructions capable of:

initiating, by a root node through an active message to all other nodes, a collective operation, the active message comprising an instruction to each node to store operating parameter data in each node'"'"'s send buffer; and

responsive to the active message;

storing, by each node, the node'"'"'s operating parameter data in the node'"'"'s send buffer and returning, by the node, the operating parameter data as a result of the collective operation.

11. The apparatus of claim 10 wherein storing, by each node, the node'"'"'s operating parameter data in the node'"'"'s send buffer further comprises storing environmental condition data produced by one or more environmental sensors in the node'"'"'s send buffer.

12. The apparatus of claim 11 wherein initiating the collective operation further comprises initiating the collective operation periodically at a predefined time interval, storing environmental condition data further comprises storing system component temperature data, and the apparatus further comprises computer program instructions capable of:

calculating an average system component temperature of the plurality of nodes including averaging the results of the collective operation; and

determining, in dependence upon the average system component temperature and predefined system component temperature criteria, whether to throttle a system component of one or more nodes.

13. The apparatus of claim 11 wherein storing environmental condition data further comprises storing system component temperature data, initiating the collective operation further comprises initiating the collective operation responsive to determining, by the root node, that the root node'"'"'s system component temperature exceeds a predetermined threshold, and the apparatus further comprises computer program instructions capable of:

determining, by the root node in dependence upon the results of the collective operation, whether to trigger a failure alert; and

if the root node does not determine to trigger a failure alert, determining, by the root node, whether to throttle the root node'"'"'s system component having the system component temperature that exceeds the predetermined threshold.

14. The apparatus of claim 10 wherein storing the node'"'"'s operating parameter data further comprises storing application performance metrics and the apparatus further comprises computer program instructions capable of:

determining, in dependence upon the results of the reduction, whether the application performance metrics of the plurality of nodes meet predefined performance criteria; and

if the application performance metrics do not meet the predefined performance criteria, notifying a user of the application performance metrics.

15. The apparatus of claim 10 wherein the active message further comprises an instruction to perform a diagnostic analysis and storing the node'"'"'s operating parameter data further comprises performing the diagnostic analysis and producing diagnostic analysis data.

16. The apparatus of claim 10 wherein the collective operation further comprises one of:

a ‘

reduce’

operation;

an ‘

allreduce’

operation; and

a ‘

gather’

operation.

17. A computer program product for monitoring operating parameters in a distributed computing system with active messages, the distributed computing system comprising a plurality of nodes organized for collective operations, the computer program product disposed in a computer readable storage medium, wherein the computer readable storage medium is not a signal, the computer program product comprising computer program instructions capable of:

initiating, by a root node through an active message to all other nodes, a collective operation, the active message comprising an instruction to each node to store operating parameter data in each node'"'"'s send buffer; and

responsive to the active message;

storing, by each node, the node'"'"'s operating parameter data in the node'"'"'s send buffer and returning, by the node, the operating parameter data as a result of the collective operation.

18. The computer program product of claim 17 wherein storing, by each node, the node'"'"'s operating parameter data in the node'"'"'s send buffer further comprises storing environmental condition data produced by one or more environmental sensors in the node'"'"'s send buffer.

19. The computer program product of claim 18 wherein initiating the collective operation further comprises initiating the collective operation periodically at a predefined time interval, storing environmental condition data further comprises storing system component temperature data, and the computer program product further comprises computer program instructions capable of:

calculating an average system component temperature of the plurality of nodes including averaging the results of the collective operation; and

determining, in dependence upon the average system component temperature and predefined system component temperature criteria, whether to throttle a system component of one or more nodes.

20. The computer program product of claim 18 wherein storing environmental condition data further comprises storing system component temperature data, initiating the collective operation further comprises initiating the collective operation responsive to determining, by the root node, that the root node'"'"'s system component temperature exceeds a predetermined threshold, and the computer program product further comprises computer program instructions capable of:

determining, by the root node in dependence upon the results of the collective operation, whether to trigger a failure alert; and

if the root node does not determine to trigger a failure alert, determining, by the root node, whether to throttle the root node'"'"'s system component having the system component temperature that exceeds the predetermined threshold.

21. The computer program product of claim 17 wherein storing the node'"'"'s operating parameter data further comprises storing application performance metrics and the computer program product further comprises computer program instructions capable of:

determining, in dependence upon the results of the reduction, whether the application performance metrics of the plurality of nodes meet predefined performance criteria; and

if the application performance metrics do not meet the predefined performance criteria, notifying a user of the application performance metrics.

22. The computer program product of claim 17 wherein the active message further comprises an instruction to perform a diagnostic analysis and storing the node'"'"'s operating parameter data further comprises performing the diagnostic analysis and producing diagnostic analysis data.