Monitoring distributed software health and membership in a compute cluster

• US 8,108,733 B2
• Filed: 05/12/2010
• Issued: 01/31/2012
• Est. Priority Date: 05/12/2010
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Richmond, Michael A.
• Primary Examiner(s): Schell, Joseph
• Application Number:
  US12/778,177
• Publication Number:
  US 20110283149A1
• Time in Patent Office: 629 Days
• Field of Search: 714/43, 714/55, 714/11, 709/223, 709/201, 707/737
• US Class Current: 714/43
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/076 by exceeding a count or rate limit, e.g. word- or bit count limit
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1425 by reconfiguration of node membership
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/302 where the computing system component is a software system
• CPC Class Codes: G06F 11/3051 Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs verification or detection of system hardware configuration G06F11/2247
• CPC Class Codes: G06F 11/3055 Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction by redundancy in operation G06F11/14; error detection or correction by redundancy in hardware G06F11/16
• CPC Class Codes: H04L 43/0817 by checking functioning

Abstract

Techniques for monitoring distributed software health and membership of nodes and software components operating in a compute cluster are disclosed. In one embodiment, each node in the compute cluster operates a watchdog monitoring component in addition to software operating components. The watchdogs are provided with a list of all nodes in a compute cluster that identifies every node'"'"'s neighboring nodes. Each watchdog checks the health of one of its neighboring node, ensuring that this neighboring node is healthy and is operating successfully. Additionally, each watchdog verifies the cluster membership of its other neighboring nodes to ensure that the cluster is operating an adequate number of operating nodes, and that an adequate number of watchdogs are present in the cluster. If an unhealthy or non-member node is identified, the watchdog may initiate corrective action and attempt to restore the node to a correct operational state.

Claims

1. A method for monitoring health and membership of distributed software in a compute cluster having a plurality of nodes, comprising:

generating an ordered list of all nodes in the plurality of nodes configured to operate in the compute cluster;

making the ordered list available to each of the plurality of nodes, each of the plurality of nodes having a watchdog component configured to perform health checks and membership checks on other nodes in the compute cluster;

performing a health check by each node in the plurality of nodes using the watchdog component, the health check comprising;

checking a health status of a first neighbor node in a first direction of the node in the ordered list of nodes; and

performing a first action on the neighbor node responsive to determining that the health status of the first neighbor node is unhealthy; and

performing a membership check by each node, using the watchdog component, on a second neighbor node in a second direction opposite the first direction, the membership check comprising;

verifying membership in the compute cluster of a second neighbor node; and

performing a second action on the second neighbor node responsive to determining that the second neighbor node is not a member of the compute cluster;

wherein the ordered list provides a circular sequence of nodes traversable in either the first direction or the second direction.

2. The method of claim 1, the membership check further comprising verifying membership in the compute cluster of additional neighbor nodes in the second direction responsive to determining that the second neighbor node is not a member of the compute cluster.

3. The method of claim 1, further comprising refreshing the ordered list of all nodes in the plurality of nodes configured to operate in the compute cluster.

4. The method of claim 1, further comprising repeating the health check and membership check steps from each node in the plurality of nodes having the watchdog component of the node in an operational state, the health check and membership check steps being repeated after a defined interval of time.

5. The method of claim 1, further comprising allowing the computer cluster to stabilize for a defined period of time prior to performing additional health check and membership check operations.

6. The method of claim 1, wherein the first action performed on the first neighbor node comprises restarting software components configured to execute on the first neighbor node.

7. The method of claim 1, wherein at least one of the first action and the second action performed on the first neighbor node and the second neighbor node respectively comprises power cycling hardware.

8. The method of claim 1, wherein the second action performed on the second neighbor node comprises placing the second neighbor node on administrative leave within the compute cluster.

9. The method of claim 1, wherein the first action and the second action performed on the first neighbor node and the second neighbor node respectively are each repeated a predefined number of times.

10. A method for monitoring health of nodes within a compute cluster having a plurality of nodes, comprising:

obtaining a list of nodes expected to operate within the compute cluster, each node of the list of nodes having an operational watchdog component;

performing a health check, by each node using the operational watchdog component, on a first neighbor node in a first direction of each node to discover unhealthy nodes in the compute cluster;

performing a membership check by each node using the watchdog component on a second neighbor node in a second direction opposite the first direction;

restoring the unhealthy nodes in the compute cluster to a healthy state; and

repeating the health check and membership check by each node using the operational watchdog component, the health checks comprising performing the health checks by nodes restored to the healthy state, wherein the list of nodes provides a circular sequence of nodes traversable in one of the first direction and the second direction.

11. A distributed cluster computing system, comprising:

a compute cluster comprising a plurality of nodes;

at least one processor within the distributed cluster computing system;

at least one memory store within the distributed cluster computing system having instructions operable with the at least one processor for monitoring health and membership of distributed software operating across the plurality of nodes, the instructions being executed on hardware components within the distributed cluster computing system for;

generating an ordered list of all nodes in the plurality of nodes which are configured to operate in the compute cluster;

making the ordered list available to each of the plurality of nodes, each of the plurality of nodes having a watchdog component configured to perform health checks and membership checks on other nodes in the compute cluster;

performing a health check by each node using the watchdog component, the health check comprising;

checking a health status of a first neighbor node, to a first direction, by each node; and

performing a first action on the first neighbor node responsive to determining that the health status of the first neighbor node is unhealthy; and

performing a membership check by each node using the watchdog component, the membership check including;

verifying membership in the compute cluster of a second neighbor node in a second direction, opposite of the first direction of a second neighbor node; and

performing a second action on the second neighbor node responsive to determining that the second neighbor node is not a member of the compute cluster;

wherein the ordered list provides a circular sequence of nodes traversable in either the first direction or the second direction.

12. The distributed cluster computing system of claim 11, the membership check further comprising instructions being executed for verifying membership in the compute cluster of additional neighbor nodes to the second direction of the healthy node in the ordered list of nodes responsive to determining that the second neighbor node is not a member of the compute cluster.

13. The distributed cluster computing system of claim 11, further comprising instructions being executed for refreshing the ordered list of all nodes in the plurality of nodes configured to operate in the compute cluster.

14. The distributed cluster computing system of claim 11, further comprising instructions being executed for repeating the health check and membership check steps from each node in the plurality of nodes having the watchdog component of the node in an operational state, the health check and membership check steps being repeated after a defined interval of time.

15. The distributed cluster computing system of claim 11, further comprising instructions being executed for allowing the computer cluster to stabilize for a defined period of time prior to performing additional health check and membership check operations.

16. The distributed cluster computing system of claim 11, wherein the first action performed on the first neighbor node includes restarting software components configured to execute on the first neighbor node.

17. The distributed cluster computing system of claim 11, wherein at least one of the first action and the second action performed on the first neighbor node and the second neighbor node respectively comprises power cycling hardware.

18. The distributed cluster computing system of claim 11, wherein the second action performed on the second neighbor node includes placing the second neighbor node on administrative leave within the compute cluster.

19. The distributed cluster computing system of claim 11, wherein the first action and the second action performed on the first neighbor node and the second neighbor node respectively are each repeated a predefined number of times.

20. A distributed cluster computing system, comprising:

a compute cluster comprising a plurality of nodes;

at least one processor within the distributed cluster computing system;

at least one memory store within the distributed cluster computing system having instructions operable with the at least one processor for monitoring health of the plurality of nodes, the instructions being executed on hardware components within the distributed cluster computing system for;

obtaining a list of nodes expected to operate within the compute cluster, each node having an operational watchdog component;

performing a health check, by each node within using the operational watchdog component, on a first neighbor node in a first direction of each node to discover unhealthy nodes in the compute cluster;

performing a membership check, by each node using the watchdog component, on a second neighbor node in a second direction opposite the first direction;

restoring the unhealthy nodes in the compute cluster to a healthy state; and

repeating the health check and membership check by each node using the operational watchdog component, the health checks comprising performing the health checks by nodes restored to the healthy state, wherein the list of nodes provides a circular sequence of nodes traversable in one of the first direction and the second direction.