Node shutdown in clustered computer system

• US 20020145983A1
• Filed: 04/06/2001
• Published: 10/10/2002
• Est. Priority Date: 04/06/2001
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Miller, Robert, Block, Timothy Roy, Thayib, Kiswanto
• Granted Patent: US 6,918,051 B2
• Time in Patent Office: Days
• US Class Current: 370/254
• CPC Class Codes: G06F 11/0715 in a system implementing multitasking multitasking per se G06F9/46
• CPC Class Codes: G06F 11/0796 Safety measures, i.e. ensuring safe condition in the event of error, e.g. for controlling element

Abstract

A clustered computer system, apparatus, program product and method utilize a group member-initiated shutdown process to terminate clustering on a node in an automated and orderly fashion, typically in the event of a failure detected by a group member residing on that node. As a component of such a process, node leave operations are initiated on the other nodes in a clustered computer system, thereby permitting any dependency failovers to occur in an automated fashion. Moreover, other group members on a node to be shutdown are preemptively terminated prior to local detection of the failure within those other group members, so that termination of clustering on the node may be initiated to complete a shutdown operation.

Claims

1. A method of shutting down a node in a clustered computer system, the method comprising:

(a) detecting a failure in a first node among a plurality of nodes in a clustered computer system with a first group member resident on the first node;

(b) in response to detecting the failure, transmitting a signal to each of the other nodes in the plurality of nodes to initiate on each of the other nodes a node leave operation that terminates clustering with the first node; and

(c) in response to detecting the failure, preemptively terminating a second group member resident on the first node prior to any detection of the failure by the second group member.

4. The method of claim 1, wherein detecting the failure comprises detecting an error that requires clustering to terminate on the first node.

5. The method of claim 4, wherein the error is selected from the group consisting of a communication error, a protocol error, failure or termination of a cluster name server, failure or termination of a cluster control group member, failure or termination of a cluster liveliness monitor, corruption of a critical cluster object, and combinations thereof.

6. The method of claim 1, wherein transmitting the signal comprises transmitting a distress signal to each other node in the plurality of nodes.

7. The method of claim 1, further comprising, at each of the other nodes in the plurality of nodes, processing the distress signal on such other node by initiating a node leave operation for each group member resident on such other node that is a member of a group that has another member resident on the first node.

8. The method of claim 1, further comprising, at another node in the plurality of nodes, performing a dependent failover in response to receiving the signal from the first node.

9. The method of claim 1, wherein preemptively terminating the group member comprises notifying each active group member resident on the first node that clustering is ending on the first node.

10. The method of claim 9, further comprising ending clustering on the first node after notifying each active group member resident on the first node.

11. The method of claim 9, wherein notifying each active group member includes sending an error message to each active group member.

12. The method of claim 1, further comprising disregarding a second failure detected from a group member, other than the first group member, that is resident on the first node if transmission of the signal has already been initiated.

14. The method of claim 13, further comprising detecting a failure in the first node with the group member, wherein initiating the shutdown of the first node is performed in response to detecting the failure.

15. The method of claim 14, wherein shutting down the first node comprises:

(a) transmitting a signal to each of the other nodes in the plurality of nodes to initiate on each of the other nodes a node leave operation that terminates clustering with the first node; and

(b) preemptively terminating a second group member resident on the first node prior to any detection of the failure by the second group member.

16. The method of claim 15, wherein initiating the shutdown of the first node comprises sending a shutdown message from the group member to a cluster control group member, and wherein shutting down the first node further comprises:

(a) notifying a clustering infrastructure resident on the first node using the cluster control group member to transmit the signal and initiate preemptive termination of the second group member; and

(b) terminating the clustering infrastructure.

17. The method of claim 15, wherein transmitting the signal comprises transmitting a distress signal to each other node in the plurality of nodes, the method further comprising, at each of the other nodes in the plurality of nodes, processing the distress signal on such other node by initiating a node leave operation for each group member resident on such other node that is a member of a group that has another member resident on the first node.

18. The method of claim 15, wherein preemptively terminating the second group member comprises notifying each active group member resident on the first node that clustering is ending on the first node.

19. The method of claim 15, further comprising disregarding a second failure detected from another group member resident on the first node if the signal has already been transmitted.

2. The method of claim 2, wherein the first group member is a cluster control group member.

3. The method of claim 2, wherein the first group member is a member of a group other than a cluster control group, wherein the method further comprises sending a shutdown message to a cluster control group member resident on the first node, and wherein transmitting the signal to each of the other nodes and preemptively terminating the second group member are each initiated by the cluster control group member in response to the shutdown message from the first group member.

13. A method of shutting down a node in a clustered computer system, the method comprising:

(a) in a group member resident on a first node among a plurality of nodes in a clustered computer system, initiating a shutdown of the first node; and

(b) shutting down the first node in response to initiation of the shutdown by the group member.

20. An apparatus, comprising:

(a) a memory accessible by a first node among a plurality of nodes in a clustered computer system; and

(b) first and second group members resident in the memory, the first group member configured to detect a failure in the first node; and

(c) a program resident in the memory, the program configured to shut down the first node in response to the detected failure by transmitting a signal to each of the other nodes in the plurality of nodes to initiate on each of the other nodes a node leave operation that terminates clustering with the first node, and preemptively terminating the second group member resident on the first node prior to any detection of the failure by the second group member.

21. The apparatus of claim 20, wherein the first group member is a member of a group other than a cluster control group, wherein the first group member is configured to send a shutdown message to a cluster control group member resident on the first node, and wherein the cluster control group member is configured to initiate transmission of the signal to each of the other nodes and termination of the second group member in response to the shutdown message from the first group member.

22. The apparatus of claim 20, wherein the program is configured to transmit the signal by transmitting a distress signal to each other node in the plurality of nodes.

23. The apparatus of claim 20, wherein the program is configured to terminate the group member by notifying each active group member resident on the first node that clustering is ending on the first node.

24. The apparatus of claim 23, wherein the program is further configured to end clustering on the first node after notifying each active group member resident on the first node.

25. The apparatus of claim 20, wherein the program is further configured to disregard a second failure detected from a group member, other than the first group member, that is resident on the first node if transmission of the signal has already been initiated.

27. The clustered computer system of claim 26, wherein the second node is configured to initiate a node leave operation for each group member resident on the second node that is a member of a group that has another member resident on the first node.

28. The clustered computer system of claim 27, wherein the second node is configured to perform a dependent failover in response to receiving the signal from the first node.

30. The program product of claim 29, wherein the signal bearing medium includes at least one of a recordable medium and a transmission medium.

26. A clustered computer system, comprising first and second nodes coupled to one another over a network, wherein:

(a) the first node is configured to shut down in response to a failure detected in the first node by a first group member resident on the first node by transmitting a signal to the second node and preemptively terminating a second group member resident on the first node prior to any detection of the failure by the second group member; and

(b) the second node is configured to initiate a node leave operation that terminates clustering with the first node in response to the signal from the first node.

29. A program product, comprising:

(a) first and second group members, the first group member configured to detect a failure in a first node among a plurality of nodes in a clustered computer system;

(b) a program configured to shut down the first node in response to the detected failure by transmitting a signal to each of the other nodes in the plurality of nodes to initiate on each of the other nodes a node leave operation that terminates clustering with the first node, and preemptively terminating the second group member resident on the first node prior to any detection of the failure by the second group member; and

(c) a signal bearing medium bearing the program.