Automated node restart in clustered computer system

• US 20020133727A1
• Filed: 03/15/2001
• Published: 09/19/2002
• Est. Priority Date: 03/15/2001
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Miller, Robert, Thayib, Kiswanto, Morey, Vicki Lynn, Dervin, Jennifer Anne
• Granted Patent: US 6,952,766 B2
• Time in Patent Office: Days
• US Class Current: 713/310
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/1425 by reconfiguration of node membership
• CPC Class Codes: G06F 11/1482 by means of middleware or OS functionality
• CPC Class Codes: G06F 11/202 where processing functionality is redundant redundant communication control functionality G06F11/2005, redundant storage control functionality G06F11/2089
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

An apparatus, program product and method initiate a restart of a node in a clustered computer system using a member of a clustering group that resides on a different node from that to be restarted. Typically, a restart operation is initiated by the member in response to a membership change message sent by another group member that is resident on the node to be restarted, with an indicator associated with the membership change message that indicates that a restart should be initiated. Typically, the restart is implemented in much the same manner as a start operation that is performed when a node is initially added to a cluster, with additional functionality utilized to preclude repeated restart attempts upon a failure of a prior restart operation.

Claims

1. A method of restarting a node in a clustered computer system, wherein the clustered computer system hosts a group including first and second members that reside respectively on first and second nodes, the method comprising:

(a) in response to a clustering failure on the first node, notifying the second member of the group the using the first member; and

(b) in response to the notification, initiating a restart of the first node using the second member.

2. The method of claim 1, wherein the group comprises a cluster control group that includes a member on each node participating in clustering in the clustered computer system, and wherein the first and second members are each members of the cluster control group.

3. The method of claim 1, wherein notifying the second member comprises issuing a membership change request to the group using the first member.

4. The method of claim 3, wherein issuing the membership change request includes indicating in association with the membership change request that the membership change request is for the purpose of restarting the first node.

5. The method of claim 4, wherein indicating that the membership change request is for the purpose of restarting the first node includes setting a reason field in the membership change request to a restart value.

6. The method of claim 1, wherein initiating the restart includes issuing a start node request to the group using the second member.

7. The method of claim 6, wherein issuing the start node request includes indicating in association with the start node request that the start node request is for the purpose of restarting the first node.

8. The method of claim 7, further comprising:

(a) detecting the clustering failure in the first node; and

(b) determining whether the clustering failure occurred during a restart of the first node;

wherein notifying the second member of the group using the first member is performed in response to detecting the clustering failure in the first node and determining that the clustering failure did not occur during a restart of the first node.

9. The method of claim 8, further comprising signaling an error in response to detecting the clustering failure in the first node if the clustering failure occurred during a restart of the first node.

10. The method of claim 8, wherein determining whether the clustering failure occurred during a restart of the first node includes determining whether the start node request indicates that the start node request is for the purpose of restarting the first node.

11. The method of claim 10, further comprising:

(a) counting protocols processed by the first node after a restart; and

(b) signaling an error in response to detecting the clustering failure in the first node if the clustering failure occurred during a restart of the first node and the number of protocols processed by the first node after the restart is less than a predetermined threshold.

12. The method of claim 6, further comprising, in response to the clustering failure on the first node, terminating clustering on the first node after notifying the second member of the group the using the first member.

13. The method of claim 1, further comprising, in response to the notification, selecting the second member from a plurality of members in the group to initiate the restart of the first node.

14. The method of claim 13, wherein selecting the second member to initiate the restart of the first node includes determining that the second member is a lowest named member among the plurality of members.

16. The method of claim 15, further comprising, in response to a second clustering failure during initiation of clustering on the first node:

(a) determining from the start node request that initiated clustering on the first node that the purpose of the start node request is for restarting the first node; and

(b) in response to determining that the start node request is for restarting the first node, signaling an error instead of initiating a second restart of the first node.

18. The apparatus of claim 17, wherein the program comprises a member of a group hosted by the clustered computer system, the group including an additional member residing on the other node.

19. The apparatus of claim 18, wherein the group comprises a cluster control group that includes a member on each node participating in clustering in the clustered computer system.

20. The apparatus of claim 18, wherein the program is configured to initiate the restart by issuing a start node request to the group.

21. The apparatus of claim 20, wherein the start node request indicates that the start node request is for the purpose of restarting the first node.

22. The apparatus of claim 17, wherein the program is configured to initiate the restart of the other node responsive to a membership change request received from the other node.

23. The apparatus of claim 19, wherein the program is configured to determine whether the membership change request indicates that the membership change request is for the purpose of restarting the first node.

25. The clustered computer system of claim 24, wherein the first member is configured to detect the clustering failure in the first node, determine whether the clustering failure occurred during a restart of the first node, and notify the second member in response to detecting the clustering failure in the first node and determining that the clustering failure did not occur during a restart of the first node.

26. The clustered computer system of claim 25, wherein the first member is further configured to signal an error in response to detecting the clustering failure in the first node if the clustering failure occurred during a restart of the first node.

27. The clustered computer system of claim 25, wherein the second member is configured to initiate the restart by issuing a start node request to the group, and wherein the first member is configured to determine whether the clustering failure occurred during a restart of the first node by determining whether the start node request indicates that the start node request is for the purpose of restarting the first node.

28. The clustered computer system of claim 27, wherein the first member is configured to signal an error in response to detecting the clustering failure in the first node if the clustering failure occurred during a restart of the first node and a tracked number of protocols processed by the first node after the restart is less than a predetermined threshold.

29. The clustered computer system of claim 24, wherein the first member is further configured to terminate clustering on the first node after notifying the second member.

30. The clustered computer system of claim 24, further comprising a third node coupled to the first and second nodes, wherein the group includes a third member resident on the third node, wherein the first member is configured to notify the third member in response to the clustering failure on the first node, and wherein each of the second and third members is configured to locally select a single member in the group to initiate the restart of the first node.

31. The clustered computer system of claim 30, wherein the second member is configured to locally select the single member by determining whether the second member is a lowest named member among the members of the group.

32. The clustered computer system of claim 24, wherein the group comprises a cluster control group that includes a member on each node participating in clustering in the clustered computer system, and wherein the first and second members are each members of the cluster control group.

33. The clustered computer system of claim 24, wherein the first member is configured to notify the second member by issuing a membership change request to the group.

35. The program product of claim 34, wherein the signal bearing medium includes at least one of a recordable medium and a transmission medium.

37. The program product of claim 36, wherein the first and second programs are borne on separate signal bearing media.

38. The program product of claim 36, wherein the first and second programs are borne on the same signal bearing medium.

15. A method of restarting a node among a plurality of nodes in a clustered computer system, wherein the clustered computer system hosts a cluster control group including a plurality of cluster control members, each residing respectively on a different node from the plurality of nodes, the method comprising:

(a) detecting a clustering failure on a first node among the plurality of nodes;

(b) in response to detecting the clustering failure on the first node, issuing a membership change request from the first node to the cluster control member on each other node in the plurality of nodes, the membership change request indicating that the membership change request is for the purpose of restarting the first node;

(c) terminating clustering on the first node after issuing the membership change request;

(d) in response to the membership change request, selecting a second node from the plurality of nodes that is different from the first node;

(d) issuing a start node request using the selected second node, the start node request indicating that the purpose of the start node request is for restarting the first node; and

(e) in response to the start node request, initiating clustering on the first node.

17. An apparatus, comprising:

(a) a memory accessible by a node in a clustered computer system; and

(b) a program resident in the memory, the program configured to initiate a restart of another node in the clustered computer system in response to a notification from the other node of a clustering failure on the other node.

24. A clustered computer system, comprising:

(a) first and second nodes coupled to one another over a network; and

(b) a group including first and second members, the first member resident on the first node and the second member resident on the second node, wherein the first member is configured to notify the second member in response to a clustering failure on the first node, and wherein the second member is configured to initiate a restart of the first node in response to the notification.

34. A program product, comprising:

(a) a program configured to reside on a node in a clustered computer system, the program configured to initiate a restart of another node in the clustered computer system in response to a notification from the other node of a clustering failure on the other node; and

(b) a signal bearing medium bearing the program.

36. A program product, comprising:

(a) first and second programs respectively configured to reside on first and second nodes in a clustered computer system, the first and second programs respectively operating as first and second members of a group, the first program configured to configured to notify the second program in response to a clustering failure on the first node, and the second program configured to initiate a restart of the first node in response to the notification; and

(b) at least one signal bearing medium bearing the first and second programs.