Automated node restart in clustered computer system

• US 6,952,766 B2
• Filed: 03/15/2001
• Issued: 10/04/2005
• Est. Priority Date: 03/15/2001
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Miller, Robert, Thayib, Kiswanto, Morey, Vicki Lynn, Dervin, Jennifer Anne
• Primary Examiner(s): Cao, Chun
• Application Number:
  US09/809,408
• Publication Number:
  US 20020133727A1
• Time in Patent Office: 1,664 Days
• Field of Search: 713/2, 713/200, 713/300, 709/223, 709/220, 714/38, 714/4, 714/47, 714/48, 714/43
• US Class Current: 713/2
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/1425 by reconfiguration of node membership
• CPC Class Codes: G06F 11/1482 by means of middleware or OS functionality
• CPC Class Codes: G06F 11/202 where processing functionality is redundant redundant communication control functionality G06F11/2005, redundant storage control functionality G06F11/2089
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

An apparatus, program product and method initiate a restart of a node in a clustered computer system using a member of a clustering group that resides on a different node from that to be restarted. Typically, a restart operation is initiated by the member in response to a membership change message sent by another group member that is resident on the node to be restarted, with an indicator associated with the membership change message that indicates that a restart should be initiated. Typically, the restart is implemented in much the same manner as a start operation that is performed when a node is initially added to a cluster, with additional functionality utilized to preclude repeated restart attempts upon a failure of a prior restart operation.

Claims

1. A method of restarting a node in a clustered computer system, wherein the clustered computer system hosts a group including first and second members that reside respectively on first and second nodes, the method comprising:

(a) in response to a clustering failure on the first node, notifying the second member of the group using the first member by issuing a request to the group; and

(b) in response to the notification, initiating a restart of the first node using the second member;

wherein the group further includes a third member that resides on a third node in the clustered computer system, and wherein issuing the request to the group includes forwarding the request to each of the second and third members of the group, the method further comprising selecting the second member to initiate the restart of the first node.

2. The method of claim 1, wherein the group comprises a cluster control group that includes a member on each node participating in clustering in the clustered computer system, and wherein the first and second members are each members of the cluster control group.

3. The method of claim 1, wherein notifying the second member comprises issuing a membership change request to the group using the first member.

4. The method of claim 3, wherein issuing the membership change request includes indicating in association with the membership change request that the membership change request is for the purpose of restarting the first node.

5. The method of claim 1, wherein initiating the restart includes issuing a start node request to the group using the second member.

6. The method of claim 5, wherein issuing the start node request includes indicating in association with the start node request that the start node request is for the purpose of restarting the first node.

7. The method of claim 6, further comprising:

(a) detecting the clustering failure in the first node; and

(b) determining whether the clustering failure occurred during a restart of the first node;

wherein notifying the second member of the group using the first member is performed in response to detecting the clustering failure in the first node and determining that the clustering failure did not occur during a restart of the first node.

8. The method of claim 7, further comprising signaling an error in response to detecting the clustering failure in the first node if the clustering failure occurred during a restart of the first node.

9. The method of claim 7, wherein determining whether the clustering failure occurred during a restart of the first node includes determining whether the start node request indicates that the start node request is for the purpose of restarting the first node.

10. The method of claim 9, further comprising:

(a) counting protocols processed by the first node after a restart; and

(b) signaling an error in response to detecting the clustering failure in the first node if the clustering failure occurred during a restart of the first node and the number of protocols processed by the first node after the restart is less than a predetermined threshold.

11. The method of claim 5, further comprising, in response to the clustering failure on the first node, terminating clustering on the first node after notifying the second member of the group using the first member.

12. A method of restarting a node in a clustered computer system, wherein the clustered computer system hosts a group including first and second members that reside respectively on first and second nodes, the method comprising:

(a) in response to a clustering failure on the first node, notifying the second member of the group using the first member; and

(b) in response to the notification, initiating a restart of the first node using the second member;

wherein notifying the second member comprises issuing a membership change request to the group using the first member, wherein issuing the membership change request includes indicating in association with the membership change request that the membership change request is for the purpose of restarting the first node; and

wherein indicating that the membership change request is for the purpose of restarting the first node includes setting a reason field in the membership change request to a restart value.

13. A method of restarting a node in a clustered computer system, wherein the clustered computer system hosts a group including first and second members that reside respectively on first and second nodes, the method comprising:

(a) in response to a clustering failure on the first node, notifying the second member of the group using the first member;

(b) in response to the notification, initiating a restart of the first node using the second member; and

(c) in response to the notification, selecting the second member from a plurality of members in the group to initiate the restart of the first node;

wherein selecting the second member to initiate the restart of the first node includes determining that the second member is a lowest named member among the plurality of members.

14. A method of restarting a node among a plurality of nodes in a clustered computer system, wherein the clustered computer system hosts a cluster control group including a plurality of cluster control members, each residing respectively on a different node from the plurality of nodes, the method comprising:

(a) detecting a clustering failure on a first node among the plurality of nodes;

(b) in response to detecting the clustering failure on the first node, issuing a membership change request from the first node to the cluster control member on at least second and third nodes among the plurality of nodes, the membership change request indicating that the membership change request is for the purpose of restarting the first node;

(c) terminating clustering on the first node after issuing the membership change request;

(d) in response to the membership change request, selecting the second node to restart the first node from among the at least second and third nodes receiving the membership change request, wherein the at least second and third nodes receiving the membership change request are different from the first node;

(d) after selecting the second node, issuing a start node request using the selected second node, the start node request indicating that the purpose of the start node request is for restarting the first node; and

(e) in response to the start node request, restarting the first node by initiating clustering on the first node.

15. A method of restarting a node among a plurality of nodes in a clustered computer system, wherein the clustered computer system hosts a cluster control group including a plurality of cluster control members, each residing respectively on a different node from the plurality of nodes, the method comprising:

(a) detecting a clustering failure on a firs node among the plurality of nodes;

(b) in response to detecting the clustering failure on the first node, issuing a membership change request from the first node to the cluster control member on each other node in the plurality of nodes, the membership change request indicating that the membership change request is for the purpose of restarting the first node;

(c) terminating clustering on the first node after issuing the membership change request;

(d) in response to the membership change request, selecting a second node from the plurality of nodes that is different from the first node;

(d) issuing a start node request using the selected second node, the start node request indicating that the purpose of the start node request is for restarting the first node;

(e) in response to the start node request, initiating clustering on the first node; and

(f) in response to a second clustering failure during initiation of clustering on the first node;

(i) determining from the start node request that initiated clustering on the first node that the purpose of the start node request is for restarting the first node; and

(ii) in response to determining that the start node request is for restarting the first node, signaling an error instead of initiating a second restart of the first node.

16. An apparatus, comprising:

(a) a memory accessible by a node in a clustered computer system; and

(b) a program resident in the memory, the program configured to initiate a restart of another node in the clustered computer system in response to a notification from the other node of a clustering failure on the other node, wherein the program comprises a member of a group hosted by the clustered computer system, the group including an additional member residing on the other node, wherein the notification comprises a request issued to the group by the additional member, and wherein the program is further configured to determine, in response to the request, that the node upon which the program is resident should initiate the restart of the other node, such that only one node in the clustered computer system initiates the restart of the other node in response to the request.

17. The apparatus of claim 16, wherein the group comprises a cluster control group that includes a member on each node participating in clustering in the clustered computer system.

18. The apparatus of claim 16, wherein the program is configured to initiate the restart by issuing a start node request to the group.

19. The apparatus of claim 18, wherein the start node request indicates that the start node request is for the purpose of restarting the first node.

20. The apparatus of claim 16, wherein the program is configured to initiate the restart of the other node responsive to a membership change request received from the other node.

21. The apparatus of claim 16, wherein the group comprises a cluster control group that includes a member on each node participating in clustering in the clustered computer system;

and wherein the program is configured to determine whether the membership change request indicates that the membership change request is for the purpose of restarting the first node.

22. A clustered computer system, comprising:

(a) first, second, and third nodes coupled to one another over a network; and

(b) a group including first, second, and third members, the first member resident on the first node, the second member resident on the second node, and the third member resident on the third node, wherein the first member is configured to notify the second and third members in response to a clustering failure on the first node by issuing a request to the group that is forwarded to each of the second and third members of the group, wherein the group is configured to select the second member to initiate a restart of the first node in response to the notification, and wherein the second member is configured to initiate the restart of the first node in response to the notification.

23. The clustered computer system of claim 22, wherein the first member is further configured to terminate clustering on the first node after notifying the second member.

24. The clustered computer system of claim 22, wherein the group comprises a cluster control group that includes a member on each node participating in clustering in the clustered computer system, and wherein the first and second members are each members of the cluster control group.

25. The clustered computer system of claim 22, wherein the first member is configured to notify the second member by issuing a membership change request to the group.

26. A clustered computer system, comprising:

(a) first and second nodes coupled to one another over a network; and

(b) a group including first and second members, the first member resident on the first node and the second member resident on the second node, wherein the first member is configured to notify the second member in response to a clustering failure on the first node, and wherein the second member is configured to initiate a restart of the first node in response to the notification;

wherein the first member is configured to detect the clustering failure in the first node, determine whether the clustering failure occurred during a restart of the first node, and notify the second member in response to detecting the clustering failure in the first node and determining that the clustering failure did not occur during a restart of the first node.

27. The clustered computer system of claim 26, wherein the first member is further configured to signal an error in response to detecting the clustering failure in the first node if the clustering failure occurred during a restart of the first node.

28. The clustered computer system of claim 26, wherein the second member is configured to initiate the restart by issuing a start node request to the group, and wherein the first member is configured to determine whether the clustering failure occurred during a restart of the first node by determining whether the start node request indicates that the start node request is for the purpose of restarting the first node.

29. The clustered computer system of claim 28, wherein the first member is configured to signal an error in response to detecting the clustering failure in the first node if the clustering failure occurred during a restart of the first node and a tracked number of protocols processed by the first node after the restart is less than a predetermined threshold.

30. The clustered computer system of claim 26, further comprising a third node coupled to the first and second nodes, wherein the group includes a third member resident on the third node, wherein the first member is configured to notify the third member in response to the clustering failure on the first node, and wherein each of the second third members is configured to locally select a single member in the group to initiate the restart of the first node.

31. The clustered computer system of claim 30, wherein the second member is configured to locally select the single member by determining whether the second member is a lowest member among the members of the group.

32. A program product, comprising:

(a) a program configured to reside on a node in a clustered computer system, the program configured to initiate a restart of another node in the clustered computer system in response to a notification from the other node of a clustering failure on the other node, wherein the program comprises a member of a group hosted by the clustered computer system, the group including an additional member residing on the other node, wherein the notification comprises a request issued to the group by the additional member, and wherein the program is further configured to determine, in response to the request, that the node upon which the program is resident should initiate the restart of the other node, such that only one node in the clustered computer system initiates the restart of the other node in response to the request; and

(b) a signal storage medium storing the program.

33. A program product, comprising:

(a) first, second and third programs respectively configured to reside on first, second and third nodes in a clustered computer system, the first, second and third programs respectively operating as first, second and third members of a group, the first program configured to notify the second program and the thrid program in response to a clustering failure on the first node by issuing a request to the group that is forwarded to each of the second and third members, the second program configured to, in response to the request, determine that the second program should initiate a restart of the first node, the second program further configured to initiate the restart of the first node after determining that the second program should initiate the restart, and the third program configured to defer to the second program to initiate the restart of the first node; and

(b) at least one signal storage medium storing the first and second programs.

34. The program product of claim 33, wherein the first and second programs are stored on separated signal storage medium.

35. The program product of claim 33, wherein the first and second programs are stored on a same signal storage medium.