Apparatus and method for quorum-based power-down of unresponsive servers in a computer cluster

• US 20060100981A1
• Filed: 11/04/2004
• Published: 05/11/2006
• Est. Priority Date: 11/04/2004
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Jones, Christopher Henry, Wallis, Graham Derek, Newport, William T.
• Application Number:
  US10/981,020
• Publication Number:
  US 20060100981A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04L 12/12 Arrangements for remote connection or disconnection of substations or of equipment thereof
• CPC Class Codes: H04L 41/0681 Configuration of triggering conditions
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route
• CPC Class Codes: Y02D 30/50 in wire-line communication networks, e.g. low power modes or reduced link rate
• CPC Class Codes: Y10S 707/955 Object-oriented

Abstract

An apparatus and method provide a quorum-based server power-down mechanism that allows a manager in a computer cluster to power-down unresponsive servers in a manner that assures that an unresponsive server does not become responsive again. In order for a manager in a cluster to power down servers in the cluster, the cluster must have quorum, meaning that a majority of the computers in the cluster must be responsive. If the cluster has quorum, and if the manager server did not fail, the manager causes the failed server(s) to be powered down. If the manager server did fail, the new manager causes all unresponsive servers in the cluster to be powered down. If the power-down is successful, the resources on the failed server(s) may be failed over to other servers in the cluster that were not powered down. If the power-down is not successful, the cluster is disabled.

Claims

1. An apparatus comprising:

at least one processor;

a memory coupled to the at least one processor;

a server process residing in the memory and executed by the at least one processor;

a cluster engine residing in the memory and executed by the at least one processor, the cluster engine handling communications between the server process and other servers in a cluster; and

a quorum-based server power-down mechanism residing in the memory and executed by the at least one processor, the quorum-based server power-down mechanism determining whether the server process is part of a group of servers that include a majority of servers in the cluster, and if so, the quorum-based server power-down mechanism issuing a command to power down at least one of the other servers in the cluster.

2. The apparatus of claim 1 further comprising a service processor that receives the command, and in response, powers down the at least one of the other servers.

3. The apparatus of claim 1 wherein the cluster engine comprises a heartbeat mechanism that sends a periodic message to the other servers in the cluster to indicate the server process is functioning properly.

4. The apparatus of claim 3 wherein the heartbeat mechanism receives periodic messages from the other servers in the cluster that indicate the other servers in the cluster are functioning properly.

5. The apparatus of claim 4 wherein the cluster engine comprises a membership change mechanism that generates a membership change message to all servers in the cluster when a selected server in the cluster becomes unresponsive by failing to send a periodic message that indicates the selected server is functioning properly.

6. The apparatus of claim 1 wherein the cluster engine comprises a membership change mechanism that generates a membership change message to all servers in the cluster when any of the servers in the cluster become unresponsive.

7. The apparatus of claim 1 wherein the quorum-based server power-down mechanism determines whether a manager of the cluster failed, and if so, the quorum-based server power-down mechanism issues at least one command to power down all unresponsive servers in the cluster.

8. The apparatus of claim 1 wherein the quorum-based server power-down mechanism determines whether a manager of the cluster failed when an indication of a server failure is received, and if not, the quorum-based server power-down mechanism issues at least one command to power down a server corresponding to the received indication of server failure.

9. The apparatus of claim 1 wherein the server process resides in a logical partition defined on the apparatus.

10. The apparatus of claim 1 wherein the quorum-based server power-down mechanism determines whether the power down of the at least one of the other servers was successful, and if the power down of the at least one of the other servers was successful, the quorum-based server power-down mechanism enables failing over any resources on the at least one of the other servers that was powered down to at least one server that is responsive, and if the power down of the at least one of the other servers was not successful, the quorum-based server power-down mechanism disables the cluster.

11. A networked computer system comprising:

a plurality of servers coupled together via a network into a cluster, each server comprising;

a cluster engine that handles communications between servers in the cluster; and

a quorum-based server power-down mechanism that determines whether a server is part of a group of servers that includes a majority of servers in the cluster, and if so, the quorum-based server power-down mechanism issuing a command to power down at least one server in the cluster.

12. The networked computer system of claim 11 further comprising a service processor coupled to each of the plurality of servers in the cluster, the service processor receiving the command, and in response, powering down the at least one server in the cluster.

13. The networked computer system of claim 11 wherein the cluster engine comprises a heartbeat mechanism that sends a periodic message to other servers in the cluster to indicate the server is functioning properly.

14. The networked computer system of claim 13 wherein the heartbeat mechanism receives periodic messages from the other servers in the cluster that indicate the other servers in the cluster are functioning properly.

15. The networked computer system of claim 14 wherein the cluster engine comprises a membership change mechanism that generates a membership change message to all servers in the cluster when a selected server in the cluster becomes unresponsive by failing to send a periodic message that indicates the selected server is functioning properly.

16. The networked computer system of claim 11 wherein the cluster engine comprises a membership change mechanism that generates a membership change message to all servers in the cluster when any of the servers in the cluster become unresponsive.

17. The networked computer system of claim 11 wherein the quorum-based server power-down mechanism determines whether a manager of the cluster failed, and if so, the quorum-based server power-down mechanism issues at least one command to power down all unresponsive servers in the cluster.

18. The networked computer system of claim 11 wherein the quorum-based server power-down mechanism determines whether a manager of the cluster failed when an indication of a server failure is received, and if not, the quorum-based server power-down mechanism issues at least one command to power down a server corresponding to the received indication of server failure.

19. The networked computer system of claim 11 wherein at least one server in the cluster resides in a logical partition.

20. The networked computer system of claim 11 wherein the quorum-based server power-down mechanism determines whether the power down of the at least one of the other servers was successful, and if the power down of the at least one of the other servers was successful, the quorum-based server power-down mechanism enables failing over any resources on the at least one of the other servers that was powered down to at least one server that is responsive, and if the power down of the at least one of the other servers was not successful, the quorum-based server power-down mechanism disables the cluster.

21. An apparatus comprising:

(A) at least one processor;

(B) a memory coupled to the at least one processor;

(C) a server process residing in the memory and executed by the at least one processor;

(D) a cluster engine residing in the memory and executed by the at least one processor, the cluster engine handling communications between the server process and other servers in a cluster, the cluster engine comprising;

(D1) a heartbeat mechanism that sends a periodic message to the other servers in the cluster to indicate the server process is functioning properly and that receives periodic messages from the other servers in the cluster that indicate the other servers in the cluster are functioning properly;

(D2) a membership change mechanism that generates a membership change message to all servers in the cluster when any of the servers in the cluster become unresponsive;

(E) a quorum-based server power-down mechanism residing in the memory and executed by the at least one processor, the quorum-based server power-down mechanism determining whether the server process is part of a group of servers that includes a majority of servers in the cluster, and if so, the quorum-based server power-down mechanism determining whether a manager of the cluster failed when an indication of a server failure is received, and if a manager of the cluster failed, the quorum-based server power-down mechanism issues at least one command to power down all unresponsive servers in the cluster, and if a manager of the cluster did not fail, the quorum-based server power-down mechanism issues at least one command to power down a server corresponding to the received indication of server failure.

22. The apparatus of claim 21 further comprising a service processor that receives the command, and in response, powers down at least one of the other servers.

23. The apparatus of claim 21 wherein the server process resides in a logical partition defined on the apparatus.

24. The apparatus of claim 21 wherein the quorum-based server power-down mechanism determines whether the power down of the at least one of the other servers was successful, and if the power down of the at least one of the other servers was successful, the quorum-based server power-down mechanism enables failing over any resources on the at least one of the other servers that was powered down to at least one server that is responsive, and if the power down of the at least one of the other servers was successful, the quorum-based server power-down mechanism disables the cluster.

25. A computer-implemented method for handling an unresponsive server in a cluster, the method comprising the steps of:

determining when a server in the cluster becomes unresponsive;

determining whether a majority of servers in the cluster are responsive; and

if a majority of servers in the cluster are responsive, issuing a command to power down at least one server in the cluster.

26. The method of claim 25 further comprising the step of each server in the cluster sending a periodic message to other servers in the cluster to indicate proper function of the server sending the periodic message.

27. The method of claim 25 further comprising the step of each server receiving periodic messages from other servers in the cluster that indicate the other servers in the cluster are functioning properly.

28. The method of claim 27 further comprising the step of generating a membership change message to all servers in the cluster when a selected server in the cluster becomes unresponsive by failing to send a periodic message that indicates the selected server is functioning properly.

29. The method of claim 25 further comprising the step of generating a membership change message to all servers in the cluster when any of the servers in the cluster become unresponsive.

30. The method of claim 25 further comprising the step of determining whether a manager of the cluster failed, and if so, issuing at least one command to power down all unresponsive servers in the cluster.

31. The method of claim 25 further comprising the step of determining whether a manager of the cluster failed when an indication of a server failure is received, and if not, issuing at least one command to power down a server corresponding to the received indication of server failure.

32. The method of claim 25 further comprising the steps of:

determining whether the power down of the at least one of the other servers was successful;

if the power down of the at least one of the other servers was successful, enabling the failing over any resources on the at least one of the other servers that was powered down to at least one server that is responsive; and

if the power down of the at least one of the other servers was not successful, disabling the cluster.

33. A computer-implemented method for handling an unresponsive server in a cluster, the method comprising the steps of:

each server in the cluster sending a periodic message to other servers in the cluster to indicate proper function of the server sending the periodic message;

each server in the cluster receiving periodic messages from other servers in the cluster that indicate the other servers in the cluster are functioning properly;

generating a membership change message to all servers in the cluster when any of the servers in the cluster become unresponsive;

determining whether a majority of servers in the cluster are responsive;

receiving an indication of a server failure;

if the majority of servers in the cluster are responsive, performing the steps of;

determining whether the indication of the server failure indicates a manager of the cluster failed;

if the manager of the cluster failed, issuing at least one command to power down all unresponsive servers in the cluster; and

if the manager of the cluster did not fail, issuing at least one command to power down a server corresponding to the received indication of server failure.

34. The method of claim 33 further comprising the steps of:

determining whether the power down of the at least one of the other servers was successful;

if the power down of the at least one of the other servers was successful, enabling the failing over any resources on the at least one of the other servers that was powered down to at least one server that is responsive; and

if the power down of the at least one of the other servers was not successful, disabling the cluster.

35. A program product comprising:

a quorum-based server power-down mechanism that determines whether a server is part of a group of servers that include a majority of servers in a cluster, and if so, the quorum-based server power-down mechanism issues a command to power down at least one server in the cluster; and

computer readable signal bearing media bearing the quorum-based server power-down mechanism.

36. The program product of claim 35 wherein the signal bearing media comprises recordable media.

37. The program product of claim 35 wherein the signal bearing media comprises transmission media.

38. The program product of claim 35 further comprising a heartbeat mechanism that sends a periodic message to the other servers in the cluster to indicate the server process is functioning properly.

39. The program product of claim 38 wherein the heartbeat mechanism receives periodic messages from the other servers in the cluster that indicate the other servers in the cluster are functioning properly.

40. The program product of claim 39 further comprising a membership change mechanism that generates a membership change message to all servers in the cluster when a selected server in the cluster becomes unresponsive by failing to send a periodic message that indicates the selected server is functioning properly.

41. The program product of claim 35 further comprising a membership change mechanism that generates a membership change message to all servers in the cluster when any of the servers in the cluster become unresponsive.

42. The program product of claim 35 wherein the quorum-based server power-down mechanism determines whether a manager of the cluster failed, and if so, the quorum-based server power-down mechanism issues at least one command to power down all unresponsive servers in the cluster.

43. The program product of claim 35 wherein the quorum-based server power-down mechanism determines whether a manager of the cluster failed when an indication of a server failure is received, and if not, the quorum-based server power-down mechanism issues at least one command to power down a server corresponding to the received indication of server failure.

44. The program product of claim 35 wherein at least one server in the cluster resides in a logical partition.

45. The program product of claim 35 wherein the quorum-based server power-down mechanism determines whether the power down of the at least one of the other servers was successful, and if the power down of the at least one of the other servers was successful, the quorum-based server power-down mechanism enables failing over any resources on the at least one of the other servers that was powered down to at least one server that is responsive, and if the power down of the at least one of the other servers was not successful, the quorum-based server power-down mechanism disables the cluster.

46. A program product comprising:

(A) a cluster engine residing in the memory and executed by the at least one processor, the cluster engine handling communications between a plurality of servers in a cluster, the cluster engine comprising;

(A1) a heartbeat mechanism that sends a periodic message to the other servers in the cluster to indicate the server process is functioning properly and that receives periodic messages from the other servers in the cluster that indicate the other servers in the cluster are functioning properly;

(A2) a membership change mechanism that generates a membership change message to all servers in the cluster when any of the servers in the cluster become unresponsive; and

(A3) a quorum-based server power-down mechanism that determines whether the server process is part of a group of servers that includes a majority of servers in the cluster, and if so, the quorum-based server power-down mechanism determines whether a manager of the cluster failed when an indication of a server failure is received, and if a manager of the cluster failed, the quorum-based server power-down mechanism issues at least one command to power down all unresponsive servers in the cluster, and if a manager of the cluster did not fail, the quorum-based server power-down mechanism issues at least one command to power down a server corresponding to the received indication of server failure; and

(B) computer readable signal bearing media bearing the cluster engine.

47. The program product of claim 46 wherein the signal bearing media comprises recordable media.

48. The program product of claim 46 wherein the signal bearing media comprises transmission media.

49. The program product of claim 46 wherein at least one server in the cluster resides in a logical partition.

50. The program product of claim 46 wherein the quorum-based server power-down mechanism determines whether the power down of the at least one of the other servers was successful, and if the power down of the at least one of the other servers was successful, the quorum-based server power-down mechanism enables failing over any resources on the at least one of the other servers that was powered down to at least one server that is responsive, and if the power down of the at least one of the other servers was not successful, the quorum-based server power-down mechanism disables the cluster.