QUORUM-BASED POWER-DOWN OF UNRESPONSIVE SERVERS IN A COMPUTER CLUSTER

• US 20080301491A1
• Filed: 08/15/2008
• Published: 12/04/2008
• Est. Priority Date: 11/04/2004
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Jones, Christopher Henry, Wallis, Graham Derek, Newport, William T.
• Granted Patent: US 7,908,251 B2
• Time in Patent Office: Days
• US Class Current: 714/4
• CPC Class Codes: H04L 12/12 Arrangements for remote connection or disconnection of substations or of equipment thereof
• CPC Class Codes: H04L 41/0681 Configuration of triggering conditions
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route
• CPC Class Codes: Y02D 30/50 in wire-line communication networks, e.g. low power modes or reduced link rate
• CPC Class Codes: Y10S 707/955 Object-oriented

Abstract

A quorum-based server power-down mechanism allows a manager in a computer cluster to power-down unresponsive servers in a manner that assures that an unresponsive server does not become responsive again. In order for a manager in a cluster to power down servers in the cluster, the cluster must have quorum, meaning that a majority of the computers in the cluster must be responsive. If the cluster has quorum, and if the manager server did not fail, the manager causes the failed server(s) to be powered down. If the manager server did fail, the new manager causes all unresponsive servers in the cluster to be powered down. If the power-down is successful, the resources on the failed server(s) may be failed over to other servers in the cluster that were not powered down. If the power-down is not successful, the cluster is disabled.

Claims

1. An apparatus comprising:

at least one processor;

a memory coupled to the at least one processor;

a server process residing in the memory and executed by the at least one processor;

a cluster engine residing in the memory and executed by the at least one processor, the cluster engine handling communications between the server process and other servers in a cluster; and

a quorum-based server power-down mechanism residing in the memory and executed by the at least one processor, the quorum-based server power-down mechanism determining whether the server process is part of a group of servers that include a majority of servers in the cluster, and if so, the quorum-based server power-down mechanism determines whether a manager of the cluster failed when an indication of a server failure is received, and if so, the quorum-based server power-down mechanism issues at least one command to power down all unresponsive servers in the cluster, wherein an unresponsive server is a server that fails to send a periodic message that indicates the server is functioning properly, and if the manager of the cluster did not fail, the quorum-based server power-down mechanism issues at least one command to power down a server corresponding to the received indication of server failure.

2. The apparatus of claim 1 further comprising a service processor that receives the command, and in response, powers down the at least one of the other servers.

3. The apparatus of claim 1 wherein the cluster engine comprises a heartbeat mechanism that sends a periodic message to the other servers in the cluster to indicate the server process is functioning properly.

4. The apparatus of claim 3 wherein the heartbeat mechanism receives periodic messages from the other servers in the cluster that indicate the other servers in the cluster are functioning properly.

5. The apparatus of claim 4 wherein the cluster engine comprises a membership change mechanism that generates a membership change message to all servers in the cluster when a selected server in the cluster becomes unresponsive by failing to send a periodic message that indicates the selected server is functioning properly.

6. The apparatus of claim 1 wherein the cluster engine comprises a membership change mechanism that generates a membership change message to all servers in the cluster when any of the servers in the cluster become unresponsive.

7. The apparatus of claim 1 wherein the server process resides in a logical partition defined on the apparatus.

8. The apparatus of claim 1 wherein the quorum-based server power-down mechanism determines whether the power down of the at least one of the other servers was successful, and if the power down of the at least one of the other servers was successful, the quorum-based server power-down mechanism enables failing over any resources on the at least one of the other servers that was powered down to at least one server that is responsive, and if the power down of the at least one of the other servers was not successful, the quorum-based server power-down mechanism disables the cluster.

9. A networked computer system comprising:

a plurality of servers coupled together via a network into a cluster, each server comprising;

a cluster engine that handles communications between servers in the cluster; and

a quorum-based server power-down mechanism that determines whether a server is part of a group of servers that includes a majority of servers in the cluster, and if so, the quorum-based server power-down mechanism determines whether a manager of the cluster failed when an indication of a server failure is received, and if so, the quorum-based server power-down mechanism issues at least one command to power down all unresponsive servers in the cluster, wherein an unresponsive server is a server that fails to send a periodic message that indicates the server is functioning properly, and if the manager of the cluster did not fail, the quorum-based server power-down mechanism issues at least one command to power down a server corresponding to the received indication of server failure.

10. The networked computer system of claim 9 further comprising a service processor coupled to each of the plurality of servers in the cluster, the service processor receiving the command, and in response, powering down the at least one server in the cluster.

11. The networked computer system of claim 9 wherein the cluster engine comprises a heartbeat mechanism that sends a periodic message to other servers in the cluster to indicate the server is functioning properly.

12. The networked computer system of claim 11 wherein the heartbeat mechanism receives periodic messages from the other servers in the cluster that indicate the other servers in the cluster are functioning properly.

13. The networked computer system of claim 12 wherein the cluster engine comprises a membership change mechanism that generates a membership change message to all servers in the cluster when a selected server in the cluster becomes unresponsive by failing to send a periodic message that indicates the selected server is functioning properly.

14. The networked computer system of claim 9 wherein the cluster engine comprises a membership change mechanism that generates a membership change message to all servers in the cluster when any of the servers in the cluster become unresponsive.

15. The networked computer system of claim 9 wherein at least one server in the cluster resides in a logical partition.

16. The networked computer system of claim 9 wherein the quorum-based server power-down mechanism determines whether the power down of the at least one of the other servers was successful, and if the power down of the at least one of the other servers was successful, the quorum-based server power-down mechanism enables failing over any resources on the at least one of the other servers that was powered down to at least one server that is responsive, and if the power down of the at least one of the other servers was not successful, the quorum-based server power-down mechanism disables the cluster.

17. A computer readable recordable medium bearing a computer program, the computer program comprising:

a quorum-based server power-down mechanism that determines whether a server is part of a group of servers that include a majority of servers in a cluster, and if so, the quorum-based server power-down mechanism determines whether a manager of the cluster failed when an indication of a server failure is received, and if so, the quorum-based server power-down mechanism issues at least one command to power down all unresponsive servers in the cluster, wherein an unresponsive server is a server that fails to send a periodic message that indicates the server is functioning properly, and if the manager of the cluster did not fail, the quorum-based server power-down mechanism issues at least one command to power down a server corresponding to the received indication of server failure.

18. The computer readable recordable medium of claim 17 wherein the computer program further comprises a heartbeat mechanism that sends a periodic message to the other servers in the cluster to indicate the server process is functioning properly.

19. The computer readable recordable medium of claim 18 wherein the heartbeat mechanism receives periodic messages from the other servers in the cluster that indicate the other servers in the cluster are functioning properly.

20. The computer readable recordable medium of claim 19 wherein the computer program further comprises a membership change mechanism that generates a membership change message to all servers in the cluster when a selected server in the cluster becomes unresponsive by failing to send a periodic message that indicates the selected server is functioning properly.

21. The computer readable recordable medium of claim 17 wherein the computer program further comprises a membership change mechanism that generates a membership change message to all servers in the cluster when any of the servers in the cluster become unresponsive.

22. The computer readable recordable medium of claim 17 wherein at least one server in the cluster resides in a logical partition.

23. The computer readable recordable medium of claim 17 wherein the quorum-based server power-down mechanism determines whether the power down of the at least one of the other servers was successful, and if the power down of the at least one of the other servers was successful, the quorum-based server power-down mechanism enables failing over any resources on the at least one of the other servers that was powered down to at least one server that is responsive, and if the power down of the at least one of the other servers was not successful, the quorum-based server power-down mechanism disables the cluster.