Dynamic Updating of Failover Policies for Increased Application Availability

• US 20110022882A1
• Filed: 07/21/2009
• Published: 01/27/2011
• Est. Priority Date: 07/21/2009
• Status: Active Grant

• Current Assignee: Taiwan Semiconductor Manufacturing Company Limited
• Original Assignee: International Business Machines Corporation
• Inventors: Weaver, Thomas V., Muttur, Manjunath B., Jaehde, Astrid A.
• Granted Patent: US 8,055,933 B2
• Time in Patent Office: Days
• US Class Current: 714/4
• CPC Class Codes: G06F 11/2015 Redundant power supplies power supply failure G06F1/30
• CPC Class Codes: G06F 11/2023 Failover techniques
• CPC Class Codes: G06F 11/203 using migration
• CPC Class Codes: G06F 11/2035 without idle spare hardware
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence
• CPC Class Codes: G06F 11/3409 for performance assessment

Abstract

Mechanisms are provided for performing a failover operation of an application from a faulty node of a high availability cluster to a selected target node. The mechanisms receive a notification of an imminent failure of the faulty node. The mechanisms further receive health information from nodes of a local failover scope of a failover policy associated with the faulty node. Moreover, the mechanisms dynamically modify the failover policy based on the health information from the nodes of the local failover scope and select a node from the modified failover policy as a target node for failover of an application running on the faulty node to the target node. Additionally, the mechanisms perform failover of the application to the target node based on the selection of the node from the modified failover policy.

Claims

1. A method, in a data processing system, for performing a failover operation of an application from a faulty node of a high availability cluster to a selected target node, comprising:

receiving a notification of an imminent failure of the faulty node;

receiving health information from one or more additional nodes of a local failover scope of a failover policy associated with the faulty node, wherein the faulty node and the one or more additional nodes are members of the local failover scope;

dynamically modifying the failover policy based on the health information from the one or more additional nodes of the local failover scope;

selecting a node from the modified failover policy as the target node for failover of an application running on the faulty node to the target node; and

performing failover of the application to the target node based on the selection of the node from the modified failover policy.

2. The method of claim 1, wherein the health information from the one or more additional nodes of the local failover scope identifies a measure of survivability of each additional node in the one or more additional nodes with regard to a cause of the imminent failure of the faulty node, wherein the method further comprises:

determining which additional nodes, if any, in the one or more additional nodes of the local failover scope are affected by the cause of the imminent failure of the faulty node; and

determining which additional nodes, if any, in the one or more additional nodes of the local failover scope that are not affected by the cause of the imminent failure of the faulty node, wherein the failover policy is dynamically modified based on the determination of which additional nodes, if any, are affected or not affected by the cause of the imminent failure of the faulty mode.

3. The method of claim 1, wherein the target node is one of the one or more additional nodes of the local failover scope or a node of a remote failover scope that is geographically remote from the one or more additional nodes of the local failover scope.

4. The method of claim 1, wherein the health information from the one or more additional nodes of the local failover scope comprises, for each additional node in the one or more additional nodes information identifying an amount of time the additional node can remain operational based on a backup power supply.

5. The method of claim 4, wherein selecting a node from the modified failover policy as the target node for failover of the application running on the faulty node to the target node comprises selecting the node based on the amount of time the additional nodes can remain operational based on their respective backup power supplies and resource utilization information from the one or more additional nodes.

6. The method of claim 5, wherein selecting a node from the modified failover policy as the target node for failover of the application running on the faulty node to the target node comprises selecting the node from the modified failover policy based on the health information and infrastructure information identifying at least one of a physical arrangement or electrical arrangement of the faulty node and one or more additional nodes in the local failover scope.

7. The method of claim 6, wherein the infrastructure information comprises information about a topology of connectivity of nodes to backup power supply sources.

8. The method of claim 7, further comprising:

modifying a configuration of the one or more additional nodes of the local failover scope, so as to consolidate resource utilization on a selected subset of the one or more additional nodes, based on the health information and the infrastructure information.

9. The method of claim 8, further comprising:

in response to a detection of a restoration of the faulty node, returning the configuration of the one or more additional nodes of the local failover scope to an original configuration in which resource utilization is no longer consolidated in the selected subset of the one or more additional nodes.

10. The method of claim 1, further comprising:

modifying a resource utilization of the application on the faulty node so as to maximize an amount of time that the faulty node can remain operational on battery backup power.

11. The method of claim 1, wherein selecting a node from the modified failover policy as the target node for failover of the application running on the faulty node to the target node comprises:

performing a collaboration between instances of a clusterware applications running on each of the nodes of the faulty node and the one or more additional nodes to select the target node.

12. The method of claim 1, wherein receiving health information from one or more additional nodes of a local failover scope of a failover policy associated with the faulty node comprises receiving heartbeat signals from the one or more additional nodes, wherein the heartbeat signals include an indication of an amount of time a corresponding node in the one or more additional nodes is able to remain operational in view of the cause of the fault on the faulty node.

13. The method of claim 1, wherein the faulty node and the one or more additional nodes have different uninterruptable power supply (UPS) capabilities and wherein the faulty node and the one or more additional nodes comprise a UPS monitor for monitoring a state of the UPS to identify power failures and an amount of available power from a backup battery of the UPS.

14. The method of claim 1, wherein selecting a node from the modified failover policy as the target node for failover of the application running on the faulty node to the target node comprises:

determining a scope of affect of a cause of the imminent failure of the faulty node;

determining an amount of time that the faulty node can operate on a battery backup power supply based on health information reported by the faulty node; and

selecting the target node as a remote node from a remote failover scope, comprising nodes that are physically located at a remote site from that of the faulty node and the one or more additional nodes in the local failover scope, in response to a determination that the scope of the cause of the imminent failure affects all of the faulty nodes and the one or more additional nodes arid the health information reported by the faulty node indicates that the faulty node is able to operate on battery backup power for a sufficient amount of time to complete a failover operation to the remote node.

15. The method of claim 14, wherein selecting a node from the modified failover policy as the target node for failover of the application running on the faulty node to the target node comprises:

determining a scope of affect of a cause of the imminent failure of the faulty node;

determining an amount of time that the faulty node can operate on a battery backup power supply based on health information reported by the faulty node; and

selecting the target node as a node in the one or more additional nodes of the local failover scope in response to a determination that the amount of time that the faulty node can-operate oh the battery backup power supply is less than a required amount of time needed to complete a failover operation to a remote node of a remote failover scope comprising nodes that are physically located at a remote site from that of the faulty node and the one or more additional nodes in the local failover scope.

16. The method of claim 15, wherein performing the failover of the application comprises performing the failover of the application to both a target node selected from the one or more additional nodes of the local failover scope and a remote node in the remote failover scope at approximately a same time, such that the failover of the application to the remote node is performed as long as the faulty node has power from the backup power supply at which point the failover of the application to the remote node terminates.

17. The method of claim 16, wherein, in the event that the target node detects an imminent failure of the target node, the failover of the application to the remote node is continued from a point at which the failover of the application terminated.

18. The method of claim 1, wherein the faulty node and the one or more additional nodes are processor cores of one or more data processing devices.

19. A computer program product comprising a computer recordable medium having a computer readable program recorded thereon, wherein the computer readable program, when executed on a computing device, causes the computing device to:

receive a notification of an imminent failure of a faulty node;

receive health information from one or more additional nodes of a local failover scope of a failover policy associated with the faulty node, wherein the faulty node and the one or more additional nodes are members of the local failover scope;

dynamically modify a failover policy based on the health information from the one or more additional nodes of the local failover scope;

select a node from the modified failover policy as the target node for failover of an application running on the faulty node to the target node; and

perform failover of the application to the target node based on the selection of the node from the modified failover policy.

20. An apparatus, comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

receive a notification of an imminent failure of a faulty node;

receive health information from one or more additional nodes of a local failover scope of a failover policy associated with the faulty node, wherein the faulty node and the one or more additional nodes are members of the local failover scope;

dynamically modify a failover policy based on the health information from the one or more additional nodes of the local failover scope;

select a node from the modified failover policy as the target node for failover of an application running on the faulty node to the target node; and

perform failover of the application to the target node based on the selection of the node from the modified failover policy.