Flexible failover policies in high availability computing systems

• US 8,769,132 B2
• Filed: 09/27/2010
• Issued: 07/01/2014
• Est. Priority Date: 03/16/2000
• Status: Expired due to Fees

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment America LLC (Sony Group Corp.)
• Inventors: Sreenivasan, Padmanabhan, Dandapani, Ajit, Nishimoto, Michael, Pramanick, Ira, Verma, Manish, Bradshaw, Robert D. Jr., Castellano, Luca, Mallena, Raghu
• Primary Examiner(s): Zong, Ruolei
• Application Number:
  US12/891,390
• Publication Number:
  US 20110214007A1
• Time in Patent Office: 1,373 Days
• Field of Search: 709/229, 709/218, 709223-228, 709/239, 709/245, 714/3, 714/4, 714/6, 714/7, 714/10, 714/11, 714/13, 714/17, 713/1, 713/100, 340/2.1, 370/420
• US Class Current: 709/229
• CPC Class Codes: G06F 11/2023 Failover techniques
• CPC Class Codes: G06F 11/2033 switching over of hardware resources
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

A system for implementing a failover policy includes a cluster infrastructure for managing a plurality of nodes, a high availability infrastructure for providing group and cluster membership services, and a high availability script execution component operative to receive a failover script and at least one failover attribute and operative to produce a failover domain. In addition, a method for determining a target node for a failover comprises executing a failover script that produces a failover domain, the failover domain having an ordered list of nodes, receiving a failover attribute and based on the failover attribute and failover domain, selecting a node upon which to locate a resource.

Claims

1. A system for implementing a failover policy in a network, the system comprising:

a processor; and

memory storing;

the failover policy, wherein the failover policy includes a failover script, at least one failover attribute, and a predetermined initial failover domain; and

instructions that when executed by the processor;

manage a plurality of nodes arranged as one or more clusters;

provide a plurality of group and cluster membership services;

define a plurality of resource groups, wherein each resource group includes two or more interdependent resources; and

execute the failover script in response to detecting a failover event, wherein the execution of the failover script;

dynamically transforms the predetermined initial failover domain into a dynamic run-time failover domain upon occurrence of a failover event, wherein the initial failover domain and the dynamic run-time failover domain are respective ordered lists of nodes in the network selected from a plurality of nodes on which a given resource group can be allocated, and wherein the dynamic transformation of the initial failover domain into the dynamic run-time failover domain is based on the contents of the failover script and a plurality of run-time conditions existing at the time of the failover event,

modifies the dynamic run-time failover domain for a respective one of the plurality of resource groups based on the at least one failover attribute,

selects a target node from the modified run-time failover domain based on the at least one failover attribute, and

allocates the given resource to the target node.

2. The system of claim 1, wherein the one or more resources includes an application and an application plug-in that provides a high-availability interface for the application.

3. A method for determining a target node for a failover, comprising:

detecting a failover event;

executing a failover script stored in memory of a computing device, the failover script dynamically transforming an initial failover domain into a dynamic run-time failover domain, wherein the dynamic transformation of the initial failover domain into the dynamic run-time failover domain is based on the contents of the failover script and a plurality of run-time conditions existing at the time of the failover event, the initial and run-time failover domains each having a respective ordered list of nodes selected from a plurality of nodes on which a given resource group can be allocated;

receiving a failover attribute associated with the failover event;

modifying the run-time failover domain for the given resource group based on the failover attribute; and

selecting from the modified run-time failover domain a node upon which to locate the given resource group.

4. The method of claim 3, wherein selecting the node includes selecting a first node in the respective ordered list of nodes of the run-time failover domain.

5. A system for implementing a failover policy in a network, the system comprising:

a server having a processor and memory, the memory storing;

the failover policy, wherein the failover policy includes a failover script, at least one failover attribute, and a predetermined initial failover domain, wherein the failover script includes a set of one or more commands; and

instructions that when executed by the processor;

manage a plurality of nodes;

provide group and cluster membership services; and

execute the failover script in response to detecting a failover event, wherein execution of the failover script;

produces a dynamic run-time failover domain from the predetermined initial failover domain, wherein the dynamic run-time failover domain and the predetermined initial failover domain are respective ordered lists of nodes in the network selected from the plurality of nodes on which a given resource group having one or more resources can be allocated,

modifies the run-time failover domain for the given resource group based on the received failover attribute, and

executes one or more action scripts that when executed cause the given resource group to failover to a target node selected from the respective ordered list of nodes in the dynamic run-time failover domain.

6. The system of claim 5, wherein the failover event includes failure of a node.

7. The system of claim 5, wherein the failover event includes a load-balancing event independent of node failure.

8. The system of claim 5, wherein the one or more resources includes an application and an application plug-in that provides a high-availability interface for the application.

9. A method for configuring nodes in a cluster, comprising:

defining a plurality of resource groups, wherein each resource group is a respective collection of interdependent resources;

defining, for each resource group, a respective failover policy for the group, wherein the respective failover policy includes a respective initial failover domain, one or more respective failover attributes and a respective failover script, and wherein the respective initial failover domain is a respective ordered list of nodes in the cluster on which the resource group can be allocated;

detecting a failover event;

executing the respective failover script for a particular resource group upon detecting the failover event, the respective failover script including a set of one or more commands that, when executed, dynamically determine a respective run-time failover domain from the respective initial failover domain associated with the particular resource group;

receiving a failover attribute;

modifying the respective run-time failover domain determined for the particular resource group based on the received failover attribute;

selecting a target node upon which to allocate the particular resource group from the respective run-time failover domain determined and modified for the particular resource group, the selection based on the received failure attribute; and

executing one or more action scripts that when executed cause the particular resource group to failover to the selected target node.

10. The method of claim 9, wherein selecting the target node includes selecting a first node in the respective ordered list of nodes of the respective run-time failover domain determined for the particular resource group.

11. The method of claim 9, wherein the action script verifies that the resource is configured on the target node.

12. The method of claim 9, wherein the action script verifies that the resource is not already running on the target node.

13. The method of claim 9, wherein the failover event includes failure of a node.

14. The method of claim 9, wherein the failover event includes a load balancing event independent of node failure.

15. The method of claim 9, further comprising:

storing the respective run-time failover domain in memory;

detecting a second failover event; and

executing the respective failover script for the particular resource group upon detection of the second failover event, wherein the respective run-time failover domain is provided as input to the respective failover script for the particular resource group, and wherein the respective failover script dynamically determines a second run-time failover domain for the particular resource group.