System and method for handling multi-node failures in a disaster recovery cluster

• US 9,811,428 B2
• Filed: 12/03/2014
• Issued: 11/07/2017
• Est. Priority Date: 09/22/2014
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Ramasubramaniam, Vaiapuri, Kadayam, Harihara, Sarfare, Parag, Cho, Yong Eun, Patel, Chaitanya, Keremane, Hrishikesh, Deshmukh, Prachi
• Primary Examiner(s): Maskulinski, Michael
• Application Number:
  US14/559,343
• Publication Number:
  US 20160085647A1
• Time in Patent Office: 1,070 Days
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/2033 switching over of hardware resources
• CPC Class Codes: G06F 11/2041 with more than one idle spare processing component
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 2201/805 Real-time

Abstract

A system and method for handling multi-node failures in a disaster recovery cluster is provided. In the event of an error condition, a switchover operation occurs from the failed nodes to one or more surviving nodes. Data stored in non-volatile random access memory is recovered by the surviving nodes to bring storage objects, e.g., disks, aggregates and/or volumes into a consistent state.

Claims

1. A method comprising:

determining that a candidate node is not available for a switchover operation to takeover for a failed node of a plurality of nodes;

identifying an alternate node for the switchover operation based upon a determination that the alternate node has resources capable of handling a load of the failed node and that an object limit option for the alternate node is disabled;

performing the switchover operation to directly transfer ownership of a storage object of the failed node to the alternate node without transferring ownership of the storage object to the candidate node;

recovering data from a non-volatile memory to the storage object; and

bringing the storage object online.

2. The method of claim 1, wherein the determining comprises:

determining that the candidate node is not available based upon a determination that the candidate node would service a number of storage objects that exceeds an object limit if the candidate node was used for the switchover operation.

3. The method of claim 1, wherein the determining comprises:

determining that the candidate node is not available based upon a determination that the candidate node is suffering an error condition.

4. The method of claim 1, wherein the plurality of nodes comprises a first node arranged in a high availability pairing with a second node.

5. The method of claim 4, comprising:

detecting a failure on the first node; and

performing a failover operation from the first node to the second node.

6. The method of claim 1, wherein the identifying comprises:

identifying the alternate node based upon a determination that the alternate node would not service a number of storage objects that exceeds an object limit if the alternate node was used for the switchover operation.

7. The method of claim 1, wherein recovering data from the non-volatile memory comprises:

replaying a portion of data stored in the non-volatile memory that was mirrored from the failed node.

8. The method of claim 1, wherein the identifying comprises:

determining that a potential alternate node is not available based upon a determination that the potential alternate node would service a number of storage objects that exceeds an object limit if the potential alternate node was used for the switchover operation.

9. The method of claim 8, comprising:

modifying the object limit to create an increased object limit for the potential alternate node; and

determining that the potential alternate node is available as the alternate node based upon a subsequent determination that the potential alternate node would not service a number of storage objects that exceeds the increased object limit if the potential alternate node was used for the switchover operation.

10. A computing device comprising:

a memory containing computer readable medium having stored thereon instructions for performing a method; and

a processor coupled with the memory, the processor configured to execute the instructions to cause the processor to;

determine that a candidate node is not available for a switchover operation to takeover for a failed node of a plurality of nodes;

identify an alternate node for the switchover operation based upon a determination that the alternate node has resources capable of handling a load of the failed node and that an object limit option for the alternate node is disabled;

perform the switchover operation to directly transfer ownership of a storage object of the failed node to the alternate node without transferring ownership of the storage object to the candidate node;

recover data from a non-volatile memory to the storage object; and

bring the storage object online.

11. The computing device of claim 10, wherein the instructions cause the processor to:

determine that the candidate node is not available based upon a determination that the candidate node would service a number of storage objects that exceeds an object limit if the candidate node was used for the switchover operation.

12. The computing device of claim 10, wherein the instructions cause the processor to:

determine that the candidate node is not available based upon a determination that the candidate node is suffering an error condition.

13. The computing device of claim 10, wherein the instructions cause the processor to:

identify the alternate node based upon a determination that the alternate node would not service a number of storage objects that exceeds an object limit if the alternate node was used for the switchover operation.

14. The computing device of claim 10, wherein the instructions cause the processor to:

replay a portion of data stored in the non-volatile memory that was mirrored from the failed node.

15. The computing device of claim 10, wherein the instructions cause the processor to:

determine that a potential alternate node is not available for the switchover operation based upon a determination that the potential alternate node would service a number of storage objects that exceeds an object limit if the potential alternate node was used for the switchover operation.

16. The computing device of claim 15, wherein the instructions cause the processor to:

modify the object limit to create an increased object limit for the potential alternate node.

17. The computing device of claim 16, wherein the instructions cause the processor to:

determine that the potential alternate node is available for the switchover operation based upon a subsequent determination that the potential alternate node would not service a number of storage objects that exceeds the increased object limit if the potential alternate node was used for the switchover operation.

18. The computing device of claim 10, wherein the storage object is stored within a storage device that is modified, based upon the switchover operation, to conform to the alternate node.

19. A computer readable medium comprising program instructions, that when executed by a processor, cause the processor to:

determine that a candidate node is not available for a switchover operation to takeover for a failed node of a plurality of nodes;

identify an alternate node for the switchover operation based upon a determination that the alternate node has resources capable of handling a load of the failed node and that an object limit option for the alternate node is disabled;

perform the switchover operation to directly transfer ownership of a storage object of the failed node to the alternate node without transferring ownership of the storage object to the candidate node;

recover data from a non-volatile memory to the storage object; and

bring the storage object online.

20. The computer readable medium of claim 19, wherein the program instructions cause the processor to:

replay a portion of data stored in the non-volatile memory that was mirrored from the failed node.