Mirror split brain avoidance

• US 7,111,194 B1
• Filed: 03/21/2003
• Issued: 09/19/2006
• Est. Priority Date: 03/21/2003
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: Network Appliance Incorporated (NetApp, Inc.)
• Inventors: Coatney, Susan M., Schoenthal, Scott, sen Sarma, Joydeep, Rodrigues, Steven H., Rowe, Alan L.
• Primary Examiner(s): PORTKA, GARY J
• Application Number:
  US10/394,854
• Time in Patent Office: 1,278 Days
• Field of Search: None
• US Class Current: 714/6.32
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

A technique includes a set of procedures utilized during the transfer of service of a failed storage system to a surviving storage system of a cluster failover (CFO) configuration during a system outage, such as a disaster scenario. The procedures are executed by a RAID subsystem of the surviving storage system during disaster takeover, storage healing and node restart operations. The procedures prevent the possibility that clients accessing the storage system will see out-of-date versions of the data either during transfer of service of the system during the disaster scenario or subsequently after the storage system is healed and service on the failed system has been restored.

Claims

1. A method for transferring service of a failed storage system to a surviving storage system of a cluster configuration during a system outage, the method comprising the steps of:

splitting data sets of the failed storage system to allow comparison of the data sets;

marking a failed data set out-of-date;

selecting a correct volume from the surviving data set of a surviving storage system;

assigning the surviving data set an ordinal

indicating, by a time stamp associated with the ordinal, a point in time when the volume claims the ordinal;

giving precedence to a volume with the most recent claim to the ordinal over volumes with less recent claims to the ordinal; and

assigning other ordinals to the any remaining volumes in a monotonically increasing order according to precedence.

2. The method as in claim 1, further comprising the step of:

selecting a root volume for the correct volume.

3. The method as in claim 2, further comprising the step of:

containing, within the root volume, configuration information that defines the configuration of any remaining volumes, their data services, shared cluster management services and any underlying hardware platform elements.

4. The method as in claim 1, further comprising the step of:

assigning a base ordinal as the ordinal assigned to the surviving data set.

5. The method as in claim 4, further comprising the step of:

assigning the base ordinal a value of zero.

6. The method as in claim 4, further comprising the steps of:

giving a base name of the volume to the volume assigned the base ordinal; and

using the assigned other ordinals together with the base name for each of the remaining volumes.

7. The method as in claim 1 wherein the assignment of ordinals occurs on a per-system basis.

8. The method as in claim 1, further comprising the step of:

updating the other ordinals to the current system time stamp each time a label is updated for a volume with a base ordinal setting.

9. The method as in claim 1, further comprising the step of:

retaining the state associated with individual volumes that existed prior to the system outage, the state including volume name, snapshot information associated with in-progress asynchronous mirroring data transfers, and online/offline/restricted state.

10. The method as in claim 1, further comprising the step of:

invalidating client handles to the volume that are currently in use, forcing all clients to remount.

11. The method as in claim 10, further comprising the step of:

disabling the capability of invalidating client handles in environments that do not require strict data integrity.

12. The method as in claim 1, further comprising:

utilizing a volume name conflict resolution procedure to select the correct volume, the conflict resolution procedure ensuring that a reappearing data set is uniquely renamed in a manner that is consistent with the previous volume name.

13. A storage server comprising:

means for splitting data sets of a failed system to allow comparison of the data sets;

means for marking a failed data set out-of-date;

means for selecting a correct volume from a surviving data set;

means for assigning the surviving data set an ordinal;

means for indicating a point in time when the volume claims the ordinal;

means for giving precedence to a volume with the most recent claim to the ordinal over volumes with less recent claims to the ordinal; and

means for assigning other ordinals to the any remaining volumes in a monotonically increasing order according to precedence.

14. A computer readable media for transferring service of a failed storage system to a surviving storage system of a cluster configuration during a system outage, said computer readable media containing instructions for causing a processor to execute a method comprising:

splitting data sets of the failed system to allow comparison of the data sets;

marking a failed data set out-of-date;

selecting a correct volume from a surviving data set;

assigning the surviving data set an ordinal;

indicating, by a time stamp associated with the ordinal, a point in time when the volume claims the ordinal.

giving precedence to a volume with the most recent claim to the ordinal over volumes with less recent claims to the ordinal; and

assigning other ordinals to the any remaining volumes in a monotonically increasing order according to precedence.

15. Electromagnetic signals propagating on a computer network for transferring service of a failed storage system to a surviving storage system of a cluster configuration during a system outage, said electromagnetic signals carrying instructions for causing a processor to execute a method comprising:

splitting data sets of the failed system to allow comparison of the data sets;

marking a failed data set out-of-date;

selecting a correct volume from the surviving data set; and

assigning the surviving data set an ordinal;

indicating, by a time stamp associated with the ordinal, a point in time when the volume claims the ordinal;

giving precedence to a volume with the most recent claim to the ordinal over volumes with less recent claims to the ordinal; and

assigning other ordinals to the any remaining volumes in a monotonically increasing order according to precedence.

16. A method for transferring service of a failed storage system to a surviving storage system during a system outage, the method comprising the steps of:

splitting an accessible data set from its original volume to thereby form a new volume;

assigning the new volume a base ordinal and a time stamp indicating a point in time when the volume claims the base ordinal; and

giving preference to a volume with a more recent claim to the base ordinal over other volumes with less recent claims to the base ordinal to resolve conflicts among volumes.

17. The method of claim 16 further including the step of:

bringing the new volume online in a file system.

18. The method of claim 16 further including the step of:

marking a failed data set as out-of-date.

19. The method of claim 16 wherein the time stamp indicating a point in time when the volume claims the base ordinal is identical to the time stamp associated with the original volume.

20. The method of claim 16 further comprising the step of:

selecting a root volume as the new volume.

21. The method of claim 20 further comprising the step of:

providing within the root volume configuration information that defines the configuration of other surviving volumes, services, and shared cluster management services.

22. The method of claim 16 further comprising the step of:

assigning the base ordinal a value of zero.

23. The method of claim 16 further comprising the step of:

assigning ordinals to one or more other volumes in a monotonically increasing order according to precedence.

24. The method of claim 16 further comprising the step of:

assigning a base name to the volume assigned the base ordinal; and

assigning the base name and a different ordinal to one or more other volumes.

25. The method of claim 16 further comprising the step of:

invalidating one more client handles to the original volume that are currently in use, forcing all clients to remount.

26. The method of claim 16 further comprising the step of:

using a volume name conflict resolution procedure to select the correct volume, the conflict resolution procedure ensuring that a reappearing data set is uniquely renamed.

27. A storage server comprising:

means for splitting an accessible data set from its original volume to thereby form a new volume;

means for assigning the new volume a base ordinal and a time stamp indicating a point in time when the volume claims the base ordinal; and

means for giving preference to a volume with a more recent claim to the base ordinal over other volumes with less recent claims to the base ordinal to resolve conflicts among volumes.

28. A computer readable media, including program instructions executing on a processor for transferring service of a failed storage system to a surviving storage system during a system outage, the program instructions including instructions for performing the steps of:

splitting an accessible data set from its original volume to thereby form a new volume;

assigning the new volume a base ordinal and a time stamp indicating a point in time when the volume claims the base ordinal; and

giving preference to a volume with a more recent claim to the base ordinal over other volumes with less recent claims to the base ordinal to resolve conflicts among volumes.