System and method for storage takeover

• US 7,613,947 B1
• Filed: 11/30/2006
• Issued: 11/03/2009
• Est. Priority Date: 11/30/2006
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Coatney, Susan M., Watanabe, Steven S., Strange, Stephen H., Patel, Kayuri
• Primary Examiner(s): Bonzo; Bryce P
• Application Number:
  US11/606,727
• Time in Patent Office: 1,069 Days
• Field of Search: 714 5- 8
• US Class Current: 714/6.1
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/2033 switching over of hardware resources
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence

Abstract

A takeover system and method for high availability in the face of component failures for a multiple-node storage system cluster is provided. When a takeover is triggered either through a loss of cluster heartbeat, or through a user-initiated command, the system of the present invention is configured such that one or more surviving nodes asserts ownership on one or more of the failed node'"'"'s disks such that all of the disks become locally owned by one of the surviving nodes. An assimilation of the disks into one or more aggregates is performed, and any duplicated file system identification numbers are corrected to avoid errors. As the disks thus appear to be locally owned by the surviving nodes, one or more newly assimilated aggregates are thus exposed to any N-module that interfaces with the surviving nodes.

Claims

1. A method for performing a takeover procedure in a storage system environment, the method comprising the steps of:

coupling two or more storage system nodes in a cluster, each storage system node configured to service one or more volumes of a disk array;

in response to a failure of one of said storage system nodes, notifying one or more surviving storage system nodes in the cluster to takeover a set of assigned disks previously associated with the failed storage system node;

at said surviving storage system nodes, asserting ownership of each disk previously associated with the failed storage system node that are assigned to that surviving storage system node;

assimilating each disk previously associated with the failed storage system node into one or more new data containers;

bringing online volumes associated with the new data container; and

processing at each surviving storage system node, at least one of data access requests for data volumes of said new data container, and data access requests for data originally associated with that surviving storage system node.

2. The method as defined in claim 1 comprising the further steps of assigning one or more ownership locations on each disk, and identifying in said ownership locations on each disk the current owner of that disk being the disk element that is currently responsible for data access requests directed to data on that disk.

3. The method as defined in claim 2 comprising the further steps of, upon a failure, at said surviving storage system node, writing said surviving storage system node as the current owner in the ownership location of each disk in said set of disks previously associated with the failed storage system node that are assigned to that storage system node.

4. The method as defined in claim 1 comprising the further steps of checking volume identifications for each volume in the new data container, and assigning new identification numbers to volumes that have duplicate identification numbers.

5. The method as defined in claim 1 comprising the further step of replaying a RAID section of a non-volatile memory of said RAID system.

6. The method as defined in claim 1 comprising the further step of replaying a file system section of a non-volatile memory log of the failed storage system node.

7. The method as defined in claim 1 comprising the further step of updating the volume location database of each surviving storage system node to reflect the new disk element of the new data container.

8. The method as defined in claim 1 comprising the further step of, after the other steps are performed, allowing a network element to send data access requests for volumes in one or more new data containers, comprising one or more aggregates.

9. The method as defined in claim 1 further comprising the step of detecting a failure by receiving a storage takeover command from an administrator that a particular surviving storage system node in the cluster is to take over one or more of the disks of another storage system node in the cluster.

10. The method as defined in claim 1 further comprising the step of detecting a failure by determining at said surviving storage system node, the absence of a heartbeat received from a failed storage system node.

11. The method as defined in claim 1 wherein the step of asserting ownership further comprises:

at said surviving storage system node, pre-empting SCSI reservations on one or more of the disks associated with the failed storage system node in the cluster, and assigning each surviving storage system node in the cluster a set of disks to take over, and each said surviving storage system node asserting its own SCSI reservations on each disk in said assigned disks.

12. The method as defined in claim 1 wherein all of the disks of said failed storage system node are assigned to a single surviving storage system node.

13. The method as defined in claim 1 wherein said step of asserting ownership further comprises at said surviving storage system node asserting ownership on a portion of a disk.

14. The method as defined in claim 13 wherein said step of asserting ownership on a portion of a disk includes using a fencing technique.

15. A system for a clustered storage environment takeover, the system comprising:

a cluster of interconnected storage system nodes that provide storage services for one or more clients, said storage system nodes each having a plurality of storage devices containing data on disks;

a cluster interconnect that provides a direct communication path between the storage system nodes in the cluster; and

a takeover monitor program executing within each storage system node that includes program instructions for detecting a failure in one or more other storage system nodes in the cluster and initiating a takeover procedure as a surviving storage system node as to one or more disks of the failed node to which it has been assigned for assimilating each assigned disk into one or more new data containers for processing incoming data access requests for the data that was originally serviced by the failed storage system node.

16. The system as defined in claim 15 further comprising a takeover monitor resource data structure that includes software program instructions for routines to be followed when a takeover is triggered.

17. The system as defined in claim 15 wherein said storage system nodes have an ownership software program that includes program instructions for reading and writing to an ownership location on a disk.

18. The system as defined in claim 17 wherein said takeover monitor program includes program instructions instructing said ownership module to change a current owner entry in said ownership location on each disk that it takes over as surviving storage system node to indicate said surviving storage system node as said current owner when a takeover is triggered.

19. The system as defined in claim 15 wherein said one or more new data containers comprise one or more new aggregates.

20. A computer readable medium for providing a takeover of data containers in a clustered storage environment, comprising program instructions for performing the steps of:

coupling two or more storage system nodes in a cluster with a cluster interconnect;

detecting a failure of a storage system node in the cluster being a failed storage system node;

notifying one or more a surviving storage system nodes in the cluster to takeover one or more volumes previously associated with the failed storage system node;

at each said surviving storage system node, asserting ownership over each disk previously associated with said failed storage system node that has been assigned to that surviving storage system node;

performing an assimilation on said disks of said failed storage system node to create a new data container;

bringing online volumes associated with each new data container; and

processing at said one or more surviving storage system nodes, data access requests for data in said volumes of said new data container, and volumes originally serviced by each said surviving storage system node.

21. The computer readable medium for providing for takeover of data containers in a clustered storage environment as defined in claim 20, comprising program instructions for performing the further step of:

upon a takeover, writing at said surviving storage system node, said surviving storage system node as the current owner in an ownership location of each disk associated with the failed storage system node that has been assigned to that storage system node.

22. The computer readable medium for providing for takeover of data containers in a clustered storage environment as defined in claim 20, comprising program instructions for performing the further step of:

checking file system identifications for each volume in the new data container, and assigning new file system identification numbers to volumes that have duplicate identification numbers.

23. The computer readable medium for providing for takeover of data containers in a clustered storage environment as defined in claim 20, comprising program instructions for performing the further step of:

at said surviving storage system node, pre-empting SCSI reservations on one or more disks associated with the failed storage system node in the cluster assigned to that surviving storage system node, and said surviving storage system node asserting its own SCSI reservations said assigned disks.

24. The computer readable medium for providing for takeover of data containers in a clustered storage environment as defined in claim 20, comprising program instructions for performing the further step of:

at said surviving storage system node, asserting ownership over a portion of one or more disks associated with the failed storage system node in the cluster assigned to that surviving storage system node.

25. The computer readable medium for providing for takeover of data containers in a clustered storage environment as defined in claim 24, comprising program instructions for performing the further step of:

asserting ownership over said portion of one or more disks using a fencing technique.

26. The computer readable medium for providing for takeover of data containers in a clustered storage environment as defined in claim 20 wherein all of the disks of the failed storage system node are assigned to a single surviving storage system node.

27. The computer readable medium as defined in claim 20 comprising further program instructions for performing the step of performing said assimilation on said disks by using a RAID system.