High availability failover manager

• US 10,360,120 B2
• Filed: 08/25/2017
• Issued: 07/23/2019
• Est. Priority Date: 09/24/2015
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Watanabe, Steven S., Strange, Stephen H., Muth, John, Malone, Kimberly A., Patel, Kayuri H.
• Primary Examiner(s): Iqbal, Nadeem
• Application Number:
  US15/687,062
• Publication Number:
  US 20170351589A1
• Time in Patent Office: 697 Days
• CPC Class Codes: G06F 11/1441 Resetting or repowering
• CPC Class Codes: G06F 11/2033 switching over of hardware resources
• CPC Class Codes: G06F 11/2035 without idle spare hardware
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 2201/805 Real-time

Abstract

A high availability (HA) failover manager maintains data availability of one or more input/output (I/O) resources in a cluster by ensuring that each I/O resource is available (e.g., mounted) on a hosting node of the cluster and that each I/O resource may be available on one or more partner nodes of the cluster if a node (i.e., a local node) were to fail. The HA failover manager (HA manager) processes inputs from various sources of the cluster to determine whether failover is enabled for a local node and each partner node in an HA group, and for triggering failover of the I/O resources to the partner node as necessary. For each I/O resource, the HA manager may track state information including (i) a state of the I/O resource (e.g., mounted or un-mounted); (ii) the partner node(s) ability to service the I/O resource; and (iii) whether a non-volatile log recording I/O requests is synchronized to the partner node(s). The HA manager interacts with various layers of a storage I/O stack to mount and un-mount the I/O resources on one or more nodes of the cluster through the use of well-defined interfaces, e.g., application programming interfaces.

Claims

1. A system comprising:

a plurality of nodes interconnected as a high availability (HA) group of a cluster, each node including a central processing unit (CPU) coupled to a storage array, a service storage device, a non-volatile random access memory (NVRAM) and a memory;

the service storage device configured to store cluster-related configuration information as a cluster database (CDB);

the NVRAM configured to maintain status information of data access requests serviced by the node, the status information indicating whether a corresponding data access request is persistently stored on the storage array, the status information stored in one or more non-volatile logs of the NVRAM, the status information mirrored over a HA interconnect connecting a local node of the HA group to a partner node of the HA group to synchronize the status information between the nodes; and

the memory configured to store a high availability (HA) failover manager executable by the CPU, the HA failover manager configured to maintain data availability of input/output (I/O) resources in the cluster by ensuring that each I/O resource is mounted on at least one of the local node and the partner node.

2. The system of claim 1 wherein each I/O resource includes one of a volume and extent store of a storage I/O stack stored in the memory and executable by the CPU.

3. The system of claim 1 wherein the HA failover manager is embodied as a state machine configured to process inputs from the cluster to (i) determine whether failover is enabled for the local node and the partner node of the HA group and (ii) trigger the failover of the I/O resources from the local node to the partner node.

4. The system of claim 3 wherein the inputs include a cluster node state via the CDB.

5. The system of claim 4 wherein the cluster node state comprises one of booting, shutdown and failure.

6. The system of claim 3 wherein the inputs include a state of storage connectivity to the storage array.

7. The system of claim 3 wherein the inputs include a state of NVRAM mirroring via a NVRAM mirroring module executable by the CPU.

8. The system of claim 7 wherein the HA failover manager interfaces with the NVRAM mirroring module to establish and implement policies including whether the mirrored status information is synchronized between the nodes.

9. The system of claim 1 further comprising an I/O resource table of the CDB configured to store state information and provide semantics for ordering and consistency of transactions across the nodes of the cluster.

10. The system of claim 9 wherein the state information includes (i) a state of each I/O resource, (ii) an ability of the partner node to mount each I/O resource, and (iii) synchronization of the NVRAM on the local and partner nodes.

11. The system of claim 9 wherein the HA failover manager is a cluster-wide component that includes a plurality of HA manager instances, and wherein each HA manger instance runs on a node of the cluster.

12. The system of claim 11 wherein in response to a first HA manager instance changing the state information of the I/O resource table, a second HA manager instance reacts to the changed state information to enable communication among the HA manager instances.

13. The system of claim 11 further comprising a CDB manager of the CDB configured to send CDB notifications to the plurality of HA manager instances in response to changes to the state information of the I/O resource table.

14. A method comprising:

interconnecting a plurality of nodes as a high availability (HA) group of a cluster, wherein each node includes a central processing unit (CPU) coupled to a storage array, a service storage device, a non-volatile random access memory (NVRAM) and a memory;

storing cluster-related configuration information as a cluster database (CDB) on the service storage device;

maintaining status information of data access requests serviced by the node on one or more non-volatile logs of the NVRAM, the status information indicating whether the corresponding data access request is persistently stored on the storage array;

mirroring the status information over a HA interconnect connecting a local node of the HA group to a partner node of the HA group to synchronize the status information between the nodes; and

implementing a high availability (HA) failover manager to maintain data availability of input/output (I/O) resources in the cluster and ensure that each I/O resource is mounted on at least one of the local node and the partner node.

15. The method of claim 14 wherein each I/O resource includes one of a volume and extent store of a storage I/O stack executable by the CPU.

16. The method of claim 14 wherein implementing the HA failover manager further comprises:

embodying the HA failover manager as a state machine configured to process inputs from the cluster to (i) determine whether failover is enabled for the local node and the partner node of the HA group and (ii) trigger the failover of the I/O resources from the local node to the partner node.

17. The method of claim 16 wherein the inputs include a cluster node state via the CDB.

18. The method of claim 16 wherein the inputs include a state of storage connectivity to the storage array.

19. The method of claim 16 wherein the inputs include a state of NVRAM mirroring via a NVRAM mirroring module.

20. A non-transitory computer readable media containing instructions for execution on a processor for a method comprising:

interconnecting a plurality of nodes as a high availability (HA) group of a cluster, wherein each node includes a central processing unit (CPU) coupled to a storage array, a service storage device, a non-volatile random access memory (NVRAM) and a memory;

storing cluster-related configuration information as a cluster database (CDB) on the service storage device;

maintaining status information of data access requests serviced by the node on one or more non-volatile logs of the NVRAM, the status information indicating whether the corresponding data access request is persistently stored on the storage array;

mirroring the status information over a HA interconnect connecting a local node of the HA group to a partner node of the HA group to synchronize the status information between the nodes; and

implementing a high availability (HA) failover manager to maintain data availability of input/output (I/O) resources in the cluster and ensure that each I/O resource is mounted on at least one of the local node and the partner node.