Distributed control protocol for high availability in multi-node storage cluster

• US 9,692,645 B2
• Filed: 04/03/2014
• Issued: 06/27/2017
• Est. Priority Date: 02/13/2014
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Banka, Deepti, Usgaonkar, Ameya Prakash
• Primary Examiner(s): Gillis, Brian J
• Assistant Examiner(s): Lobos, Boris Grijalva
• Application Number:
  US14/244,337
• Publication Number:
  US 20150227318A1
• Time in Patent Office: 1,181 Days
• CPC Class Codes: G06F 3/0617 in relation to availability
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0679 Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
• CPC Class Codes: G06F 3/0688 Non-volatile semiconductor memory arrays
• CPC Class Codes: H04L 41/08 Configuration management of networks or network elements address allocation H04L61/50
• CPC Class Codes: H04L 41/0816 the condition being an adaptation, e.g. in response to network events
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A distributed control protocol dynamically establishes high availability (HA) partner relationships for nodes in a cluster. A HA partner relationship may be established by copying (mirroring) information maintained in a non-volatile random access memory (NVRAM) of a node over a HA interconnect to the NVRAM of a partner node in the cluster. The distributed control protocol leverages a Cluster Liveliness and Availability Manager (CLAM) utility of a storage operating system executing on the nodes to rebalance NVRAM mirroring and alter HA partner relationships of the nodes in the cluster. The CLAM utility is configured to maintain various cluster related issues, such as CLAM quorum events, addition or subtraction of a node in the cluster and other changes in configuration of the cluster. Notably, the CLAM utility is an event based manager that implements the control protocol to keep the nodes informed of any cluster changes through event generation and propagation.

Claims

1. A method comprising:

receiving, by a first node computing device, a plurality of quorum change messages from a plurality of nodes via a cluster switching fabric, wherein the plurality of nodes comprises at least a second node of an existing cluster and a third node;

determining, by the first node computing device, when there is a quorum change event comprising a new node join event based on absence of one or more parameters from one of the quorum change messages received from the third node; and

establishing, by the first node computing device, one or more high availability (HA) partner relationships in a new cluster based on one or more other parameters from another one of the quorum change messages received from the second node in order to join the third node to the new cluster, when the determining indicates that the quorum change event comprises a new node join event.

2. The method of claim 1 further comprising mirroring, by the first node computing device, information maintained in a first non-volatile random access memory (NVRAM) to a second NVRAM of the second node to establish a HA partner relationship with balanced NVRAM mirroring, wherein the mirroring the information comprises copying the information from the first NVRAM over a HA interconnect to the second NVRAM.

3. The method of claim 1 wherein the establishing comprises establishing one of the HA partner relationships such that the second node mirrors information maintained in a second NVRAM of the second node to a third NVRAM of the third node.

4. The method of claim 1 wherein the establishing comprises establishing one of the HA partner relationships such that the third node mirrors information maintained in a third NVRAM of the third node to a second NVRAM of the second node.

5. The method of claim 1 wherein the establishing further comprises using the one or more other parameters to reconfigure the existing cluster to the new cluster.

6. The method of claim 5 wherein the using further comprises using the one or more other parameters to determine one or more new mirror partners and HA partners for the second node or the third node of the new cluster.

7. A node computing device, comprising:

one or more processors;

and

a memory coupled to the one or more processors and containing machine readable medium comprising machine executable code having stored thereon instructions for dynamically establishing high availability (HA) partner relationships, the one or more processors configured to execute the machine executable code to cause the one or more processors to;

receive a plurality of quorum change messages from a plurality of nodes via a cluster switching fabric, wherein the plurality of nodes comprises at least a first node of an existing cluster and a second node;

determine when there is a quorum change event comprising a new node join event based on an absence of one or more parameters from one of the quorum change messages received from the second node; and

establish one or more HA partner relationships in a new cluster based on one or more other parameters from another one of the quorum change messages received from the first node in order to join the second node to the new cluster, when the determining indicates that the quorum change event comprises a new node join event.

8. The computing device of claim 7, wherein the one or more processor are further configured to execute the machine executable code to cause the one or more processors to mirror information maintained in a first non-volatile random access memory (NVRAM) to a second NVRAM of the first node over a HA interconnect to the second NVRAM to establish a HA partner relationship with balanced NVRAM mirroring.

9. The computing device of claim 7, wherein the one or more processor are further configured to execute the machine executable code to cause the one or more processors to establish one of the HA partner relationships such that the first node mirrors information maintained in an NVRAM of the first node to another NVRAM of the second node.

10. The computing device of claim 7, wherein the one or more processor are further configured to execute the machine executable code to cause the one or more processors to establish one of the HA partner relationships such that the second node mirrors information maintained in an NVRAM of the second node to another NVRAM of the first node.

11. The computing device of claim 7, wherein the one or more processor are further configured to execute the machine executable code to cause the one or more processors to use the one or more other parameters to reconfigure the existing cluster to the new cluster.

12. The computing device of claim 11, wherein the one or more processor are further configured to execute the machine executable code to cause the one or more processors to use the one or more other parameters to determine one or more new mirror partners and HA partners for the first node or the second node of the new cluster.

13. A non-transitory machine readable medium having stored thereon instructions for dynamically establishing high availability (HA) partner relationships comprising machine executable code which when executed by at least one machine causes the machine to:

receive a plurality of quorum change messages from plurality of nodes via a cluster switching fabric, wherein the plurality of nodes comprises at least a first node of an existing cluster and a second node;

determine when there is a quorum change event comprising a new node join event based on an absence of one or more parameters from one of the quorum change messages received from the second node; and

establish one or more HA partner relationships in a new cluster based on one or more other parameters from another one of the quorum change messages received from the first node in order to join the second node to the new cluster, when the determining indicates that the quorum change event comprises a new node join event.

14. The non-transitory machine readable medium of claim 13, wherein the machine executable code, which when executed by the at least one machine, further causes the machine to mirror information maintained in a first non-volatile random access memory (NVRAM) to a second NVRAM of the first node over a HA interconnect to the second NVRAM to establish a HA partner relationship with balanced NVRAM mirroring.

15. The non-transitory machine readable medium of claim 13, wherein the machine executable code, which when executed by the at least one machine, further causes the machine to establish one of the HA partner relationships such that the first node mirrors information maintained in an NVRAM of the first node to another NVRAM of the second node.

16. The non-transitory machine readable medium of claim 13, wherein the machine executable code, which when executed by the at least one machine, further causes the machine to establish one of the HA partner relationships such that the second node mirrors information maintained in an NVRAM of the second node to another NVRAM of the first node.

17. The non-transitory machine readable medium of claim 13, wherein the machine executable code, which when executed by the at least one machine, further causes the machine to use the one or more other parameters to reconfigure the existing cluster to the new cluster.

18. The non-transitory machine readable medium of claim 17, wherein the machine executable code, which when executed by the at least one machine, further causes the machine to use the one or more other parameters to determine one or more new mirror partners and HA partners for the first node or the second node of the new cluster.