DISTRIBUTED CONTROL PROTOCOL FOR HIGH AVAILABILITY IN MULTI-NODE STORAGE CLUSTER

• US 20150227318A1
• Filed: 04/03/2014
• Published: 08/13/2015
• Est. Priority Date: 02/13/2014
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Usgaonkar, Ameya Prakash, Banka, Deepti
• Granted Patent: US 9,692,645 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0617 in relation to availability
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0679 Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
• CPC Class Codes: G06F 3/0688 Non-volatile semiconductor memory arrays
• CPC Class Codes: H04L 41/08 Configuration management of networks or network elements address allocation H04L61/50
• CPC Class Codes: H04L 41/0816 the condition being an adaptation, e.g. in response to network events
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A distributed control protocol dynamically establishes high availability (HA) partner relationships for nodes in a cluster. A HA partner relationship may be established by copying (mirroring) information maintained in a non-volatile random access memory (NVRAM) of a node over a HA interconnect to the NVRAM of a partner node in the cluster. The distributed control protocol leverages a Cluster Liveliness and Availability Manager (CLAM) utility of a storage operating system executing on the nodes to rebalance NVRAM mirroring and alter HA partner relationships of the nodes in the cluster. The CLAM utility is configured to maintain various cluster related issues, such as CLAM quorum events, addition or subtraction of a node in the cluster and other changes in configuration of the cluster. Notably, the CLAM utility is an event based manager that implements the control protocol to keep the nodes informed of any cluster changes through event generation and propagation.

Claims

1. A method comprising:

minoring information maintained in a first non-volatile random access memory (NVRAM) of a first node to a second NVRAM of a second node in an existing cluster to establish a high availability (HA) partner relationship with balanced NVRAM mirroring between the first and second nodes;

generating a message at a third node to request joining of the third node to the existing cluster;

broadcasting the message over a cluster switching fabric coupled to the nodes, the message broadcasted as a first message of a distributed control protocol;

in response to receiving the first broadcasted message, broadcasting a second message from each node over the cluster switching fabric of the cluster, wherein the second message includes parameters about the node; and

in response to receiving the second broadcasted message, implementing a daisy-chain processing technique at a master node of the cluster to rebalance the NVRAM minoring and alter the HA partner relationship among the nodes of a new cluster.

2. The method of claim 1 wherein mirroring the information comprises copying the information from the first NVRAM over a HA interconnect to the second NVRAM.

3. The method of claim 1 wherein implementing the daisy-chain processing technique comprises rebalancing of the NVRAM minoring in a forward daisy-chain fashion.

4. The method of claim 1 wherein implementing the daisy-chain processing technique comprises altering of the HA partner relationship in a reverse daisy-chain fashion.

5. The method of claim 1 wherein implementing the daisy-chain processing technique comprises using the parameters of the second broadcasted message to reconfigure the existing cluster to the new cluster.

6. The method of claim 5 wherein using the parameters comprises using the parameters to determine new mirror partners and HA partners for the nodes of the new cluster.

7. The method of claim 6 wherein implementing the daisy-chain processing technique further comprises comparing contents of a data structure maintained by the master node with the parameters to determine a type of event represented by the broadcasted messages.

8. A system comprising:

one or more processors of a first node interconnected by a cluster switching fabric to a plurality of nodes, wherein the first node is further coupled to one or more of the nodes by a high availability (HA) interconnect;

a first non-volatile random access memory (NVRAM) connected to the one or more processors, the first NVRAM configured to maintain information associated with the first node; and

a memory coupled to the one or more processors and configured to store a storage operating system implemented as one or processes, the one or more processes executable by the one or more processors to;

minor the information maintained in the first NVRAM to a second NVRAM of a second node in an existing cluster to establish a HA partner relationship with balanced NVRAM mirroring between the first and second nodes;

receive a first message of a distributed control protocol over the cluster switching fabric, the first message configured to request joining of a third node to the existing cluster;

receive a second message from each node of the existing cluster over the cluster switching fabric, wherein the second message includes first parameters about the node; and

implement a daisy-chain processing technique using the first parameters to rebalance the NVRAM mirroring and alter the HA partner relationship among the nodes of a new cluster.

9. The system of claim 8 wherein mirror of the information is achieved by copying the information from the first NVRAM over the HA interconnect to the second NVRAM.

10. The system of claim 8 wherein the third node includes a third NVRAM and wherein each node includes local storage configured to store an identification (ID) of a HA partner and an ID of a mirror partner.

11. The system of claim 10 wherein rebalance of the NVRAM mirroring is achieved by forward daisy-chain processing such that the local storage of the first node identifies the second node as the mirror partner of the first node, the local storage of the second node identifies the third node as the mirror partner of the second node, and the local storage of the third node identifies the first node as the mirror partner for the third node.

12. The system of claim 11 wherein alter of the HA partner relationship is achieved by reverse daisy-chain processing such that the local storage of the first node identifies the third node as the HA partner of the first node, the local storage of the second node identifies the first node as the HA partner of the second node, and the local storage of the third node identifies the second node as the HA partner for the third node.

13. The system of claim 12 wherein the one or more processes are further executable by the one or more processors to:

encapsulate second parameters as determined by the forward and reverse daisy-chain processing into a third message; and

propagate the third message to the nodes of the new cluster, such that each node interprets the second parameters and makes necessary changes to the minor partner ID and the HA partner ID in the local storage.

14. The system of claim 8 wherein the one or more processes comprises a Cluster Liveliness and Availability Manager (CLAM) utility of the storage operating system configured to maintain changes in configuration of the cluster, including events of a CLAM quorum.

15. The system of claim 14 wherein the distributed control protocol is a CLAM protocol and wherein the CLAM utility implements the CLAM protocol to keep all of the nodes informed of any cluster changes through event generation and propagation.

16. The system of claim 15 wherein an event is generated by the CLAM utility in response to a change in an eligibility criterion of a node for the CLAM quorum.

17. The system of claim 16 wherein the joining of the third node to the existing cluster comprises a change in the eligibility criterion of the node participating in the quorum.

18. The system of claim 17 wherein the joining of the third node to the existing cluster satisfies the CLAM protocol and constitutes the event that triggers commencement of the CLAM protocol.

19. The system of claim 18 wherein the event is a quorum change event.

20. A non-transitory computer readable medium including program instructions for execution on one or more processors, the program instructions when executed operable to:

minor information maintained in a first non-volatile random access memory (NVRAM) of a first node to a second NVRAM of a second node in an existing cluster to establish a high availability (HA) partner relationship with balanced NVRAM mirroring between the first and second nodes;

generate a message at a third node to request joining of the third node to the existing cluster;

broadcast the message over a cluster switching fabric coupled to the nodes, the message broadcasted as a first message of a distributed control protocol;

in response to receiving the first broadcasted message, broadcast a second message from each node over the cluster switching fabric of the cluster, wherein the second message includes parameters about the node; and

in response to receiving the second broadcasted message, implement a daisy-chain processing technique at a master node of the cluster to rebalance the NVRAM minoring and alter the HA partner relationship among the nodes of a new cluster.