THIRD VOTE CONSENSUS IN A CLUSTER USING SHARED STORAGE DEVICES

• US 20170116095A1
• Filed: 10/27/2015
• Published: 04/27/2017
• Est. Priority Date: 10/27/2015
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Schatz, Bob, Muth, John, Strange, Stephen H.
• Granted Patent: US 9,836,366 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 11/1425 by reconfiguration of node membership
• CPC Class Codes: G06F 11/2033 switching over of hardware resources
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence
• CPC Class Codes: G06F 2201/805 Real-time

Abstract

A third vote consensus technique enables a first node, i.e., a surviving node, of a two-node cluster to establish a quorum and continue to operate in response to failure of a second node of the cluster. Each node maintains configuration information organized as a cluster database (CDB) which may be changed according to a consensus-based protocol. Changes to the CDB are logged on a third copy file system (TCFS) stored on a local copy of TCFS (L-TCFS). A shared copy of the TCFS (i.e., S-TCFS) may be stored on shared storage devices of one or more storage arrays coupled to the nodes. The local copy of the TCFS (i.e., L-TCFS) represents a quorum vote for each node of the cluster, while the S-TCFS represents an additional “tie-breaker” vote of a consensus-based protocol. The additional vote may be obtained from the shared storage devices by the surviving node as a third vote to establish the quorum and enable the surviving node to cast two of three votes (i.e., a majority of votes) needed to continue operation of the cluster. That is, the majority of votes allows the surviving node to update the CDB with the configuration information changes so as to continue proper operation of the cluster.

Claims

1. A method comprising:

receiving a write request directed towards a logical unit (LUN), the write request having data and received at a first node of a cluster, the first node and a second node of the cluster connected to a storage array, the first node connected to a local storage device;

maintaining configuration information of the cluster on the storage array;

maintaining a copy of the configuration information on the local storage device;

in response to the second node failing, obtaining control to update the configuration information on the storage array at the first node using a consensus protocol;

establishing a first vote of the consensus protocol by ownership of the copy of the configuration information on the local storage device; and

establishing a second vote of the consensus protocol by ownership of the configuration information on the storage array.

2. The method of claim 1 further comprising accessing the configuration information on the storage array at the first and second nodes via non-exclusive small computer systems interface (SCSI) reservations.

3. The method of claim 1 further comprising logging changes to the configuration information on the local storage device.

4. The method of claim 1 further comprising establishing a majority vote of the consensus protocol and servicing the write request at the first node.

5. The method of claim 1 further comprising updating the configuration information on the storage array in order of update events from the nodes of the cluster.

6. The method of claim 1 wherein the consensus protocol is Raft.

7. The method of claim 1 further comprising detecting failure of the second note by failure to receive a heartbeat from the second node.

8. The method of claim 1 wherein further comprising failing to establish a quorum using the consensus protocol absent a tie-breaker vote among a number of surviving nodes.

9. The method of claim 8 wherein the second vote is the tie-breaker vote.

10. A method comprising:

receiving a write request directed towards a logical unit (LUN), the write request having data and received at a first node of a cluster, the first node and a second node of the cluster connected to a storage array;

maintaining a local copy of a configuration database of the cluster at each node of the cluster;

maintaining a shared copy of the configuration database on the storage array;

in response to the first node failing, obtaining control to update the shared configuration database at the second node using a consensus protocol;

obtaining exclusive access to the storage array at the second node;

establishing a first vote of the consensus protocol by ownership of the local copy of the configuration database at the second node;

establishing a second vote of the consensus protocol by ownership of the shared copy configuration database at the second node;

updating the database at the second node to reflect the failure of the first node; and

servicing the write request at the second node.

11. A system comprising:

a cluster having first and second nodes, each node having a memory connected to a processor via a bus;

a storage array coupled to each node of the cluster;

a local storage device coupled to each node of the cluster;

a storage I/O stack executing on the processor of each node of the cluster, the storage I/O stack when executed operable to;

receive a write request directed towards a logical unit (LUN), the write request having data and received at the first node of a cluster;

maintain configuration information of the cluster on the storage array;

maintain a copy of the configuration information on the local storage device;

in response to the second node failing, obtain control to update the configuration information on the storage array at the first node using a consensus protocol;

establish a first vote of the consensus protocol by ownership of the copy of the configuration information on the local storage device; and

establish a second vote of the consensus protocol by ownership of the configuration information on the storage array.

12. The system of claim 11 wherein the storage I/O stack is further operable to:

access the configuration information on the storage array at the first and second nodes via non-exclusive small computer systems interface (SCSI) reservations.

13. The system of claim 11 wherein the storage I/O stack is further operable to:

log changes to the configuration information on the local storage device.

14. The system of claim 11 wherein the storage I/O stack is further operable to:

establish a majority vote of the consensus protocol and service the write request at the first node.

15. The system of claim 11 wherein the storage I/O stack is further operable to:

update the configuration information on the storage array in order of update events from the nodes of the cluster

16. The system of claim 11 wherein the consensus protocol is Raft.

17. The system of claim 11 wherein the storage I/O stack is further operable to:

detect failure of the second note by failure to receive a heartbeat from the second node.

18. The system of claim 11 wherein the storage I/O stack

fails to establish a quorum using the consensus protocol absent a tie-breaker vote among a number of surviving nodes.

19. The system of claim 18 wherein the second vote is the tie-breaker vote.

20. The system of claim 11 wherein the cluster further comprises:

a plurality of high-availability (HA) groups of nodes, wherein a first HA group includes the first and second nodes.