System and method for splitting a cluster for disaster recovery

• US 7,496,782 B1
• Filed: 06/01/2004
• Issued: 02/24/2009
• Est. Priority Date: 06/01/2004
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: Network Appliance Incorporated (NetApp, Inc.)
• Inventors: Kownacki, Ronald William
• Primary Examiner(s): Beausoliel; Robert
• Assistant Examiner(s): Manoskey; Joseph D
• Application Number:
  US10/858,150
• Time in Patent Office: 1,729 Days
• Field of Search: 714/4, 714/6, 714/11, 714/12, 714/13, 714/27, 714/16, 714/43, 714/20, 714/56, 709/220, 709/221
• US Class Current: 714/4.4
• CPC Class Codes: G06F 11/1425 by reconfiguration of node membership
• CPC Class Codes: G06F 11/1479 Generic software techniques for error detection or fault masking
• CPC Class Codes: G06F 11/202 where processing functionality is redundant redundant communication control functionality G06F11/2005, redundant storage control functionality G06F11/2089
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

The present invention provides a system and method for disaster recovery split of a node from a cluster to enable cluster management operations using quorum-based data replication services to continue. A split command is executed on a selected node and a new site list data structure describing the cluster is generated. The site list data structure marks all nodes other than the selected node as ineligible, thereby placing the selected node in quorum.

Claims

1. A method for operating a cluster of servers, comprising:

detecting that an error condition has occurred in a plurality of servers, the plurality of servers making up the cluster, the error condition initiated in response to the clusters inability to meet a conventional quorum requirement and preventing any write operations from occurring on any one server of the plurality of servers and wherein the conventional quorum state requires a majority of the nodes to be healthy to have quorum;

determining that a selected server of the plurality of servers is functioning correctly;

executing a split command by a user to override the quorum requirement by designating the selected server as a full read/write replica of the cluster and forming a cluster of one server from the selected server, the cluster now having a stand alone server and a cluster configuration of one to one thereby allowing the server to be modified without use of a voting system; and

assigning a workload of the cluster to the cluster of one server.

2. The method of claim 1, further comprising:

determining that another server of the plurality of servers is functioning correctly, and

executing an unsplit command to bring the another server into the cluster of one server to form a cluster of two servers.

3. The method of claim 2, further comprising:

detecting further servers which are functioning correctly, and bringing each into the cluster of two to form a new cluster of a plurality of servers.

4. The method of claim 1, further comprising:

executing the split command by a person, the split command executed in a command line interface of a management framework process executing on the selected server.

5. The method of claim 1, further comprising:

executing the split command by a person, the split command executed via a graphical user interface of a management framework process executing on the selected server.

6. The method of claim 1, further comprising:

creating a modified site list data structure for the cluster of one server.

7. The method of claim 1, further comprising:

creating a modified site list data structure for the cluster of one server by assigning an epsilon value to the selected server.

8. The method of claim 1, further comprising:

executing an unsplit command to reintroduce an additional server to the cluster.

9. The method of claim 1, further comprising:

introducing an additional server to the cluster by;

selecting a server to be added;

stopping processes using a replicated database directory on the server to be added;

removing a replicated database directory from the server to be added;

marking the server to be added as eligible in the modified site list data structure;

restarting the process using the replicated database directory on the server to be added; and

replicating the replicated database directory to the server to be added from another server in the cluster.

10. The method of claim 1, further comprising:

creating a site list data structure having a version field, a cluster identification field, an epsilon field, and one or more entries, each of the one or more entries associated with a server in the cluster.

11. The method of claim 10, further comprising:

creating as entries a site identifier field and a state field.

12. The method of claim 1, wherein each node is healthy when the node is active and responding to one or more client requests.

14. The cluster of servers of claim 11, further comprising:

another server of the plurality of servers is determined to be functioning correctly, and

an unsplit command executed to bring the another server into the cluster of one server to form a cluster of two servers.

13. A cluster of servers, comprising:

a plurality of servers that develops an error condition, the plurality of servers making up the cluster, the error condition is detected, the error condition initiated in response to the clusters inability to meet a conventional quorum requirement and preventing any write operations from occurring on any one server of the plurality of servers and wherein the conventional quorum state requires a majority of the nodes to be healthy to have quorum;

a server which is determined to be functioning correctly is selected, hereinafter the selected server;

a split command that is executed by a user to override the quorum requirement by designating the selected server as a full read/write replica of the cluster and forming a cluster of one server from the selected server, the cluster now having a stand alone server and a cluster configuration of one to one thereby allowing the server to be modified without use of a voting system; and

a workload of the cluster assigned to the cluster of one server.

15. The cluster of servers of claim 13, further comprising:

further servers which are functioning correctly, and each functioning further server is brought into the cluster of two to form a new cluster of a plurality of servers.

16. The cluster of servers of claim 13, further comprising:

the split command is executed by a person, the split command executed in a command line interface of a management framework process executing on the selected server.

17. The cluster of servers of claim 13, further comprising:

the split command executed by a person, the split command executed via a graphical user interface of a management framework process executing on the selected server.

18. The cluster of servers of claim 13, further comprising:

a modified site list data structure is created for the cluster of one server.

19. The cluster of servers of claim 13, further comprising:

a modified site list data structure for the cluster of one server has an epsilon value assigned to the selected server.

20. The cluster of servers of claim 13, further comprising:

an unsplit command executed to reintroduce an additional server to the cluster.

21. The cluster of servers of claim 13, further comprising:

an additional server is introduced to the cluster by;

selecting a server to be added;

stopping processes using a replicated database directory on the server to be added;

removing the replicated database directory from the server to be added;

marking the server to be added as eligible in the modified site list data structure to be added;

restarting the process using the replicated database directory on the server to be added; and

replicating the replicated database directory to the server to be added from another server in the cluster.

22. The cluster of servers of claim 13, further comprising:

a site list data structure is created, the site list data structure having a version field, a cluster identification field, an epsilon field, and one or more entries, each of the one or more entries associated with a server in the cluster.

23. The cluster of servers of claim 22, further comprising:

entries of the site list data structure including a site identifier field and a state field.

24. The cluster of servers of claim 13, wherein each node is healthy when the node is active and responding to one or more client requests.

25. A computer readable media, comprising:

said computer readable media containing instructions for execution on a processor for a method of operating a cluster of servers, the method having the steps of,

detecting that an error condition has occurred in a plurality of servers, the plurality of servers making up the cluster, the error condition initiated in response to the clusters inability to meet a conventional quorum requirement and preventing any write operations from occurring on any one server of the plurality of servers and wherein the conventional quorum state requires a majority of the nodes to be healthy to have quorum;

determining that a selected server of the plurality of servers is functioning correctly;

executing a split command to override the quorum requirement by designating the selected server as a full read/write replica of the cluster and forming, a cluster of one server from the selected server computer, the cluster now having a stand alone server and a cluster configuration of one to one thereby allowing the server to be modified without use of a voting system; and

assigning a workload of the cluster to the cluster of one server.

26. A method for operating a cluster of servers, comprising:

detecting that an error condition has occurred in a plurality of servers, the plurality of servers making up the cluster;

determining that a failure of the cluster resulted in the impossibility of a conventional quorum being achieved preventing a write operation from occurring on any one server of the plurality of servers and wherein the conventional quorum state requires a majority of the nodes to be healthy to have quorum;

executing a split command by a user to override the quorum requirement by designating a selected server as a full read/write replica of the cluster and forming a cluster of one server from the selected server, the cluster now having a stand alone server and a cluster configuration of one to one thereby allowing the server to be modified without use of a voting system; and

assigning a workload of the cluster to the cluster of one server.

27. The method of claim 26, further comprising:

determining that another server of the plurality of servers is functioning correctly, and

executing an unsplit command to bring the another server into the cluster of one server to form a cluster of two servers.

28. The method of claim 27, further comprising:

detecting further servers which are functioning correctly, and bringing each into the cluster of two to form a new cluster of a plurality of servers.

29. The method of claim 26, further comprising:

executing the split command by a user, the split command executed in a command line interface of a management framework process executing on the selected server.

30. The method of claim 26, further comprising:

executing the split command by a person, the split command executed via a graphical user interface of a management framework process executing on the selected server.

31. The method of claim 26, further comprising:

creating a modified site list data structure for the cluster of one server.

32. The method of claim 26, further comprising:

creating a modified site list data structure for the cluster of one server by assigning an epsilon value to the selected server.

33. The method of claim 26, further comprising:

executing an unsplit command to reintroduce an additional server to the cluster.

34. The method of claim 26, further comprising:

introducing an additional server to the cluster by;

selecting a server to be added;

stopping processes using a replicated database directory on the server to be added;

removing a replicated database directory from the server to be added;

marking the server to be added as eligible in the modified site list data structure;

restarting the process using the replicated database directory on the server to be added; and

replicating the replicated database directory to the server to be added from another server in the cluster.

35. The method of claim 26, further comprising:

creating a site list data structure having a version field, a cluster identification field, an epsilon field, and one or more entries, each of the one or more entries associated with a server in the cluster.

36. The method of claim 35, further comprising:

creating as entries a site identifier field and a state field.

37. A cluster of servers, comprising:

a plurality of servers that develops an error condition, the plurality of servers making up the cluster;

a server which determined that a failure of the cluster resulted in the impossibility of a conventional quorum being achieved preventing a write operation from occurring on any one server of the plurality of servers and wherein the conventional quorum state requires a majority of the nodes to be healthy to have quorum;

a split command by a user to override the quorum requirement by that is executed to designate a selected server as a full read/write replica of the cluster and forming a cluster of one server from the selected server, the cluster now having a stand alone server and a cluster configuration of one to one thereby allowing the server to be modified without use of a voting system; and

a workload of the cluster assigned to the cluster of one server.

38. A computer readable media, comprising:

said computer readable media containing instructions for execution on a processor for a method of operating a cluster of servers, the method having the steps of;

detecting that an error condition has occurred in a plurality of servers, the plurality of servers making up the cluster;

determining that a failure of the cluster resulted in the impossibility of a conventional quorum being achieved preventing a write operation from occurring on any one server of the plurality of servers and wherein the conventional quorum state requires a majority of the nodes to be healthy to have quorum;

executing a split command by a user to override the quorum requirement by designating a selected server as a full read/write replica of the cluster and forming a cluster of one server from the selected server, the cluster now having a stand alone server and a cluster configuration of one to one thereby allowing the server to be modified without use of a voting system; and

assigning a workload of the cluster to the cluster of one server.