Method and system for consistent cluster operational data in a server cluster using a quorum of replicas

• US 6,401,120 B1
• Filed: 03/26/1999
• Issued: 06/04/2002
• Est. Priority Date: 03/26/1999
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Massa, Michael T., Gamache, Rod
• Primary Examiner(s): Luu, Le Hien
• Application Number:
  US09/277,450
• Time in Patent Office: 1,166 Days
• Field of Search: 707/10, 707/202, 707/204, 709/219, 709/214, 709/217, 709/218, 709/226, 709/220, 709/223
• US Class Current: 709/226
• CPC Class Codes: G06F 11/1425 by reconfiguration of node membership
• CPC Class Codes: G06F 11/1482 by means of middleware or OS functionality
• CPC Class Codes: G06F 11/1662 the resynchronized component or unit being a persistent storage device re-synchronization of failed mirror storage G06F11/2082; rebuild or reconstruction of parity RAID storage G06F11/1008
• CPC Class Codes: G06F 11/181 Eliminating the failing redundant component
• CPC Class Codes: G06F 11/182 based on mutual exchange of the output between redundant processing components
• CPC Class Codes: G06F 11/2023 Failover techniques
• CPC Class Codes: G06F 11/2035 without idle spare hardware

Abstract

A method and system for increasing the availability of a server cluster while reducing its cost by requiring at a minimum only one node and a quorum replica set of storage devices (replica members) to form and continue operating as a cluster. A plurality of replica members maintain the cluster operational data and are independent from any given node. A cluster may be formed and continue to operate as long as one server node possesses a quorum (majority) of the replica members. This ensures that a new or surviving cluster has a least one replica member that belonged to the immediately prior cluster and is thus correct with respect to the cluster operational data. Update sequence numbers and/or timestamps are used to determine the most updated replica member from among those in the quorum for reconciling the other replica members.

Claims

1. A method of operating a server cluster of at least one node, comprising:

storing cluster operational data on a plurality of replica members that are each independent of any node; and

at one node;

obtaining control of a replica set of a number of replica members;

comparing the number of replica members in the replica set with a total number of replica members configured to operate in the cluster; and

forming the cluster if the number of replica members in the replica set is a quorum of the total number of replica members.

2. The method of claim 1 wherein a quorum is one more than half of the total number of replica members.

3. The method of claim 1 further comprising, determining which of the replica members of the replica set has the most updated operational data, and replicating that operational data to the other replica members of the replica set.

4. The method of claim 3 wherein determining which of the replica members of the replica set has the most updated operational data includes, maintaining in association with the cluster operational data on each replica member an identifier that identifies a state of the operational data on that replica member.

5. The method of claim 1 wherein storing cluster operational data includes storing information indicative of the total number of replica members configured to operate in the cluster.

6. The method of claim 1 wherein storing cluster operational data includes storing the state of at least one other storage device of the cluster.

7. The method of claim 1 further comprising, determining that at least one replica member of the replica set is no longer operational, and terminating the cluster if the replica members in the replica set remaining operational do not constitute a quorum of the total number of replica members.

8. The method of claim 1 wherein obtaining control of a replica set includes issuing a reserve command for each replica member.

9. The method of claim 1 wherein obtaining control of a replica set includes arbitrating for exclusive ownership of each member of the replica set.

10. The method of claim 9 wherein arbitrating for exclusive ownership includes executing a mutual exclusion algorithm.

11. The method of claim 1 wherein obtaining control of a replica set includes, arbitrating for exclusive ownership of each member of the replica set using a mutual exclusion algorithm, and exclusively reserving each member of the replica set successfully arbitrated for.

12. The method of claim 1 wherein obtaining control of a replica set includes issuing a reset command, delaying for a period of time, and issuing a reserve command.

13. The method of claim 1 wherein obtaining control of a replica set includes issuing a reset command.

14. A computer-readable medium having computer-executable instructions for performing the method of claim 1.

15. A system for providing consistent operational data of a previous server cluster to a new server cluster, comprising, a plurality of nodes, a plurality of replica members, each replica member being independent of any node, at least one replica member having updated cluster operational data stored thereon by a first node including information indicative of a quorum requirement of a number of replica members needed to form a cluster, and a cluster service on a second node configured to 1) obtain control of a replica set of a number of replica members, 2) compare the number of replica members in the replica set with the quorum requirement, and 3) form the new server cluster if the quorum requirement is met by the number of replica members in the replica set.

16. The system of claim 15 wherein the cluster service further determines which of the replica members of the replica set has the most updated data.

17. The system of claim 16 wherein at least some of the most updated operational data is replicated to at least one other replica member of the replica set.

18. The system of claim 15 wherein each of the replica members maintains in association with the cluster operational data an identifier that identifies a state of the cluster operational data, and wherein the cluster service determines which of the replica members of the replica set has the most updated operational data based on the identifier of each replica member.

19. The system of claim 15 wherein the cluster service terminates the cluster if the number of operational replica members falls below the quorum requirement.

20. The system of claim 15 wherein the second node comprises the first node following a cluster shutdown.

21. A method of forming and operating a server cluster, comprising:

storing cluster operational data on a plurality of replica members, each replica member being independent of any node of the cluster;

arbitrating among a plurality of nodes for exclusive control of a replica set of a number of the replica members; and

at a first node having exclusive control of the replica set;

comparing the number of replica members in the replica set with a total number of replica members configured to operate in the cluster;

forming the cluster if the number of replica members in the replica set is a quorum of the total number of replica members; and

after a failure of at least one replica member, continuing to operate the cluster if the replica members in the replica set remaining operational constitute a quorum of the total number of replica members.

22. The method of claim 21 further comprising terminating the cluster if the replica members in the replica set remaining operational do not constitute a quorum of the total number of replica members.

23. The method of claim 21 further comprising challenging for exclusive ownership of the replica set at a second node.

24. The method of claim 23 wherein the second node is not communicating with the cluster.

25. The method of claim 23 wherein challenging for exclusive ownership of the replica set is performed separately by the second node for each replica member of the replica set.

26. The method of claim 23 further comprising defending exclusive ownership of the replica set at the first node.

27. The method of claim 26 wherein the first node separately defends exclusive ownership for each replica member of the replica set.

28. The method of claim 21 further comprising, determining which of the replica members of the replica set has the most updated cluster operational data, and replicating that operational data to the other replica members of the replica set.

29. The method of claim 21 wherein arbitrating among a plurality of nodes for exclusive control of the replica set includes executing a mutual exclusion algorithm on each of the plurality of nodes.

30. The method of claim 29 further comprising, obtaining exclusive control of the replica set by exclusively reserving each of the members.

31. A computer-readable medium having computer-executable instructions for performing the method of claim 21.

32. A method of obtaining representation of a server cluster at a second node that is not presently representing the cluster, comprising:

storing cluster operational data on a plurality of replica members of a replica set, each replica member being independent of any node of the cluster;

representing the cluster at a first node by obtaining exclusive control of a quorum of the replica members;

defending ownership of the cluster at the first node by regularly persisting exclusive ownership of each of the replica members; and

challenging for exclusive ownership of each of the replica members at the second node, such that if the first node fails and is unable to defend ownership of the replica members, the second node reserves exclusive ownership of a quorum of the replica members and thereby represents the cluster.

33. The method of claim 32 wherein a quorum is one more than half of the total number of replica members.

34. The method of claim 32 further comprising, determining which of the replica members of the replica set has the most updated operational data, and replicating that operational data to the other replica members of the replica set.

35. The method of claim 34, wherein determining which of the replica members of the replica set has the most updated operational data includes, maintaining in association with the cluster operational data on each replica member an identifier that identifies a state of the operational data on that replica member.

36. The method of claim 32 wherein storing cluster operational data includes storing information indicative of the total number of replica members configured to operate in the cluster.

37. The method of claim 32 further comprising, determining that at least one replica member of the replica set is no longer operational, and preventing updates to the cluster configuration data if the replica members in the replica set remaining operational do not constitute a quorum of the total number of replica members.

38. The method of claim 32 wherein obtaining control of a replica set includes arbitrating for exclusive ownership of each member of the replica set.

39. The method of claim 38 wherein arbitrating for exclusive ownership includes executing a mutual exclusion algorithm.

40. The method of claim 32 wherein obtaining control of a replica set includes issuing a reset command.

41. The method of claim 32 wherein obtaining control of a replica set includes issuing a reserve command for each replica member.

42. The method of claim 32 wherein defending ownership includes issuing a reserve command for each replica member.

43. The method of claim 32 wherein obtaining control of a replica set includes, arbitrating for exclusive ownership of each member of the replica set using a mutual exclusion algorithm, and exclusively reserving each member of the replica set successfully arbitrated for.

44. The method of claim 32 wherein challenging for exclusive ownership includes issuing a reset command.

45. The method of claim 44 wherein challenging for exclusive ownership further includes, delaying for a period of time following the reset command, and issuing a reserve command.

46. A computer-readable medium having computer-executable instructions for performing the method of claim 32.

47. In a server cluster, a method, comprising:

storing cluster operational data on a plurality of replica members that are independent of nodes of the server cluster;

arbitrating at a first node for control of the plurality of replica members;

determining whether a quorum of replica members is controlled by the first node, and if so, forming a cluster, determining which replica member has the most updated operational data, and replicating the updated operational data to at least one other replica member; and

while a quorum of the replica members continues to be controlled by the first node, operating the server cluster, defending control of the replica members at the first node, and challenging for control of the replica members at a second node.

48. The method of claim 47 wherein the second node successfully challenges for control of at least some of the replica members, and further comprising, operating the server cluster if the second node controls a quorum of the replica members.

49. The method of claim 47 wherein the second node successfully challenges for control of at least one of the replica members such that the first node no longer controls a quorum of replica members, and further comprising, preventing updates to the cluster operational data until at least one node controls a quorum of the replica members.