CONSISTENT CLUSTER OPERATIONAL DATA IN A SERVER CLUSTER USING A QUORUM OF REPLICAS

• US 20100250750A1
• Filed: 06/14/2010
• Published: 09/30/2010
• Est. Priority Date: 03/26/1999
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: NISHANOV, GOR V., GAMACHE, ROD N., SHRIVASTAVA, SUNITA, DION, DAVID A., LOMET, DAVID B., BERNSTEIN, PHILIP A., DAS, RAJSEKHAR, JAIN, ROHIT, DOSHI, RUSHABH A., MASSA, MICHAEL T.
• Granted Patent: US 7,984,155 B2
• Time in Patent Office: Days
• US Class Current: 709/226
• CPC Class Codes: G06F 11/1425 by reconfiguration of node membership
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 11/1479 Generic software techniques for error detection or fault masking
• CPC Class Codes: G06F 11/1482 by means of middleware or OS functionality
• CPC Class Codes: G06F 11/1662 the resynchronized component or unit being a persistent storage device re-synchronization of failed mirror storage G06F11/2082; rebuild or reconstruction of parity RAID storage G06F11/1008
• CPC Class Codes: G06F 11/181 Eliminating the failing redundant component
• CPC Class Codes: G06F 11/182 based on mutual exchange of the output between redundant processing components
• CPC Class Codes: G06F 11/2023 Failover techniques
• CPC Class Codes: G06F 11/2035 without idle spare hardware
• CPC Class Codes: G06F 11/2058 using more than 2 mirrored copies
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency ensuring consistency in mirrored systems G06F11/2064

Abstract

A method and system for increasing server cluster availability by requiring at a minimum only one node and a quorum replica set of replica members to form and operate a cluster. Replica members maintain cluster operational data. A cluster operates when one node possesses a majority of replica members, which ensures that any new or surviving cluster includes consistent cluster operational data via at least one replica member from the immediately prior cluster. Arbitration provides exclusive ownership by one node of the replica members, including at cluster formation, and when the owning node fails. Arbitration uses a fast mutual exclusion algorithm and a reservation mechanism to challenge for and defend the exclusive reservation of each member. A quorum replica set algorithm brings members online and offline with data consistency, including updating unreconciled replica members, and ensures consistent read and update operations.

Claims

1. A method comprising:

maintaining operational data of a cluster on a replica set comprising a plurality of replica members, wherein each of the plurality of replica members is a device configured to maintain a replica of the operational data, and wherein the plurality of replica members includes at least one independent replica member that is not an element of any node of the cluster, and wherein the at least one independent replica member is configured to be online in the replica set regardless of how many of the any nodes are operable in the cluster;

maintaining an epoch number in association with each replica member of the plurality of replica members, wherein a value of each epoch number indicates a state of the operational data on the corresponding each replica member;

maintaining a sequence number in association with each replica member of the plurality of replica members;

determining one of the plurality of replica members that comprises operational data that is most updated, wherein the determining includes determining that at least two replica members of the plurality of replica members have epoch numbers that equal a largest epoch number and determining that one of the at least two replica members has a sequence number that equals a largest sequence number; and

replicating, in response to the determining, at least some of the operational data that is most updated from the one of the at least two replica members to other replica members of the plurality of replica members.

2. The method of claim 1 further comprising, evaluating a last record logged on at least one of the other replica members to which data is being replicated against at least one record of the at least some of the operational data to determine whether to discard the last record.

3. The method of claim 2 further comprising, evaluating a second-to-last record logged on the at least one of the other replica members to which data is being replicated against at least one record of at least some of the operational data to determine whether to discard the second-to-last record.

4. The method of claim 1 wherein each of the replica members can be controlled by only one of the any nodes at a time.

5. The method of claim 1 further comprising, representing the cluster if a number of the plurality of replica members exclusively owned and controlled by only one of the any nodes comprises at least a majority of a total number of the plurality of replica members of the replica set.

6. The method of claim 1 further comprising:

detecting availability of a new replica member that is configured to operate in the replica set; and

reconciling the cluster operational data of the new replica member.

7. The method of claim 1 further comprising:

detecting unavailability of a non-operational replica member of the plurality of replica members wherein the non-operational replica member was previously operational;

determining, in response to the detecting, that a majority of the plurality of replica members are not operational;

halting, in response to the determining, the replicating; and

executing, in response to the halting, a recovery process to attempt to obtain control of the majority of the plurality of replica members.

8. The method of claim 1 wherein the maintaining operational data includes storing a total number of configured replica members of the plurality of replica members wherein the configured replica members are configured to operate in the cluster.

9. The method of claim 1 wherein the replica set includes at least one local replica member wherein the at least one local replica member is an element of one of the any nodes of the cluster.

20. The computer-readable device of claim 1 wherein the maintaining operational data includes storing a total number of configured replica members of the plurality of replica members wherein the configured replica members are configured to operate in the cluster.

10. A clustering system comprising:

a first node of a plurality of nodes of the clustering system, the first node including a first local replica member of a replica set, and wherein each replica member of the replica set is a device configured to maintain a replica of operational data of the clustering system;

an independent replica member of the replica set wherein the independent replica member is a device that is not an element of any node of the plurality of nodes, and wherein the independent replica member is configured to be online in the replica set regardless of how many of the plurality of nodes are operable in the clustering system, and wherein the independent replica member can be controlled by only one node of the plurality of nodes at a time;

the first node and the independent replica member configured to form at least a portion of the clustering system; and

the first local replica member and the independent replica member configured to be exclusively owned and controlled by the only one node of the plurality of nodes and configured to form a quorum replica set, wherein others of the plurality of nodes access the operational data of the clustering system, based on the exclusively owned and controlled, only through the only one node.

11. The clustering system of claim 10 wherein the quorum replica set is comprised of the first local replica member in the event that the independent replica member fails to operate.

12. The clustering system of claim 10 further comprising:

a first epoch number in association with the first local replica member, wherein the first local epoch number is configured to indicate a state of the replica of the operational data on the first local replica member;

a sequence number in association with the first local replica member;

a second epoch number in association with the independent replica member, wherein the second epoch number is configured to indicate a state of the replica of the operational data on the independent replica member;

a sequence number in association with the independent replica member; and

wherein a largest of the first epoch number and the second epoch number in combination with a largest of the first sequence number and the second sequence number indicates a replica member that is most updated.

13. A computer-readable device comprising computer-readable instructions that, when executed by a computer, cause the computer to perform a method comprising:

maintaining operational data of a cluster on a replica set comprising a plurality of replica members, wherein each of the plurality of replica members is a device configured to maintain a replica of the operational data, and wherein the plurality of replica members includes at least one independent replica member that is not an element of any node of the cluster, and wherein the at least one independent replica member is configured to be online in the replica set regardless of how many of the any nodes are operable in the cluster;

maintaining an epoch number in association with each replica member of the plurality of replica members, wherein a value of each epoch number indicates a state of the operational data on the corresponding each replica member;

maintaining a sequence number in association with each replica member of the plurality of replica members;

determining one of the plurality of replica members that comprises operational data that is most updated, wherein the determining includes determining that the at least two replica members have epoch numbers that equal a largest epoch number and determining that one of the at least two replica members has a sequence number that equals a largest sequence number; and

replicating, in response to the determining, at least some of the operational data that is most updated from the one of the at least two replica members to other replica members of the plurality of replica members.

14. The computer-readable device of claim 13, the method further comprising evaluating a last record logged on at least one of the other replica members to which data is being replicated against at least one record of the at least some of the operational data to determine whether to discard the last record.

16. The computer-readable device of claim 13 wherein each of the replica members can be controlled by only one of the any nodes at a time.

17. The computer-readable device of claim 13, the method further comprising representing the cluster if a number of the plurality of replica members exclusively owned and controlled by only one of the any nodes comprises at least a majority of a total number of the plurality of replica members of the replica set.

18. The computer-readable device of claim 13, the method further comprising:

detecting availability of a new replica member that is configured to operate in the replica set; and

reconciling the cluster operational data of the new replica member.

19. The computer-readable device of claim 13, the method further comprising:

detecting unavailability of a non-operational replica member of the plurality of replica members wherein the non-operational replica member was previously operational;

determining, in response to the detecting, that a majority of the plurality of replica members are not operational;

halting, in response to the determining, the replicating; and

executing, in response to the halting, a recovery process to attempt to obtain control of the majority of the plurality of replica members.

15. The computer-readable device of claim 15, the method further comprising evaluating a second-to-last record logged on the at least one of the other replica members to which data is being replicated against at least one record of at least some of the operational data to determine whether to discard the second-to-last record.