DYNAMICALLY CHANGING MEMBERS OF A CONSENSUS GROUP IN A DISTRIBUTED SELF-HEALING COORDINATION SERVICE

• US 20160019125A1
• Filed: 07/17/2014
• Published: 01/21/2016
• Est. Priority Date: 07/17/2014
• Status: Active Grant

• Current Assignee: Cohesity, Inc.
• Original Assignee: Cohesity, Inc.
• Inventors: ARON, Mohit, MADDURI, Sashikanth, REDDY, Vinay, GUPTA, Vipin
• Granted Patent: US 9,690,675 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1425 by reconfiguration of node membership
• CPC Class Codes: G06F 11/1658 Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit
• CPC Class Codes: G06F 11/2005 using redundant communication controllers
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: H04L 41/0668 by dynamic selection of recovery network elements, e.g. replacement by the most appropriate element after failure
• CPC Class Codes: H04L 41/12 Discovery or management of network topologies
• CPC Class Codes: H04L 41/5009 Determining service level performance parameters or violations of service level contracts, e.g. violations of agreed response time or mean time between failures [MTBF]
• CPC Class Codes: H04L 41/5096 wherein the managed service relates to distributed or central networked applications
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

Systems, methods, and computer program products for managing a consensus group in a distributed computing cluster, by determining that an instance of an authority module executing on a first node, of a consensus group of nodes in the distributed computing cluster, has failed; and adding, by an instance of the authority module on a second node of the consensus group, a new node to the consensus group to replace the first node. The new node is a node in the computing cluster that was not a member of the consensus group at the time the instance of the authority module executing on the first node is determined to have failed.

Claims

1. A method for managing a consensus group in a distributed computing cluster, the method comprising:

determining that an instance of an authority module executing on a first node, of a consensus group of nodes in the distributed computing cluster, has failed; and

adding, by an instance of the authority module on a second node of the consensus group, a new node to the consensus group to replace the first node, wherein the new node comprises a node in the computing cluster that was not a member of the consensus group at the time the instance of the authority module executing on the first node is determined to have failed.

2. The method of claim 1, further comprising:

transmitting to the new node;

a set of nodes in the distributed computing cluster eligible to join the consensus group; and

a state of a set of locks issued to nodes in the distributed computing cluster.

3. The method of claim 1, wherein the instance of the authority module executing on the second node was elected by members of the consensus group to serve as a master instance of the authority module for the consensus group prior to the failure of the instance of the authority module executing on the first node

4. The method of claim 1, further comprising:

wherein the instance of the authority module executing on the first node was elected by members of the consensus group to serve as a master instance of the authority module for the consensus group prior to the failure of the instance of the authority module executing on the first node, and wherein the method further comprises, prior to adding the new node to the consensus group, electing, by remaining members of the consensus group, the instance of the authority module executing on the second node to serve as the master instance of the authority module for the consensus group.

5. The method of claim 4, further comprising:

transmitting, by the instance of the authority module on the second node, to each node in the distributed computing cluster, an indication that the instance of the authority module on the second node has been elected as the master authority module for the consensus group.

6. The method of claim 1, further comprising:

discovering a new node added to the computing cluster;

adding the new node to a set of available nodes in the computing cluster; and

sharing the set of available nodes with members of the consensus group.

7. The method of claim 1, further comprising:

removing, by the second node, the first node from the consensus group.

8. A distributed computing cluster, comprising:

a plurality of nodes configured to provide a service to clients, each node having a processor and a memory;

a consensus group of nodes formed from a subset of the plurality of nodes, the consensus group of nodes, each executing an instance of an authority module which performs operations for managing the consensus group, the operation, comprising;

determining that an instance of an authority module executing on a first node of the consensus group, has failed, and

adding, by an instance of the authority module on a second node of the consensus group, a new node to the consensus group to replace the first node, wherein the new node comprises a node in the computing cluster that was not a member of the consensus group at the time the instance of the authority module executing on the first node is determined to have failed.

9. The distributed computing cluster of claim 8, wherein the operation further comprises:

transmitting to the new node;

a set of nodes in the distributed computing cluster eligible to join the consensus group; and

a state of a set of locks issued to nodes in the distributed computing cluster.

10. The distributed computing cluster of claim 8, wherein the instance of the authority module executing on the second node was elected by members of the consensus group to serve as a master instance of the authority module for the consensus group prior to the failure of the instance of the authority module executing on the first node

11. The distributed computing cluster of claim 8, wherein the operation further comprises:

wherein the instance of the authority module executing on the first node was elected by members of the consensus group to serve as a master instance of the authority module for the consensus group prior to the failure of the instance of the authority module executing on the first node, and wherein the method further comprises, prior to adding the new node to the consensus group, electing, by remaining members of the consensus group, the instance of the authority module executing on the second node to serve as the master instance of the authority module for the consensus group.

12. The distributed computing cluster of claim 11, wherein the operation further comprises:

transmitting, by the instance of the authority module on the second node, to each node in the distributed computing cluster, an indication that the instance of the authority module on the second node has been elected as the master authority module for the consensus group.

13. The distributed computing cluster of claim 8, wherein the operation further comprises:

discovering a new node added to the computing cluster;

adding the new node to a set of available nodes in the computing cluster; and

sharing the set of available nodes with members of the consensus group.

14. The distributed computing cluster of claim 8, wherein the operation further comprises:

removing, by the second node, the first node from the consensus group.

15. A computer-readable storage medium storing instructions, which, when executed on a processor, perform operations for managing a consensus group in a distributed computing cluster, the operation comprising:

determining that an instance of an authority module executing on a first node, of a consensus group of nodes in the distributed computing cluster, has failed; and

adding, by an instance of the authority module on a second node of the consensus group, a new node to the consensus group to replace the first node, wherein the new node comprises a node in the computing cluster that was not a member of the consensus group at the time the instance of the authority module executing on the first node is determined to have failed.

16. The computer-readable storage medium of claim 15, further comprising:

transmitting to the new node;

a set of nodes in the distributed computing cluster eligible to join the consensus group; and

a state of a set of locks issued to nodes in the distributed computing cluster.

17. The computer-readable storage medium of claim 15, wherein the instance of the authority module executing on the second node was elected by members of the consensus group to serve as a master instance of the authority module for the consensus group prior to the failure of the instance of the authority module executing on the first node

18. The computer-readable storage medium of claim 15, further comprising:

wherein the instance of the authority module executing on the first node was elected by members of the consensus group to serve as a master instance of the authority module for the consensus group prior to the failure of the instance of the authority module executing on the first node, and wherein the method further comprises, prior to adding the new node to the consensus group, electing, by remaining members of the consensus group, the instance of the authority module executing on the second node to serve as the master instance of the authority module for the consensus group.

19. The computer-readable storage medium of claim 18, further comprising:

transmitting, by the instance of the authority module on the second node, to each node in the distributed computing cluster, an indication that the instance of the authority module on the second node has been elected as the master authority module for the consensus group.

20. The computer-readable storage medium of claim 15, further comprising:

discovering a new node added to the computing cluster;

adding the new node to a set of available nodes in the computing cluster; and

sharing the set of available nodes with members of the consensus group.

21. The computer-readable storage medium of claim 15, further comprising:

removing, by the second node, the first node from the consensus group.