Hybrid cluster recovery techniques

• US 10,454,754 B1
• Filed: 12/16/2016
• Issued: 10/22/2019
• Est. Priority Date: 12/16/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Patel, Parth, Chawla, Rachit, Cheema, Aseem, Kanawati, Antoun Joubran, Kumar, Manoj
• Primary Examiner(s): Wang, Liang Che A
• Application Number:
  US15/382,340
• Time in Patent Office: 1,040 Days
• Field of Search: 709201, 709223, 709224, 709230
• CPC Class Codes: H04L 41/0654 using network fault recovery ring fault isolation or reconfiguration in loop networks without recovery actions by a network management system H04L12/437
• CPC Class Codes: H04L 41/0668 by dynamic selection of recovery network elements, e.g. replacement by the most appropriate element after failure
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

At a recovery manager associated with a cluster, a determination is made as to whether a replacement for a first node of the cluster can be elected by the other nodes of the cluster using a first election protocol. The recovery manager selects a second node of the cluster as a replacement for the first node, based on data item replication progress made at the node, and transmits an indication that the second node has been selected to one or more nodes of the cluster.

Claims

1. A system, comprising:

memory storing program instructions that, if executed, cause one or more processors to;

determine that a replacement master node of a first cluster either cannot be elected by nodes of the first cluster using a first election protocol, or has a probability below a particular threshold of being elected by the nodes of the first cluster using the first election protocol;

select, responsive to the determination, and based at least in part on an indication of more progress of replication of one or more data items at a first node of the first cluster than at one or more other nodes of the first cluster, the first node as the replacement master node; and

transmit, by a first recovery manager to at least one node of the first cluster, an indication that the first node has been selected as the replacement master node.

2. The system as recited in claim 1, wherein the first node stores at least a portion of a distributed in-memory database.

3. The system as recited in claim 1, wherein the program instructions, if executed, cause the one or more processors to select the first node as the replacement master node based at least in part on an indication of connectivity of the first node to other nodes of the first cluster.

4. The system as recited in claim 1, wherein the program instructions, if executed, cause the one or more processors to:

transmit to at least one node of the cluster, a directive indicating that the first node is not to be designated as the replacement master node if the first cluster enters a first state.

5. The system as recited in claim 1, wherein to determine whether a replacement master node of a first cluster can be elected by nodes of the first cluster the program instructions, if executed, cause the one or more processors to:

obtain, in response to one or more queries, inter-node connectivity status information from at least one node of the first cluster.

6. A method, comprising:

determining that a replacement master node of a first cluster either cannot be elected by nodes of the first cluster using a first election protocol, or has a probability below a particular threshold of being elected by the nodes of the first cluster using the first election protocol;

selecting, responsive to the determination, a first node of the first cluster as the replacement master node of the first cluster, wherein said selecting is based at least in part on an indication of more progress of replication of one or more data items at the first node than at one or more other nodes of the first cluster; and

transmitting, by a first recovery manager to at least one node of the first cluster, an indication that the first node has been selected as the replacement master node.

7. The method as recited in claim 6, wherein the first node stores at least a portion of a distributed in-memory database.

8. The method as recited in claim 6, wherein said selecting the first node as the replacement master node is based at least in part on an indication of a connectivity of the first node to other nodes of the first cluster.

9. The method as recited in claim 6, further comprising:

transmitting, by the first recovery manager to at least one node of the one or more nodes, a directive indicating that the first node is not to be designated as the replacement master node if the first cluster enters a first state.

10. The method as recited in claim 6, wherein said determining whether the replacement master node of the first cluster can be elected by nodes of the first cluster is performed at the first recovery manager, and wherein the method further comprises:

determining, at the first recovery manager, based at least in part on configuration status information pertaining to one or more nodes of a second cluster, whether a replacement master node of the second cluster can be elected by nodes of the second cluster.

11. The method as recited in claim 6, wherein according to the first election protocol, a quorum comprising a plurality of nodes of the first cluster is required to elect a replacement master node.

12. The method as recited in claim 6, further comprising:

receiving, by the first recovery manager, a first response to a first query, wherein the first response comprises an indication of connectivity status pertaining to the first node and at least one other node of the first cluster; and

determining, by the first recovery manager based at least in part on the indication of connectivity status, that a replacement master node of the first cluster cannot be elected by nodes of the first cluster using the first election protocol.

13. The method as recited in claim 12, further comprising:

determining, by the first node based at least in part on a gossip protocol, a connectivity status of the first node with respect to a second node of the first cluster; and

transmitting, by the first node to the first recovery manager, the first response.

14. The method as recited in claim 6, wherein at least one node of the one or more nodes is instantiated at a first resource group of a provider network, wherein the provider network comprises a second resource group, the method further comprising:

obtaining, at the first recovery manager, from a second recovery manager, an indication of connectivity between the second resource group and the at least one node, wherein said determining whether a replacement master node of the first cluster can be elected is based at least in part on the indication of connectivity.

15. The method as recited in claim 6, further comprising:

determining, at the first recovery manager, that a second recovery manager has failed, wherein prior to the failure the second recovery manager was designated to monitor at least a second node of a second cluster; and

designating, by a plurality of recovery managers using a workload distribution protocol, the first recovery manager to monitor the second node.

16. A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors cause the one or more processors to perform a method comprising:

determining, based at least in part on one or more queries directed to one or more nodes of a first cluster, that a replacement for a first node of the first cluster either cannot be elected by nodes of the first cluster using a first election protocol, or has a probability below a particular threshold of being elected by the nodes of the first cluster using the first election protocol;

selecting, responsive to the determination, a second node of the one or more nodes as the replacement for the first node, wherein said selecting is based at least in part on an indication of more progress of replication of one or more data items at the second node of the first cluster than at one or more other nodes of the first cluster; and

transmitting, by a first recovery manager to at least one node of the first cluster, an indication that the second node has been selected as the replacement for the first node.

17. The non-transitory computer-accessible storage medium as recited in claim 16, wherein the first node stores at least a portion of a distributed in-memory database.

18. The non-transitory computer-accessible storage medium as recited in claim 16, wherein said selecting the second node as the replacement is based at least in part on an indication of a connectivity of the second node to other nodes of the first cluster.

19. The non-transitory computer-accessible storage medium as recited in claim 16, wherein determining whether the replacement node can be elected comprises determining whether, based at least in part on an expiration of a timeout, the second node has entered a state in which the second node cannot initiate a failover from the first node.

20. The non-transitory computer-accessible storage medium as recited in claim 16, wherein the method further comprises:

determining, at the first recovery manager, based at least in part on one or more queries directed to one or more nodes of a second cluster, whether a replacement for a first node of the second cluster can be elected by one or more other nodes of the second cluster.