Failover and recovery for replicated data instances

• US 10,860,439 B2
• Filed: 11/13/2017
• Issued: 12/08/2020
• Est. Priority Date: 10/26/2009
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: McAlister, Grant Alexander MacDonald, Sivasubramanian, Swaminathan
• Primary Examiner(s): Ehne, Charles
• Application Number:
  US15/811,565
• Publication Number:
  US 20180067822A1
• Time in Patent Office: 1,121 Days
• CPC Class Codes: G06F 11/1443 Transmit or communication errors
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 11/2025 using centralised failover control functionality
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/2041 with more than one idle spare processing component
• CPC Class Codes: G06F 11/2048 where the redundant components share neither address space nor persistent storage
• CPC Class Codes: G06F 11/2056 by mirroring
• CPC Class Codes: G06F 11/2064 while ensuring consistency
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 11/2076 Synchronous techniques
• CPC Class Codes: G06F 11/2082 Data synchronisation
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: G06F 16/275 Synchronous replication

Abstract

Replicated instances in a database environment provide for automatic failover and recovery. A monitoring component can periodically communicate with a primary and a secondary replica for an instance, with each capable of residing in a separate data zone or geographic location to provide a level of reliability and availability. A database running on the primary instance can have information synchronously replicated to the secondary replica at a block level, such that the primary and secondary replicas are in sync. In the event that the monitoring component is not able to communicate with one of the replicas, the monitoring component can attempt to determine whether those replicas can communicate with each other, as well as whether the replicas have the same data generation version. Depending on the state information, the monitoring component can automatically perform a recovery operation, such as to failover to the secondary replica or perform secondary replica recovery.

Claims

1. A system, comprising:

a plurality of computing nodes, respectively comprising at least one processor and a memory that together implement a control plane for a data store, the control plane configured to;

store data generation information, that identifies a particular generation of replicated data, for each of a primary instance replica and a secondary instance replica, wherein the primary instance replica sends data to the secondary instance replica over a connection between the primary instance replica and the secondary instance replica to perform data replication;

monitor communication over the connection between the primary instance replica and the secondary instance replica and connections between the control plane and the primary instance replica and the control plane and the secondary instance replica; and

perform a type of recovery process responsive to detecting a loss of communication, wherein the type of recovery process is determined based at least in part on the connection over which the loss of communication occurred and respective data generation information, that identifies respective generations of the replicated data, for the primary instance replica and the secondary instance replica.

2. The system of claim 1, wherein the loss of communication comprises:

a failure to respond to a request for status from the control plane by the primary instance replica or the secondary instance replica;

or

receipt of an indication of a loss of communication between the primary instance replica and the secondary instance replica from the primary instance replica or the secondary instance replica.

3. The system of claim 1,

wherein the loss of communication is on the connection between the control plane and the primary instance replica;

wherein a further loss of communication is detected on the connection between the secondary instance replica and the primary instance replica;

wherein the control plane is further configured to determine that the data generation information for the primary instance replica and the secondary instance replica match; and

wherein the particular type of recovery process is a failover operation to promote the secondary instance replica to operate as a new primary instance replica.

4. The system of claim 1, wherein the loss of communication is on the connection between the control plane and the secondary instance replica, wherein a further loss of communication is detected on the connection between the primary instance replica and the secondary instance replica, and wherein the particular type of recovery process is a secondary instance recovery process.

5. The system of claim 1, wherein the control plane is further configured to:

store, in a control plane data store, recovery information comprising the data generation information for each of the primary instance replica and the secondary instance replica and the detected loss of communication; and

access the recovery information in the control plane data store to generate a workflow to be executed, wherein the particular failover operation or recovery process is performed as part of executing the generated workflow.

6. The system of claim 1, wherein a component of the control plane configured to monitor the communication over the connections is located in a data zone different than either the primary instance replica or the secondary instance replica.

7. The system of claim 1, wherein the control plane comprises a plurality of monitoring components to monitor a plurality of primary and secondary instance replicas including the primary instance replica and the secondary instance replica, wherein a first monitoring component is configured to obtain a lease to monitor the primary instance replica and the secondary instance replica that excludes other ones of the plurality of monitoring components from monitoring the primary instance replica and the secondary instance replica.

8. A method, comprising:

performing, by one or more computers,

obtaining, by a monitoring component, data generation information, that identifies respective generations of replicated data, for each of a primary instance replica and a secondary instance replica, wherein the primary instance replica sends data to the secondary instance replica over a connection between the primary instance replica and the secondary instance replica to perform data replication;

monitoring communication over connections between the monitoring component and the primary instance replica, the monitoring component and the secondary instance replica, and the primary instance replica and the secondary instance replica; and

performing a type of recovery process responsive to a loss of communication detected by the monitoring component, wherein the type of recovery process is determined based at least in part on the connection over which the loss of communication occurred and the respective data generation information, that identifies the respective generations of the replicated data, for the primary instance replica and the secondary instance replica.

9. The method of claim 8, wherein the loss of communication detected by the monitoring component comprises:

a failure to respond to a request for status from the monitoring component by the primary instance replica or the secondary instance replica;

or

receipt, by the monitoring component, of an indication of a loss of communication between the primary instance replica and the secondary instance replica from the primary instance replica or the secondary instance replica.

10. The method of claim 8,

wherein the loss of communication is on the connection between the monitoring component and the primary instance replica;

wherein a further loss of communication is detected on the connection between the secondary instance replica and the primary instance replica;

wherein the method further comprises determining that the data generation information for the primary instance replica and the secondary instance replica match; and

wherein the particular type of recovery process is a failover operation to promote the secondary instance replica to operate as a new primary instance replica.

11. The method of claim 8, wherein the loss of communication is on the connection between the monitoring component and the secondary instance replica, wherein a further loss of communication is detected on the connection between the primary instance replica and the secondary instance replica, and wherein the particular type of recovery process is a secondary instance recovery process.

12. The method of claim 8, further comprising:

storing, by the monitoring component, recovery information comprising the data generation information for each of the primary instance replica and the secondary instance replica and the detected loss of communication in a control plane data store; and

accessing the recovery information in the control plane data store to generate a workflow to be executed, wherein the particular failover operation or recovery process is performed as part of executing the generated workflow.

13. The method of claim 8, wherein the monitoring component is located in a data zone different than either the primary instance replica or the secondary instance replica.

14. The method of claim 8, wherein the monitoring component is one of a plurality of monitoring components implemented as part of a control plane that monitors a plurality of primary and secondary instance replicas including the primary instance replica and the secondary instance replica, wherein prior to obtaining the data generation information for the primary instance replica and the secondary instance replica, the monitoring component obtains a lease to monitor the primary instance replica and the secondary instance replica that excludes other ones of the plurality of monitoring components from monitoring the primary instance replica and the secondary instance replica.

15. A non-transitory, computer-readable storage medium, comprising program instructions that when executed by one or more computing devices cause the one or more computing devices to implement:

obtaining, by a monitoring component, data generation information, that identifies respective generations of replicated data, for each of a primary instance replica and a secondary instance replica, wherein the primary instance replica sends data to the secondary instance replica over a connection between the primary instance replica and the secondary instance replica to perform data replication;

monitoring communication over connections between the monitoring component and the primary instance replica, the monitoring component and the secondary instance replica, and the primary instance replica and the secondary instance replica; and

performing a type of recovery process responsive to a loss of communication detected by the monitoring component, wherein the type of recovery process is determined based at least in part on the connection over which the loss of communication occurred and the respective data generation information, that identifies the respective generations of the replicated data, for the primary instance replica and the secondary instance replica.

16. The non-transitory, computer-readable storage medium of claim 15, wherein the loss of communication detected by the monitoring component comprises:

a failure to respond to a request for status from the monitoring component by the primary instance replica or the secondary instance replica;

or

receipt, by the monitoring component, of an indication of a loss of communication between the primary instance replica and the secondary instance replica from the primary instance replica or the secondary instance replica.

17. The non-transitory, computer-readable storage medium of claim 15,

wherein the loss of communication is on the connection between the monitoring component and the primary instance replica;

wherein a further loss of communication is detected on the connection between the secondary instance replica and the primary instance replica;

wherein the program instructions further cause the one or more computing devices to implement;

determining that the data generation information for the primary instance replica and the secondary instance replica match; and

wherein the particular type of recovery process is a failover operation to promote the secondary instance replica to operate as a new primary instance replica.

18. The non-transitory, computer-readable storage medium of claim 15, wherein the loss of communication is on the connection between the monitoring component and the secondary instance replica, wherein a further loss of communication is detected on the connection between the primary instance replica and the secondary instance replica, and wherein the particular type of recovery process is a secondary instance recovery process.

19. The non-transitory, computer-readable storage medium of claim 15, wherein the program instructions cause the one or more computing devices to implement:

storing, by the monitoring component, recovery information comprising the data generation information for each of the primary instance replica and the secondary instance replica and the detected loss of communication in a control plane data store; and

accessing the recovery information in the control plane data store to generate a workflow to be executed, wherein the particular failover operation or recovery process is performed as part of executing the generated workflow.

20. The non-transitory, computer-readable storage medium of claim 15, wherein the monitoring component is located in a data zone different than either the primary instance replica or the secondary instance replica.