Failover and recovery for replicated data instances

• US 8,074,107 B2
• Filed: 10/26/2009
• Issued: 12/06/2011
• Est. Priority Date: 10/26/2009
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Sivasubramanian, Swaminathan, McAlister, Grant Alexander MacDonald
• Primary Examiner(s): Ehne, Charles
• Application Number:
  US12/606,097
• Publication Number:
  US 20110099420A1
• Time in Patent Office: 771 Days
• Field of Search: 714/4, 714/6, 714/11, 714/12, 714/43, 714/4.1, 714/4.12, 714/6.1, 714/6.2, 714/6.3, 714/6.31
• US Class Current: 714/6.3
• CPC Class Codes: G06F 11/1443 Transmit or communication errors
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 11/2025 using centralised failover control functionality
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/2041 with more than one idle spare processing component
• CPC Class Codes: G06F 11/2048 where the redundant components share neither address space nor persistent storage
• CPC Class Codes: G06F 11/2056 by mirroring
• CPC Class Codes: G06F 11/2064 while ensuring consistency
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 11/2076 Synchronous techniques
• CPC Class Codes: G06F 11/2082 Data synchronisation
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: G06F 16/275 Synchronous replication

Abstract

Replicated instances in a database environment provide for automatic failover and recovery. A monitoring component can periodically communicate with a primary and a secondary replica for an instance, with each capable of residing in a separate data zone or geographic location to provide a level of reliability and availability. A database running on the primary instance can have information synchronously replicated to the secondary replica at a block level, such that the primary and secondary replicas are in sync. In the event that the monitoring component is not able to communicate with one of the replicas, the monitoring component can attempt to determine whether those replicas can communicate with each other, as well as whether the replicas have the same data generation version. Depending on the state information, the monitoring component can automatically perform a recovery operation, such as to failover to the secondary replica or perform secondary replica recovery.

Claims

1. A computer-implemented method of managing recovery of a replicated instance for a relational database instance from a control environment, comprising:

under control of one or more computer systems configured with executable instructions,

periodically communicating with a primary instance replica and a secondary instance replica in a database environment using a monitoring component of a separate control environment, each response received by the at least one monitoring component including status information and data generation information for a respective one of the first and second instance replicas, data updates for the primary instance replica being synchronously replicated to the secondary instance replica for a single data generation;

in response to the at least one monitoring component being unable to communicate with one of the first and second instance replicas, determining whether the first and second instance replicas are able to communicate with each other and whether the first and second instance replicas have common data generation information;

when the monitoring component is unable to communicate with the primary replica for a minimum period of time, the secondary instance replica is unable to communicate with the primary replica, and the second instance replica has the same data generation information as a last known state of the primary replica, causing the secondary instance replica to perform a failover operation to become a new primary replica for the relational database instance;

when the monitoring component is unable to communicate with the secondary replica for a minimum period of time, and the primary instance replica is unable to communicate with the secondary replica, causing a secondary instance replica recovery process to be executed that generates a new secondary instance replica for the relational database instance; and

when the monitoring component is unable to communicate with either of the primary replica and the secondary replica for a minimum period of time, the primary and secondary instance replicas are able to communicate with each other, and the primary and secondary instance replicas have the same data generation information, no failover or recovery operation is performed for the primary and secondary instance replicas.

2. The computer-implemented method of claim 1, further comprising:

when the monitoring component is unable to communicate with the primary replica for a minimum period of time, the secondary instance replica is unable to communicate with the primary replica, and the second instance replica has different data generation information as a last known state of the primary replica, causing a point-in-time recovery operation to be performed for the primary instance replica.

3. The computer-implemented method of claim 1, further comprising:

when the monitoring component is unable to communicate with the primary replica for a minimum period of time and the secondary instance replica is unable to communicate with the primary replica, causing the primary instance replica to terminate itself.

4. The computer-implemented method of claim 1, wherein a user is provided with an alias name enabling the user to communicate with a current primary instance replica, including when a failover operation causes the secondary instance replica to become a new current primary instance replica.

5. A computer-implemented method of managing a replicated database instance in a database environment using a separate control environment, comprising:

under control of one or more computer systems configured with executable instructions,

monitoring state information for each of a primary instance replica and a secondary instance replica in a database environment using a monitoring component of a separate control environment; and

in response to the monitoring component being unable to communicate with one of the first and second instance replicas;

determining failure information including whether the first and second instance replicas are able to communicate with each other and whether the first and second instance replicas have a common data generation identifier;

based at least in part upon the failure information, determining a workflow to be executed in the control environment, the workflow including one or more tasks to be executed in the database environment in response to the monitoring component being unable to communicate with one of the first and second instance replicas; and

executing the workflow in the control environment.

6. The computer-implemented method of claim 5, wherein the workflow includes tasks to cause the secondary instance replica to perform a failover operation to become a new primary replica for the relational database instance when the monitoring component is unable to communicate with the primary replica for a minimum period of time, the secondary instance replica is unable to communicate with the primary replica, and the second instance replica has a common data generation identifier as a last known state of the primary replica.

7. The computer-implemented method of claim 6, wherein performing the failover operation further comprises mounting a file system and starting the database for the new primary replica.

8. The computer-implemented method of claim 5, wherein the workflow includes tasks to cause a secondary instance replica recovery process to be executed that generates a new secondary instance replica for the relational database instance when the monitoring component is unable to communicate with the secondary replica for a minimum period of time and the primary instance replica is unable to communicate with the secondary replica.

9. The computer-implemented method of claim 5, wherein the workflow includes tasks to store information to a data store in the control environment without performing a failover or recovery operation when the monitoring component is unable to communicate with either of the primary replica and the secondary replica for a minimum period of time, the primary and secondary instance replicas are able to communicate with each other, and the primary and secondary instance replicas have a common data generation identifier.

10. The computer-implemented method of claim 5, wherein the first and second instance replicas are provisioned in a single data zone, in separate data zones at separate geographical locations, in a single data zone across multiple geographical locations, or across multiple data zones in a single geographical region.

11. The computer-implemented method of claim 10, wherein the at least one monitoring component is located in a third data zone or geographical location, or in one of a first or second data zone or geographical location.

12. The computer-implemented method of claim 5, wherein a user is provided with an alias name enabling the user to communicate with a current primary instance replica, including when a failover operation causes the secondary instance replica to become a new current primary instance replica.

13. The computer-implemented method of claim 5, further comprising:

storing state information and a data generation identifier for the first and second instance replicas in the control environment.

14. The computer-implemented method of claim 5, wherein each of the first and second instance replicas is run on a separate data instance in the database environment, each data instance being attached to one or more dedicated block storage volumes.

15. The computer-implemented method of claim 14, further comprising:

synchronously replicating data from the primary instance replica to the secondary instance replica using a block-level replication mechanism operable to synchronously replicate data between the one or more dedicated block storage volumes of the first and second instance replicas.

16. The computer-implemented method of claim 5, wherein the primary instance replica is executing a relational database for a customer.

17. A system for managing a replicated database instance in a database environment using a separate control environment, comprising:

a processor; and

a memory device including instructions that, when executed by the processor, cause the processor to;

monitor state information for each of a primary instance replica and a secondary instance replica in a database environment using at least one monitoring component of a separate control environment; and

in response to the at least one monitoring component being unable to communicate with one of the first and second instance replicas;

determine failure information including whether the first and second instance replicas are able to communicate with each other and whether the first and second instance replicas have a common data generation identifier;

based at least in part upon the failure information, determine a workflow to be executed in the control environment, the workflow including one or more tasks to be executed in the database environment in response to the monitoring component being unable to communicate with one of the first and second instance replicas; and

execute the workflow in the control environment.

18. The system of claim 17, wherein the workflow includes tasks to cause the secondary instance replica to perform a failover operation to become a new primary replica for the relational database instance when the monitoring component is unable to communicate with the primary replica for a minimum period of time, the secondary instance replica is unable to communicate with the primary replica, and the first and second instance replicas have a common data generation identifier at a last known state of the primary replica.

19. The system of claim 17, wherein the workflow includes tasks to cause a secondary instance replica recovery process to be executed that generates a new secondary instance replica for the relational database instance when the monitoring component is unable to communicate with the secondary replica for a minimum period of time, and the primary instance replica is unable to communicate with the secondary replica.

20. The system of claim 17, wherein the workflow includes tasks to store information to a data store in the control environment without performing a failover or recovery operation when the monitoring component is unable to communicate with either of the primary replica and the secondary replica for a minimum period of time, the primary and secondary instance replicas are able to communicate with each other, and the primary and secondary instance replicas have a common data generation identifier.

21. The system of claim 17, wherein the first and second instance replicas are provisioned in a single data zone, in separate data zones at separate geographical locations, in a single data zone across multiple geographical locations, or across multiple data zones in a single geographical region.

22. The system of claim 17, wherein a user is provided with an alias name enabling the user to communicate with a current primary instance replica, including when a failover operation causes the secondary instance replica to become a new current primary instance replica.

23. A non-transitory computer-readable storage medium storing instructions for managing a replicated database instance in a database environment using a separate control environment, the instructions when executed by a processor causing the processor to:

monitor state information for each of a primary instance replica and a secondary instance replica in a database environment using at least one monitoring component of a separate control environment; and

in response to the at least one monitoring component being unable to communicate with one of the first and second instance replicas;

determine failure information including whether the first and second instance replicas are able to communicate with each other and whether the first and second instance replicas have a common data generation identifier;

based at least in part upon the failure information, determine a workflow to be executed in the control environment, the workflow including one or more tasks to be executed in the database environment in response to the monitoring component being unable to communicate with one of the first and second instance replicas; and

execute the workflow in the control environment.

24. The non-transitory computer-readable storage medium of claim 23, wherein the workflow includes tasks to cause the secondary instance replica to perform a failover operation to become a new primary replica for the relational database instance when the monitoring component is unable to communicate with the primary replica for a minimum period of time, the secondary instance replica is unable to communicate with the primary replica, and the first and second instance replicas have a common data generation identifier at a last known state of the primary replica.

25. The non-transitory computer-readable storage medium of claim 23, wherein the workflow includes tasks to cause a secondary instance replica recovery process to be executed that generates a new secondary instance replica for the relational database instance when the monitoring component is unable to communicate with the secondary replica for a minimum period of time, and the primary instance replica is unable to communicate with the secondary replica.

26. The non-transitory computer-readable storage medium of claim 23, wherein the workflow includes tasks to store information to a data store in the control environment without performing a failover or recovery operation when the monitoring component is unable to communicate with either of the primary replica and the secondary replica for a minimum period of time, the primary and secondary instance replicas are able to communicate with each other, and the primary and secondary instance replicas have a common data generation identifier.

27. The non-transitory computer-readable storage medium of claim 23, wherein the first and second instance replicas are provisioned in a single data zone, in separate data zones at separate geographical locations, in a single data zone across multiple geographical locations, or across multiple data zones in a single geographical region.