Failover and recovery for replicated data instances

• US 9,298,728 B2
• Filed: 11/25/2013
• Issued: 03/29/2016
• Est. Priority Date: 10/26/2009
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: McAlister, Grant Alexander MacDonald, Sivasubramanian, Swaminathan
• Primary Examiner(s): Ehne, Charles
• Application Number:
  US14/089,616
• Publication Number:
  US 20140081916A1
• Time in Patent Office: 855 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1443 Transmit or communication errors
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 11/2025 using centralised failover control functionality
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/2041 with more than one idle spare processing component
• CPC Class Codes: G06F 11/2048 where the redundant components share neither address space nor persistent storage
• CPC Class Codes: G06F 11/2056 by mirroring
• CPC Class Codes: G06F 11/2064 while ensuring consistency
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 11/2076 Synchronous techniques
• CPC Class Codes: G06F 11/2082 Data synchronisation
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: G06F 16/275 Synchronous replication

Abstract

Replicated instances in a database environment provide for automatic failover and recovery. A monitoring component can periodically communicate with a primary and a secondary replica for an instance, with each capable of residing in a separate data zone or geographic location to provide a level of reliability and availability. A database running on the primary instance can have information synchronously replicated to the secondary replica at a block level, such that the primary and secondary replicas are in sync. In the event that the monitoring component is not able to communicate with one of the replicas, the monitoring component can attempt to determine whether those replicas can communicate with each other, as well as whether the replicas have the same data generation version. Depending on the state information, the monitoring component can automatically perform a recovery operation, such as to failover to the secondary replica or perform secondary replica recovery.

Claims

1. A computer-implemented method for managing a replicated database, comprising:

under control of one or more computer systems configured with executable instructions,

obtaining a generation identifier for a primary instance replica and a secondary instance replica of the replicated database upon initial pairing of the primary instance replica and the secondary instance replica, the primary instance replica and the secondary replica associated with a data environment;

synchronizing data between the primary instance replica and the secondary instance replica using a block-level replication mechanism;

periodically providing status information to a monitoring component of a control environment, the control environment being separate from the data environment;

providing failure information to the monitoring component in response to the primary instance replica being unable to communicate with the secondary instance replica, the failure information including at least the generation identifier;

determining that the primary instance replica is able to communicate with the monitoring component;

obtaining a second generation identifier for the primary instance replica; and

performing one or more input/output (I/O) operations via the primary instance replica.

2. The computer-implemented method of claim 1, further comprising:

re-pairing the primary instance replica and the secondary instance replica;

synchronizing the data between the primary instance replica and the secondary instance replica based on the one or more I/O operations performed via the primary instance replica, the one or more I/O operations performed after generating the second generation identifier; and

obtaining a third generation identifier for the primary instance replica and the secondary instance replica.

3. The computer-implemented method of claim 1, further comprising:

pairing the primary instance replica with a new secondary instance replica;

synchronizing the data between the primary instance replica and the new secondary instance replica; and

generating a third generation identifier for the primary instance replica and the new secondary instance replica.

4. The computer-implemented method of claim 1, further comprising:

determining that the primary instance replica is unable to communicate with the monitoring component;

verifying that the generation identifier of the secondary instance replica corresponds to a last known generation identifier of the primary instance replica;

promoting the secondary instance replica to be a new primary instance replica;

pairing the new primary instance replica with a new secondary instance replica;

synchronizing the data between the new primary instance replica and the new secondary instance replica; and

obtaining a second generation identifier for the new primary instance replica and the new secondary instance replica.

5. A system for managing a replicated database, comprising:

a processor; and

a memory device including instructions that, upon being executed by the processor, cause the processor to;

synchronize data between a primary instance replica and a secondary instance replica of the replicated database, the primary instance replica and the secondary replica associated with a data environment;

provide status information to a monitoring component of a control environment, the control environment being separate from the data environment;

provide failure information to the monitoring component in response to the primary instance replica being unable to communicate with the secondary instance replica;

determine that the primary instance replica is able to communicate with the monitoring component;

obtain second data generation information for the primary instance replica; and

perform one or more I/O operations via the primary instance replica.

6. The system of claim 5, wherein the instructions upon being executed further cause the processor to:

obtain data generation information for the primary instance replica and the secondary instance replica upon initial pairing of the primary instance replica and the secondary instance replica,

wherein the failure information includes at least the data generation information.

7. The system of claim 6 wherein the instructions upon being executed further cause the processor to:

verify that the data generation information of the secondary instance replica corresponds to last known data generation information of the primary instance replica;

promote the secondary instance replica to be a new primary instance replica;

pair the new primary instance replica with a new secondary instance replica;

synchronize the data between the new primary instance replica and the new secondary instance replica; and

obtain second data generation information for the new primary instance replica and the new secondary instance replica.

8. The system of claim 6 wherein the data generation information comprises a universally unique identifier.

9. The system of claim 5, wherein the instructions upon being executed further cause the processor to:

re-pair the primary instance replica and the secondary instance replica;

synchronize the data between the primary instance replica and the secondary instance replica based on the one or more I/O operations performed via the primary instance replica, the one or more I/O operations performed after generating the second data generation information; and

obtain third generation identifier for the primary instance replica and the secondary instance replica.

10. The system of claim 5, wherein the instructions upon being executed further cause the processor to:

pair the primary instance replica with a new secondary instance replica;

synchronize the data between the primary instance replica and the new secondary instance replica; and

obtain third data generation information for the primary instance replica and the new secondary instance replica.

11. The system of claim 5, wherein the instructions upon being executed cause the processor to synchronize the data between the primary instance replica and the secondary instance replica based at least in part upon using a block-level replication mechanism.

12. A non-transitory computer-readable storage medium storing instructions for managing a replicated database, the instructions, upon being executed by a processor, causing the processor to:

obtain data generation information for a primary instance replica and a secondary instance replica of the replicated database upon initial pairing of the primary instance replica and the secondary instance replica, the primary instance replica and the secondary replica associated with a data environment

synchronize data between the primary instance replica and the secondary instance replica;

provide status information to a monitoring component of a control environment, the control environment being separate from the data environment;

provide failure information to the monitoring component in response to the primary instance replica being unable to communicate with the secondary instance replica, the failure information including at least the data generation information;

verify that the data generation information of the secondary instance replica corresponds to last known data generation information of the primary instance replica;

promote the secondary instance replica to be a new primary instance replica;

pair the new primary instance replica with a new secondary instance replica;

synchronize the data between the new primary instance replica and the new secondary instance replica; and

obtain second data generation information for the new primary instance replica and the new secondary instance replica.

13. The non-transitory computer-readable storage medium of claim 12, wherein the instructions upon being executed further cause the processor to:

determine that the primary instance replica is able to communicate with the monitoring component;

obtain second data generation information for the primary instance replica; and

perform one or more I/O operations via the primary instance replica.

14. The non-transitory computer-readable storage medium of claim 13, wherein the instructions upon being executed further cause the processor to:

re-pair the primary instance replica and the secondary instance replica;

synchronize the data between the primary instance replica and the secondary instance replica based on the one or more I/O operations performed via the primary instance replica, the one or more I/O operations performed after generating the second data generation information; and

obtain third generation identifier for the primary instance replica and the secondary instance replica.

15. The non-transitory computer-readable storage medium of claim 13, wherein the instructions upon being executed further cause the processor to:

pair the primary instance replica with a new secondary instance replica;

synchronize the data between the primary instance replica and the new secondary instance replica; and

obtain third data generation information for the primary instance replica and the new secondary instance replica.

16. The non-transitory computer-readable storage medium of claim 12, wherein the data generation information comprises a universally unique identifier.

17. The non-transitory computer-readable storage medium of claim 12, wherein the instructions upon being executed cause the processor to synchronize the data between the primary instance replica and the secondary instance replica based at least in part upon using a block-level replication mechanism.