Monitoring of replicated data instances

• US 8,676,753 B2
• Filed: 10/26/2009
• Issued: 03/18/2014
• Est. Priority Date: 10/26/2009
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Sivasubramanian, Swaminathan, McAlister, Grant Alexander MacDonald, Hunter, Barry B. Jr., Brazil, Silas M.
• Primary Examiner(s): Alam, Hosain
• Assistant Examiner(s): OBERLY, VAN HONG
• Application Number:
  US12/606,106
• Publication Number:
  US 20110099146A1
• Time in Patent Office: 1,604 Days
• Field of Search: 707/614
• US Class Current: 707/614
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/203 using migration
• CPC Class Codes: G06F 11/2035 without idle spare hardware
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 11/2076 Synchronous techniques
• CPC Class Codes: G06F 11/2082 Data synchronisation
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3055 Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction by redundancy in operation G06F11/14; error detection or correction by redundancy in hardware G06F11/16
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route

Abstract

Replicated instances in a database environment provide for automatic failover and recovery. A monitoring component can obtain a lease enabling the component to periodically communicate with, and monitor, one or more data instances in the data environment, where the data instance can be a replicated instance including a primary and a secondary replica. For a large number of instances, the data environment can be partitioned such that each monitoring component can be assigned a partition of the workload. In the event of a failure of a monitoring component, the instances can be repartitioned and the remaining monitoring components can be assigned to the new partitions to substantially evenly distribute the workload.

Claims

1. A computer-implemented method of monitoring replicated database instances from a control environment, comprising:

under control of one or more computer systems configured with executable instructions,

assigning each of a plurality of replicated database instances in a database environment to one of a plurality of workload partitions, the plurality of workload partitions grouped according to one of a plurality of data zones or a plurality of geographical locations;

assigning each of a plurality of monitoring components in the control environment to one of the plurality of workload partitions, each monitoring component being assigned to a workload partition that is grouped to one of a first data zone or a first geographic zone that is different from a second data zone or a second geographic zone of the workload partition to which at least one of a primary instance replica or a secondary instance replica of a replicated database instance is assigned;

for each replicated database instance of the plurality of replicated database instances;

causing an assigned monitoring component to send a communication to a host manager for the primary instance replica of the replicated database instance, the assigned monitoring component being assigned to the workload partition to which the replicated database instance is assigned;

when data for the primary instance replica is synchronized with data for the secondary instance replica for the replicated database instance, receiving to the assigned monitoring component a lease for the replicated database instance, the lease specifying at least a lease period during which the assigned monitoring component will be able to monitor the replicated database instance,

wherein, during the lease period, other monitoring components of the plurality of monitoring components are prohibited from initiating failover of the replicated database instance upon determination of failure of the replicated database instance; and

in response to receiving the lease to the assigned monitoring component, monitoring at least a status of the replicated database instance during the lease period using the assigned monitoring component.

2. The computer-implemented method of claim 1, wherein the assigned monitoring component receives the lease only when a current lease exists for the replicated database instance.

3. The computer-implemented method of claim 1, further comprising:

storing an identifier for the assigned monitoring component and information for the lease period to a first block storage mechanism for the primary instance replica, the first block storage mechanism causing the identifier and the information for the lease period to be synchronously stored to a second block storage mechanism for the secondary instance replica.

4. A computer-implemented method of monitoring database instances in a database environment from a control environment, comprising:

under control of one or more computer systems configured with executable instructions,

assigning a monitoring component of a plurality of monitoring components in the control environment to a database instance in a database environment, the plurality of monitoring components grouped according to one of a plurality of data zones or a plurality of geographic locations, the database instance capable of being replicated to include at least a primary instance replica and a secondary instance replica, the monitoring component being assigned to at least one of a first data zone or a first geographical location that is different from a second data zone or a second geographical location of at least one of the primary instance replica or the secondary instance replica in the database environment when the database instance is replicated;

causing the monitoring component to send a communication to the database instance, the communication being sent to at least the primary instance replica when the database instance is replicated; and

in response to receiving lease information from the database instance, monitoring at least a status of the database instance using the monitoring component during a lease period associated with the lease information,

wherein, during the lease period, other monitoring components of the plurality of monitoring components in the control environment are prohibited from initiating failover of the database instance upon determination of failure of the database instance.

5. The computer-implemented method of claim 4, wherein the lease information is received from the primary instance replica only when data is synchronized between the primary instance replica and the secondary instance replica.

6. The computer-implemented method of claim 4, wherein the lease information is received from the primary instance replica only when a current lease exists for the database instance.

7. The computer-implemented method of claim 4, wherein the database environment includes a plurality of database instances, the method further comprising:

assigning each of the plurality of monitoring components in the control environment to a portion of the plurality of database instances in the database environment in order to substantially evenly distribute workload across the plurality of monitoring components.

8. The computer-implemented method of claim 7, further comprising:

assigning each of the plurality of database instances in the database environment to one of a plurality of workload partitions,

wherein assigning each of the plurality of monitoring components in the control environment to a portion of the plurality of database instances comprises assigning each of the plurality of monitoring components to one of the plurality of workload partitions.

9. The computer-implemented method of claim 8, further comprising:

when at least one of the plurality of monitoring components is unable to monitor a workload partition to which the at least one of the plurality of monitoring components is assigned, repartitioning the plurality of workload partitions and reassigning a remaining group of the plurality of monitoring components to a repartitioned plurality of workload partitions.

10. The computer-implemented method of claim 8, further comprising:

causing heartbeat messages to be periodically sent between the plurality of monitoring components in order to determine when the at least one of the plurality of monitoring components is unable to monitor the workload partition to which the at least one of the plurality of monitoring components is assigned.

11. The computer-implemented method of claim 4, further comprising:

when the monitoring component determines that the primary instance replica is unavailable, causing the secondary instance replica to failover to a new primary instance replica.

12. The computer-implemented method of claim 4, further comprising:

storing an identifier for the monitoring component and information for the lease period to a first block storage mechanism for the primary instance replica, the first block storage mechanism causing the identifier and the information for the lease period to be synchronously stored to a second block storage mechanism for the secondary instance replica.

13. The computer-implemented method of claim 12, wherein the identifier is a random long identifier.

14. The computer-implemented method of claim 4, further comprising:

storing state information and a data generation identifier for the primary instance replica and the secondary instance replica, a monitoring component identifier, and lease period information in memory for the monitoring component in the control environment.

15. A system for monitoring database instances in a database environment from a control environment, comprising:

a processor; and

a memory device including instructions that, when executed by the processor, cause the processor to;

assign a monitoring component of a plurality of monitoring components in the control environment to a database instance in the database environment, the plurality of monitoring components grouped according to one of a plurality of data zones or a plurality of geographic locations, the database instance capable of being replicated to include at least a primary instance replica and a secondary instance replica, the monitoring component being assigned to at least one of a first data zone or a first geographical location that is different from a second data zone or a second geographical location of at least one of the primary instance replica or the secondary instance replica in the database environment when the database instance is replicated;

cause the monitoring component to send a communication to the database instance, the communication being sent to at least the primary instance replica when the database instance is replicated; and

in response to receiving lease information from the database instance, monitor at least a status of the database instance using the monitoring component during a lease period associated with the lease information,

wherein, during the lease period, other monitoring components of the plurality of monitoring components in the control environment are prohibited from initiating failover of the database instance upon determination of failure of the database instance.

16. The system of claim 15, wherein the lease information is received from the primary instance replica only when data is synchronized between the primary instance replica and the secondary instance replica, and a current lease exists for the database instance.

17. The system of claim 15, wherein the database environment includes a plurality of database instances, and the instructions when executed further cause the processor to:

assign each of the plurality of monitoring components in the control environment to a portion of the plurality of database instances in the database environment in order to substantially evenly distribute workload across the plurality of monitoring components.

18. The system of claim 17, wherein the instructions when executed further cause the processor to:

assign each of the plurality of database instances in the database environment to one of a plurality of workload partitions,

wherein assigning each of the plurality of monitoring components in the control environment to a portion of the plurality of database instances comprises assigning each of the plurality of monitoring components to one of the plurality of workload partitions.

19. The system of claim 18, wherein the instructions when executed further cause the processor to:

when at least one of the plurality of monitoring components is unable to monitor a workload partition to which the at least one of the plurality of monitoring components is assigned, reparation the plurality of workload partitions and reassign a remaining group of the plurality of monitoring components to a repartitioned plurality of workload partitions.

20. The system of claim 15, wherein the instructions when executed further cause the processor to:

store an identifier for the monitoring component and information for the lease period to a first block storage mechanism for the primary instance replica, the first block storage mechanism causing the identifier and the information for the lease period to be synchronously stored to a second block storage mechanism for the secondary instance replica.

21. A non-transitory computer-readable storage medium storing instructions for monitoring database instances in a database environment from a control environment, the instructions when executed by a processor causing the processor to:

assign a monitoring component of a plurality of monitoring components in the control environment to a database instance in the database environment, the plurality of monitoring components grouped according to one of a plurality of data zones or a plurality of geographic locations, the database instance capable of being replicated to include at least a primary instance replica and a secondary instance replica, the monitoring component being assigned to at least one of a first data zone or a first geographical location that is different from a second data zone or a second geographical location of at least one of the primary instance replica or the secondary instance replica in the database environment when the database instance is replicated;

cause the monitoring component to send a communication to the database instance, the communication being sent to at least the primary instance replica when the database instance is replicated; and

in response to receiving lease information from the database instance, monitor at least a status of the database instance using the monitoring component during a lease period associated with the lease information,

wherein, during the lease period, other monitoring components of the plurality of monitoring components in the control environment are prohibited from initiating failover of the database instance upon determination of failure of the database instance.

22. The non-transitory computer-readable storage medium of claim 21, wherein the lease information is received from the primary instance replica only when data is synchronized between the primary instance replica and the secondary instance replica, and a current lease exists for the database instance.

23. The non-transitory computer-readable storage medium of claim 21, wherein the database environment includes a plurality of database instances, and the instructions when executed further cause the processor to:

assign each of the plurality of monitoring components in the control environment to a portion of the plurality of database instances in the database environment in order to substantially evenly distribute workload across the plurality of monitoring components.

24. The non-transitory computer-readable storage medium of claim 23, wherein the instructions when executed further cause the processor to:

assign each of the plurality of database instances in the database environment to one of a plurality of workload partitions,

wherein assigning each of the plurality of monitoring components in the control environment to a portion of the plurality of database instances comprises assigning each of the plurality of monitoring components to one of the plurality of workload partitions.