MONITORING OF REPLICATED DATA INSTANCES

• US 20110099146A1
• Filed: 10/26/2009
• Published: 04/28/2011
• Est. Priority Date: 10/26/2009
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Brazil, Silas M., McAlister, Grant Alexander MacDonald, Hunter, Barry B. JR., Sivasubramanian, Swaminathan
• Granted Patent: US 8,676,753 B2
• Time in Patent Office: Days
• US Class Current: 707/634
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/203 using migration
• CPC Class Codes: G06F 11/2035 without idle spare hardware
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 11/2076 Synchronous techniques
• CPC Class Codes: G06F 11/2082 Data synchronisation
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3055 Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction by redundancy in operation G06F11/14; error detection or correction by redundancy in hardware G06F11/16
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route

Abstract

Replicated instances in a database environment provide for automatic failover and recovery. A monitoring component can obtain a lease enabling the component to periodically communicate with, and monitor, one or more data instances in the data environment, where the data instance can be a replicated instance including a primary and a secondary replica. For a large number of instances, the data environment can be partitioned such that each monitoring component can be assigned a partition of the workload. In the event of a failure of a monitoring component, the instances can be repartitioned and the remaining monitoring components can be assigned to the new partitions to substantially evenly distribute the workload.

Claims

1. A computer-implemented method of monitoring replicated instances for a relational database instance from a control environment, comprising:

under control of one or more computer systems configured with executable instructions,

assigning each of a plurality of replicated database instances in a database environment to one of a plurality of workload partitions;

assigning one of a plurality of monitoring components in the control environment to each of the plurality of workload partitions; and

for each replicated instance in a partition;

causing the assigned monitoring component to send a communication to a host manager for a primary instance replica of the replicated instance;

if data for the primary instance replica is synchronized with data for a secondary instance replica for the replicated instance, receiving to the assigned monitoring component a lease for the replicated instance, the lease specifying at least a lease period during which the assigned monitoring component will be able to monitor the replicated instance; and

in response to receiving the lease to the assigned monitoring component, monitoring at least a status of the replicated instance during the lease period using the assigned monitoring component.

2. The computer-implemented method of claim 1, wherein the assigned monitoring component is further able to receive a lease for a provisioned replicated instance only when a current lease exists for the replicated instance.

3. The computer-implemented method of claim 1, further comprising:

storing an identifier for the assigned monitoring component and information for the lease period to a block storage mechanism for the primary instance replica, the block storage mechanism causing the identifier and the information for the lease period to be synchronously stored to a block storage mechanism for the secondary instance replica.

4. The computer-implemented method of claim 1, wherein the monitoring component and at least one of the primary instance replica and the secondary instance replica are located in different data zones or different geographical locations.

5. A computer-implemented method of monitoring database instances in a database environment from a control environment, comprising:

under control of one or more computer systems configured with executable instructions,

assigning a monitoring component in the control environment to a database instance in at least one of a different data zone or a different geographical location in the database environment, the database instance capable of being a replicated instance including at least a primary instance replica and a secondary instance replica, the assigned monitoring component being in at least one of a different data zone and a different geographical location from at least one of the primary instance replica and the secondary instance replica when the database instance is a replicated instance;

causing the monitoring component to send a communication to the database instance, the communication being sent to at least the primary instance replica if the database instance is a replicated instance; and

in response to receiving lease information from the database instance, monitoring at least a status of the database instance using the monitoring component during a lease period of the lease.

6. The computer-implemented method of claim 5, wherein the lease information is received from the primary instance replica of a replicated instance only when data is synchronized between the primary instance replica and the secondary instance replica for the replicated instance.

7. The computer-implemented method of claim 5, wherein the lease information is received from the primary instance replica of a provisioned replicated instance only when a current lease exists for the provisioned replicated instance.

8. The computer-implemented method of claim 5, wherein the control environment includes a plurality of monitoring components and the database environment includes a plurality of database instances, further comprising:

assigning each of the plurality of monitoring components in the control environment to a portion of the database instances in the database environment in order to substantially evenly distribute workload across the plurality of monitoring components.

9. The computer-implemented method of claim 8, further comprising:

assigning each of the database instances in the database environment to one of a plurality of workload partitions,

wherein assigning each of the plurality of monitoring components in the control environment to a portion of the database instances comprises assigning each monitoring component to one of the plurality of workload partitions.

10. The computer-implemented method of claim 9, further comprising:

when a monitoring component is unable to monitor the assigned workload partition, repartitioning the database instances and reassigning a remaining group of the monitoring components to the partitions after repartitioning.

11. The computer-implemented method of claim 9, further comprising:

causing heartbeat messages to be periodically sent between the monitoring components in order to determine when a monitoring component is unable to monitor the assigned workload partition.

12. The computer-implemented method of claim 5, further comprising:

when the assigned monitoring component determines that the primary instance replica is unavailable for a replicated instance, causing the secondary instance replica to failover to a new primary instance replica for the replicated instance.

13. The computer-implemented method of claim 5, further comprising:

storing an identifier for the assigned monitoring component and information for the lease period to a block storage mechanism for the primary instance replica of a replicated instance, the block storage mechanism causing the identifier and the information for the lease period to be synchronously stored to a block storage mechanism for the secondary instance replica.

14. The computer-implemented method of claim 13, wherein the identifier is a random long identifier.

15. The computer-implemented method of claim 5, wherein the first and second instance replicas for a replicated instance are provisioned in a single data zone, in separate data zones at separate geographical locations, in a single data zone across multiple geographical locations, or across multiple data zones in a single geographical region, and

wherein the monitoring component is located in a third data zone or geographical location, or in a data zone or geographical location with one of the first and second instance replicas.

16. The computer-implemented method of claim 5, further comprising:

storing state information and a data generation identifier for the first and second instance replicas for a replicated instance, a monitoring component identifier, and lease period information in memory for a monitoring component in the control environment.

17. A system for monitoring database instances in a database environment from a control environment, comprising:

a processor; and

a memory device including instructions that, when executed by the processor, cause the processor to;

assign a monitoring component in the control environment to a database instance in at least one of a different data zone or a different geographical location in the database environment, the database instance capable of being a replicated instance including at least a primary instance replica and a secondary instance replica, the assigned monitoring component being in at least one of a different data zone and a different geographical location from at least one of the primary instance replica and the secondary instance replica when the database instance is a replicated instance;

cause the monitoring component to send a communication to the database instance, the communication being sent to at least the primary instance replica if the database instance is a replicated instance; and

in response to receiving lease information from the database instance, monitor at least a status of the database instance using the monitoring component during a lease period of the lease.

18. The system of claim 17, wherein the lease information is received from the primary instance replica of a replicated instance only when data is synchronized between the primary instance replica and the secondary instance replica for the replicated instance, and a current lease exists for the provisioned replicated instance.

19. The system of claim 17, wherein the control environment includes a plurality of monitoring components and the database environment includes a plurality of database instances, wherein the instructions when executed further cause the processor to:

assign each of the plurality of monitoring components in the control environment to a portion of the database instances in the database environment in order to substantially evenly distribute workload across the plurality of monitoring components.

20. The system of claim 19, wherein the instructions when executed further cause the processor to:

assign each of the database instances in the database environment to one of a plurality of workload partitions,

wherein assigning each of the plurality of monitoring components in the control environment to a portion of the database instances comprises assigning each monitoring component to one of the plurality of workload partitions.

21. The system of claim 20, wherein the instructions when executed further cause the processor to:

when a monitoring component is unable to monitor the assigned workload partition, repartition the database instances and reassign a remaining group of the monitoring components to the partitions after repartitioning.

22. The system of claim 17, wherein the instructions when executed further cause the processor to:

store an identifier for the assigned monitoring component and information for the lease period to a block storage mechanism for the primary instance replica of a replicated instance, the block storage mechanism causing the identifier and the information for the lease period to be synchronously stored to a block storage mechanism for the secondary instance replica.

23. The system of claim 17, wherein the first and second instance replicas for a replicated instance are provisioned in a single data zone, in separate data zones at separate geographical locations, in a single data zone across multiple geographical locations, or across multiple data zones in a single geographical region, and

wherein the monitoring component is located in a third data zone or geographical location, or in one of a first or second data zone or geographical location.

24. A computer-readable storage medium storing instructions for monitoring database instances in a database environment from a control environment, the instructions when executed by a processor causing the processor to:

assign a monitoring component in the control environment to a database instance in at least one of a different data zone or a different geographical location in the database environment, the database instance capable of being a replicated instance including at least a primary instance replica and a secondary instance replica, the assigned monitoring component being in at least one of a different data zone and a different geographical location from at least one of the primary instance replica and the secondary instance replica when the database instance is a replicated instance;

cause the monitoring component to send a communication to the database instance, the communication being sent to at least the primary instance replica if the database instance is a replicated instance; and

in response to receiving lease information from the database instance, monitor at least a status of the database instance using the monitoring component during a lease period of the lease.

25. The computer-readable storage medium of claim 24, wherein the lease information is received from the primary instance replica of a replicated instance only when data is synchronized between the primary instance replica and the secondary instance replica for the replicated instance, and a current lease exists for the provisioned replicated instance.

26. The computer-readable storage medium of claim 24, wherein the control environment includes a plurality of monitoring components and the database environment includes a plurality of database instances, wherein the instructions when executed further cause the processor to:

assign each of the plurality of monitoring components in the control environment to a portion of the database instances in the database environment in order to substantially evenly distribute workload across the plurality of monitoring components.

27. The computer-readable storage medium of claim 26, wherein the instructions when executed further cause the processor to:

assign each of the database instances in the database environment to one of a plurality of workload partitions,

wherein assigning each of the plurality of monitoring components in the control environment to a portion of the database instances comprises assigning each monitoring component to one of the plurality of workload partitions.