Monitoring of replicated data instances

• US 9,806,978 B2
• Filed: 03/17/2014
• Issued: 10/31/2017
• Est. Priority Date: 10/26/2009
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: McAlister, Grant Alexander MacDonald, Sivasubramanian, Swaminathan, Hunter, Jr., Barry B., Brazil, Silas M.
• Primary Examiner(s): OBERLY, VAN HONG
• Application Number:
  US14/216,521
• Publication Number:
  US 20140201363A1
• Time in Patent Office: 1,324 Days
• Field of Search: 707634, 707687
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/203 using migration
• CPC Class Codes: G06F 11/2035 without idle spare hardware
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 11/2076 Synchronous techniques
• CPC Class Codes: G06F 11/2082 Data synchronisation
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3055 Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction by redundancy in operation G06F11/14; error detection or correction by redundancy in hardware G06F11/16
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route

Abstract

Replicated instances in a database environment provide for automatic failover and recovery. A monitoring component can obtain a lease enabling the component to periodically communicate with, and monitor, one or more data instances in the data environment, where the data instance can be a replicated instance including a primary and a secondary replica. For a large number of instances, the data environment can be partitioned such that each monitoring component can be assigned a partition of the workload. In the event of a failure of a monitoring component, the instances can be repartitioned and the remaining monitoring components can be assigned to the new partitions to substantially evenly distribute the workload.

Claims

1. A system for monitoring provisioned data stores, comprising:

a processor; and

a memory device including instructions that, upon being executed by the processor, cause the system to;

allocate one or more monitoring components of a plurality of monitoring components to each data zone of a plurality of data zones;

assign, for each data zone of the plurality of data zones, each monitoring component, allocated to the data zone, to one or more respective data stores associated with the data zone;

determine, for at least one data zone of the plurality of data zones, whether a respective heartbeat message is received from each monitoring component allocated to the at least one data zone, wherein the determination is made by at least one of the monitoring components allocated to the at least one data zone;

determine, within a specified heartbeat failure time, that a first respective heartbeat message is not received from at least one first monitoring component; and

reassign at least some of the plurality of monitoring components such that there is at least one second monitoring component reassigned to the one or more data stores previously assigned to the at least one first monitoring component, wherein a second respective heartbeat message was received from the second monitoring component within the specified heartbeat failure time.

2. The system of claim 1, wherein the instructions, that cause the system to reassign the at least some of the plurality of monitoring components, include causing the system to:

cause the at least one second monitoring component to obtain a respective lease for each of the one or more data stores previously assigned to the at least one first monitoring component, each lease indicating a respective period of time during which the at least one second monitoring component is authorized to initiate failover of one of the data stores,

wherein no other monitoring component is authorized to initiate failover of the one of the data stores during the period of time.

3. The system of claim 1, wherein multiple monitoring components are allocated to the at least one data zone, multiple data stores are associated with the at least one data zone, and the instructions, that cause the system to reassign the at least some of the plurality of monitoring components, include causing the system to:

reassign each of the multiple monitoring components, whose respective heartbeat message was received within the specified heartbeat failure time, to a substantially equal portion of the multiple data stores to which the at least one first monitoring component was previously assigned.

4. The system of claim 1, wherein multiple monitoring components are allocated to the at least one data zone, multiple data stores are associated with the at least one data zone, and the instructions, that cause the system to reassign the at least some of the plurality of monitoring components, include causing the system to:

apportion the multiple data stores by substantially equal portions, a number of the substantially equal portions equal to a number of the multiple monitoring components whose respective heartbeat message was received within the specified heartbeat failure time; and

reassign each of the multiple monitoring components, whose respective heartbeat message was received within the specified heartbeat failure time, to one of the substantially equal portions.

5. A computer-implemented method for monitoring provisioned data stores, comprising:

under control of one or more computer systems configured with executable instructions,

assigning one or more monitoring components of a plurality of monitoring components to one or more respective data stores associated with each data zone of a plurality of data zones;

determining, for at least one data zone, a respective availability of each monitoring component assigned to the one or more data stores associated with the at least one data zone, wherein the determination is made by at least one of the monitoring components assigned to the at least one data zone; and

determining that at least one first monitoring component is unavailable; and

reassigning at least one second monitoring component to the one or more data stores previously assigned to the at least one first monitoring component, wherein the at least one second monitoring component is determined to be available.

6. The computer-implemented method of claim 5, wherein determining the respective availability of each monitoring component includes:

receiving a respective heartbeat message from each monitoring component within a first specified period of time.

7. The computer-implemented method of claim 6, wherein determining that the at least one first monitoring component is unavailable includes:

sending, to the at least one first monitoring component, a request for the availability of the at least one first monitoring component after the first specified period of time; and

receiving no response to the request after a second specified period of time.

8. The computer-implemented method of claim 5, wherein reassigning the at least one second monitoring component to the one or more data stores includes:

cause the at least one second monitoring component to obtain a respective lease for each of the one or more data stores, each lease indicating a respective period of time during which the at least one second monitoring component is authorized to initiate failover of one of the data stores,

wherein no other monitoring component is authorized to initiate failover of the one of the data stores during the respective period of time.

9. The computer-implemented method of claim 8, wherein at least one data store is a replicated data store, the replicated data store includes at least a primary instance replica and a secondary instance replica, and the lease for the primary instance replica can only be obtained upon data being synchronized between the primary instance replica and the secondary instance replica.

10. The computer-implemented method of claim 8, wherein the lease for at least one data store can only be obtained upon a current lease existing for the at least one data store.

11. The computer-implemented method of claim 5, wherein multiple data stores are associated with the at least one zone, multiple monitoring components are assigned to the multiple data stores, and reassigning the at least one second monitoring component includes:

reassigning each of the multiple monitoring components, that has been determined to be available, to a substantially equal portion of the multiple data stores to which the at least one first monitoring component was previously assigned.

12. The computer-implemented method of claim 5, wherein multiple data stores are associated with the at least one zone, multiple monitoring components are assigned to the multiple data stores, and reassigning the at least one second monitoring component includes:

apportioning the multiple data stores by substantially equal portions, a number of the substantially equal portions equal to a number of the multiple monitoring components that have been determined to be available; and

reassigning each of the multiple monitoring components, that has been determined to be available, to one of the substantially equal portions.

13. The computer-implemented method of claim 5, wherein at least one additional monitoring component becomes available for the at least one zone, and the method further comprises:

apportioning, by substantially equal portions, the one or more data stores associated with the at least one data zone, a number of the substantially equal portions equal to a number of the one or more monitoring components that have been determined to be available and a number of the at least one additional monitoring component; and

reassigning each of the one or more monitoring components, that has been determined to be available, and the at least one additional monitoring component, to one of the substantially equal portions.

14. The computer-implemented method of claim 5, wherein at least one data store is a replicated data store, the replicated data store includes at least a primary instance replica and a secondary instance replica, and at least one of:

the monitoring component assigned to the primary instance replica is associated with a different data zone than the primary instance replica;

or

the monitoring component assigned to the secondary instance replica is associated with the different data zone than the secondary instance replica.

15. The computer-implemented method of claim 14, further comprising:

causing the secondary instance replica to failover to a new primary instance replica in response to determining that the primary instance replica is unavailable.

16. A non-transitory computer-readable medium including instructions for monitoring provisioned data stores, the instructions, upon being executed by a processor, causes the processor to:

assign one or more monitoring components of a plurality of monitoring components to one or more respective data stores associated with each data zone of a plurality of data zones;

determine, for at least one data zone, a respective availability of each monitoring component assigned to the one or more data stores associated with the at least one data zone, wherein the determination is made by at least one of the monitoring components assigned to the at least one data zone;

determine that at least one first monitoring component is unavailable; and

reassign at least one second monitoring component to the one or more data stores previously assigned to the at least one first monitoring component, wherein the at least one second monitoring component is determined to be available.

17. The non-transitory computer-readable medium of claim 16, wherein the instructions, that cause the processor to reassign the at least one second monitoring component, include causing the processor to:

cause the at least one second monitoring component to obtain a respective lease for each of the one or more data stores previously assigned to the at least one first monitoring component, each lease indicating a respective period of time during which the at least one second monitoring component is authorized to initiate failover of one of the data stores,

wherein no other monitoring component is authorized to initiate failover of the one of the data stores during the period of time.

18. The non-transitory computer-readable medium of claim 17, wherein at least one data store is a replicated data store, the replicated data store includes at least a primary instance replica and a secondary instance replica, and the lease for the primary instance replica can only be obtained based upon one of:

data being synchronized between the primary instance replica and the secondary instance replica;

or

a current lease existing for the at least one data store.

19. The non-transitory computer-readable medium of claim 16, wherein at least one data store is a replicated data store, the replicated data store includes at least a primary instance replica and a secondary instance replica, and at least one of:

the monitoring component assigned to the primary instance replica is associated with a different data zone than the primary instance replica;

or

the monitoring component assigned to the secondary instance replica is associated with the different data zone than the secondary instance replica.

20. The non-transitory computer-readable medium of claim 19, wherein the instructions, upon being executed by the processor, further cause the processor to:

cause the secondary instance replica to failover to a new primary instance replica in response to determining that the primary instance replica is unavailable.