Monitoring and automated recovery of data instances

• US 8,631,283 B1
• Filed: 11/10/2011
• Issued: 01/14/2014
• Est. Priority Date: 03/31/2009
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Sivasubramanian, Swaminathan, McAlister, Grant Alexander MacDonald, Hunter, Barry B. Jr., Pol, Parikshit S.
• Primary Examiner(s): Maskulinski, Michael
• Assistant Examiner(s): MILES, NEIL D
• Application Number:
  US13/294,099
• Time in Patent Office: 796 Days
• Field of Search: 714/47.1
• US Class Current: 714/47.1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 2209/505 Clust
• CPC Class Codes: G06F 9/5061 Partitioning or combining of resources

Abstract

The monitoring and recovery of data instances, data stores, and other such components in a data environment can be performed automatically using a separate control environment. A monitoring component of the control plane can include a set of event processors for monitoring a workload of the data environment, where an event processor detecting a problem in the data plane can cause a recovery workflow to be generated in order to recover from the detected problem. The event processors can communicate with each other such that if one of the event processors becomes unavailable, the other event processors in a set are able to automatically redistribute responsibility for the workload.

Claims

1. A computer-implemented method of recovering from a failure in a data environment, comprising:

under control of one or more computer systems configured with executable instructions,

monitoring a plurality of host managers by a set of event processors in a control environment, the plurality of host managers each being associated with an identifier from a range of consecutively ordered identifiers and responsible for monitoring a status of at least one data instance in a data environment, wherein a respective first substantially equivalent portion of the range of consecutively ordered identifiers is allocated to each event processor of the set of event processors;

causing a heartbeat message to be sent from each event processor in an active state to each other event processor in the set to indicate that the event processor sending the heartbeat message is in the active state;

identifying an event processor of the set being in an inactive state based at least in part upon the heartbeat message not being received from the event processor in the inactive state; and

reallocating the range of identifiers to the event processors in the active state from which heartbeat messages were received, wherein a respective second substantially equivalent portion of the range of identifiers is reallocated to each of the event processors in the active state.

2. The computer-implemented method of claim 1, further comprising:

sending a status request, from an event processor of the set, to at least one host manager;

analyzing a response received from at least one of the host managers to the event processor to determine whether a problem exists with one of the host managers or one of the data instances; and

causing at least one task of a recovery workflow to be executed when the problem is determined to exist.

3. The computer-implemented method of claim 2, wherein:

when the problem is a failure of a data instance, the recovery workflow includes rebooting the data instance or re-provisioning another data instance.

4. The computer-implemented method of claim 2, wherein:

the problem is selected from a group of problems including at least one of failure of a data instance, a failure of a host device, a network outage, a data center outage, a data store error, an input/output (I/O) error, and a failure of a host manager.

5. The computer-implemented method of claim 2, further comprising:

when the problem is determined to involve a number of concurrent failures exceeding a specified threshold, determining a recovery workflow includes at least one of contacting an operator before the determined recovery workflow is executed or performing a staged recovery.

6. The computer-implemented method of claim 2, further comprising:

determining whether a problem exists when a response is not received from one of the plurality of host managers.

7. The computer-implemented method of claim 2, further comprising:

resending the status request at least once for a host manager when a response is not received before determining whether a problem exists.

8. A system for recovering from a failure in a data environment, comprising:

at least one event processor; and

memory including instructions that, when executed by the at least one event processor, cause the system to;

monitor, by a set of event processors in a control environment, a plurality of host managers each having an identifier over a range of consecutively ordered identifiers and being responsible for monitoring a status of at least one data instance in a data environment, wherein a respective first substantially equivalent portion of the range of consecutively ordered identifiers is allocated to each event processor of the set;

cause a heartbeat message to be sent from each event processor in an active state to each other event processor in the set to indicate that the event processor sending the heartbeat message is in the active state;

identify an event processor of the set being in an inactive state in response to the heartbeat message not being received from the event processor in the inactive state; and

reallocate the range of identifiers to the event processors in the active state from which status messages were received, wherein a respective second substantially equivalent consecutively ordered portion of the range of identifiers is reallocated to each of the event processors in the active state.

9. The system of claim 8, wherein the instructions, when executed by the at least one processor, further cause the system to:

send a status request, from an event processor of the set, to at least one host managers having an identifier in a portion of the range of identifiers allocated to the event processor;

analyze a response received from at least one of the host manager to the event processor to determine whether a problem exists with one of the host manager or one of the data instances; and

cause at least one task of a recovery workflow to be executed when a problem is determined to exist.

10. The system of claim 9, wherein the instructions, when executed by the at least one processor, further cause the system to:

determine whether a problem exists when a response is not received from one of the host managers.

11. A computer-implemented method of monitoring components in a data environment, comprising:

under control of one or more computer systems configured with executable instructions,

allocating a plurality of components in a data environment to be monitored by a set of event processors in a control environment, the plurality of components each having an identifier over a range of consecutively ordered identifiers, each event processor being allocated a respective first substantially equivalent portion of the range of identifiers for monitoring and operable to periodically send status messages to each component in the respective portion of the range of identifiers allocated to a respective event processor;

causing a status message to be sent from each event processor in an active state to be received by other event processors in the set in order to indicate that the event processor sending the status message is in the active state; and

upon identifying an event processor as being in an inactive state, reallocating the range of identifiers to the event processors in the active state from which status messages were received, wherein a respective second substantially equivalent consecutively ordered portion of the range of identifiers is reallocated to each of the event processors in the active state.

12. The computer-implemented method of claim 11, wherein:

each event processor is further operable to store information for a monitored component to a job queue when a problem is detected, the information being used to determine a recovery workflow to be executed for the monitored component.

13. The computer-implemented method of claim 11, further comprising:

storing information for the event processor from which a status message was not received to a job queue; and

using the information to generate a workflow to restart the event processor or add a new event processor to the set of event processors.

14. The computer-implemented method of claim 13, further comprising:

when a newly started event processor is activated, reallocating a respective substantially equivalent consecutively ordered portion of the range of identifiers to each of the set of event processors.

15. The computer-implemented method of claim 11, further comprising:

running each event processor at a portion of a processing capacity, wherein each event processor is able to accept a portion of a workload from an unavailable event processor without negatively impacting performance.

16. A computer-implemented method of monitoring components in a data environment, comprising:

under control of one or more computer systems configured with executable instructions,

allocating a plurality of components in a data environment to be monitored by a set of event processors in a control environment, the plurality of components each having an identifier over a range of identifiers, each event processor being allocated a respective first substantially equivalent consecutively ordered portion of the range of identifiers for monitoring;

causing a status message to be sent from each event processor in an active state to be received by other event processors in the set to indicate that the event processor sending the status message is in the active state;

storing information for an event processor from which a status message was not received to a job queue in a control environment;

using the stored information to generate a workflow to restart the event processor from which the status message was not received or add a new event processor to the set of event processors; and

reallocating the range of identifiers to each of the set of event processors when a newly started event processor is activated, wherein a respective second substantially equivalent consecutively ordered portion of the range of identifiers is reallocated to each of the event processors in the active state.

17. The computer-implemented method of claim 16, further comprising:

sorting the identifiers and allocating the sorted identifiers substantially uniformly across the set of event processors.

18. The computer-implemented method of claim 16, further comprising:

running each event processor at a portion of a processing capacity, wherein each event processor is able to accept a portion of a workload from an unavailable event processor without negatively impacting performance.

19. A computer-implemented method of monitoring components in a data environment, comprising:

under control of one or more computer systems configured with executable instructions,

allocating a plurality of components in a data environment to be monitored by a set of event processors in a control environment, the plurality of components each having an identifier over a range of identifiers, and each event processor being allocated a respective first substantially equivalent consecutively ordered portion of the range of identifiers for monitoring;

sorting the identifiers and allocating the sorted identifiers substantially uniformly across the set of event processors;

causing a status message to be sent from each event processor in an active state to be received by other event processors in the set to indicate that the event processor sending the status message is in the active state; and

upon identifying an event processor as being in an inactive state, reallocating the range of identifiers to the event processors in the active state from which status messages were received, wherein a respective second substantially equivalent consecutively ordered portion of the range of identifiers is reallocated to each of the event processors in the active state.

20. The computer-implemented method of claim 19, further comprising:

running each event processor at a portion of a processing capacity, wherein each event processor is able to accept a portion of a workload from an unavailable event processor without negatively impacting performance.

21. The computer-implemented method of claim 19, further comprising:

storing information for the event processor from which a status message was not received to a job queue; and

using the information to generate a workflow to restart the event processor or add a new event processor to the set of event processors.