Monitoring and automated recovery of data instances

• US 8,060,792 B2
• Filed: 03/31/2009
• Issued: 11/15/2011
• Est. Priority Date: 03/31/2009
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Hunter, Barry B. Jr., Pol, Parikshit S., Sivasubramanian, Swaminathan, McAlister, Grant Alexander MacDonald
• Primary Examiner(s): Maskulinski; Michael
• Assistant Examiner(s): MILES, NEIL D
• Application Number:
  US12/415,987
• Publication Number:
  US 20100251002A1
• Time in Patent Office: 959 Days
• Field of Search: None
• US Class Current: 714/47.1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 2209/505 Clust
• CPC Class Codes: G06F 9/5061 Partitioning or combining of resources

Abstract

The monitoring and recovery of data instances, data stores, and other such components in a data environment can be performed automatically using a separate control environment. A monitoring component of the control plane can include a set of event processors for monitoring a workload of the data environment, where an event processor detecting a problem in the data plane can cause a recovery workflow to generated in order to recover from the detected problem. The event processors can communicate with each other such that if one of the event processors becomes unavailable, the other event processors in a set are able to automatically redistribute responsibility for the workload.

Claims

1. A computer-implemented method of monitoring components in a data environment, comprising:

under control of one or more computer systems configured with executable instructions,

determining a set of event processors in a control environment for monitoring a plurality of components in the data environment, the plurality of components each having an identifier over a range of identifiers;

allocating a portion of the range of identifiers to each of the set of event processors, each event processor being allocated a substantially equivalent portion of the range of identifiers for monitoring;

periodically sending a status message from each of the event processors to be received by the other event processors in the set indicating that the event processor sending the status message is active; and

in response to not receiving a status message from one of the event processors for at least a determined period of time, automatically reallocating the range of identifiers to the active event processors from which status messages were received,

wherein each active event processor receives a different substantially equivalent portion of the range of identifiers based on the number of active event processors, and wherein each event processor is operable to periodically send status messages to each component in the data environment having an identifier in the portion of the range of identifiers allocated to the event processor.

2. A computer-implemented method according to claim 1, wherein:

each event processor is further operable to store information for a monitored component to a job queue when a potential problem is detected, the information being used to determine a recovery workflow to be executed for the monitored component in the data environment.

3. A computer-implemented method according to claim 1, further comprising:

storing information for the event processor from which a status message was not received to a job queue in the control environment; and

using the information to generate a workflow to restart the event processor or add a new event processor to the set of event processors.

4. A computer-implemented method according to claim 3, further comprising:

causing the newly started event processor to send and receive heartbeats to other event processors in the set before activating the newly started event processor.

5. A computer-implemented method according to claim 3, further comprising:

when the newly started event processor is activated, reallocating the portion of the range of identifiers to each of the set of event processors.

6. A computer-implemented method according to claim 1, further comprising:

sorting the identifiers and allocating the sorted identifiers substantially uniformly across the set of event processors.

7. A computer-implemented method according to claim 1, further comprising:

running each event processor at a portion of a processing capacity, wherein each event processor is able to accept a portion of the workload from an unavailable event processor without negatively impacting performance.

8. A system for monitoring components in a data environment, comprising:

at least one processor; and

memory including instructions that, when executed by the at least one processor, cause the system to;

determine a set of event processors in a control environment for monitoring a plurality of components in the data environment, the plurality of components each having an identifier over a range of identifiers;

allocate a portion of the range of identifiers to each of the set of event processors, each event processor being allocated a substantially equivalent portion of the range of identifiers for monitoring;

periodically send a status message from each of the event processors to be received by the other event processors in the set indicating that the event processor sending the status message is active;

in response to not receiving a status message from one of the event processors for at least a determined period of time, automatically reallocate the range of identifiers to the active event processors from which status messages were received;

store information for the event processor from which a status message was not received to a job queue in the control environment; and

use the information to generate a workflow to restart the event processor or add a new event processor to the set of processors,

wherein each active event processor receives a different substantially equivalent portion of the range of identifiers based on the number of active event processors.

9. A system according to claim 8, wherein the instructions, when executed by the at least one processor, further cause the system to:

when the newly started event processor is activated, reallocate the portion of the range of identifiers to each of the set of event processors.

10. A system according to claim 8, wherein the instructions, when executed by the at least one processor, further cause the system to:

sort the identifiers and allocate the sorted identifiers substantially uniformly across the set of event processors.

11. A computer-implemented method of monitoring components in a data environment, comprising:

under control of one or more computer systems configured with executable instructions,

determining a set of event processors in a control environment for monitoring a plurality of components in the data environment, the plurality of components each having an identifier over a range of identifiers;

allocating a portion of the range of identifiers to each of the set of event processors, each event processor being allocated a substantially equivalent portion of the range of identifiers for monitoring;

periodically sending a status message from each of the event processors to be received by the other event processors in the set indicating that the event processor sending the status message is active;

in response to not receiving a status message from one of the event processors for at least a determined period of time, automatically reallocating the range of identifiers to the active event processors from which status messages were received;

storing information for the event processor from which a status message was not received to a job queue in the control environment; and

using the information to generate a workflow to restart the event processor or add a new event processor to the set of event processors,

wherein each active event processor receives a different substantially equivalent portion of the range of identifiers based on the number of active event processors.

12. The computer-implemented method of claim 11, further comprising:

causing the newly started event processor to send and receive heartbeats to other event processors in the set before activating the newly started event processor.

13. The computer-implemented method of claim 11, further comprising:

when the newly started event processor is activated, reallocating the portion of the range of identifiers to each of the set of event processors.

14. The computer-implemented method of claim 11, further comprising:

sorting the identifiers and allocating the sorted identifiers substantially uniformly across the set of event processors.

15. The computer-implemented method of claim 11, further comprising:

running each event processor at a portion of a processing capacity, wherein each event processor is able to accept a portion of the workload from an unavailable event processor without negatively impacting performance.

16. A computer-implemented method of monitoring components in a data environment, comprising:

under control of one or more computer systems configured with executable instructions,

determining a set of event processors in a control environment for monitoring a plurality of components in the data environment, the plurality of components each having an identifier over a range of identifiers;

allocating a portion of the range of identifiers to each of the set of event processors, each event processor being allocated a substantially equivalent portion of the range of identifiers for monitoring;

sorting the identifiers and allocating the sorted identifiers substantially uniformly across the set of event processors;

periodically sending a status message from each of the event processors to be received by the other event processors in the set indicating that the event processor sending the status message is active; and

in response to not receiving a status message from one of the event processors for at least a determined period of time, automatically reallocating the range of identifiers to the active event processors from which status messages were received,

wherein each active event processor receives a different substantially equivalent portion of the range of identifiers based on the number of active event processors.

17. A computer-implemented method of monitoring components in a data environment, comprising:

under control of one or more computer systems configured with executable instructions,

determining a set of event processors in a control environment for monitoring a plurality of components in the data environment, the plurality of components each having an identifier over a range of identifiers;

allocating a portion of the range of identifiers to each of the set of event processors, each event processor being allocated a substantially equivalent portion of the range of identifiers for monitoring;

running each event processor at a portion of a processing capacity, wherein each event processor is able to accept a portion of the workload from an unavailable event processor without negatively impacting performance;

periodically sending a status message from each of the event processors to be received by the other event processors in the set indicating that the event processor sending the status message is active; and

in response to not receiving a status message from one of the event processors for at least a determined period of time, automatically reallocating the range of identifiers to the active event processors from which status messages were received,

wherein each active event processor receives a different substantially equivalent portion of the range of identifiers based on the number of active event processors.

18. A system for monitoring components in a data environment, comprising:

at least one processor; and

memory including instructions that, when executed by the at least one processor, cause the system to;

determine a set of event processors in a control environment for monitoring a plurality of components in the data environment, the plurality of components each having an identifier over a range of identifiers;

allocate a portion of the range of identifiers to each of the set of event processors, each event processor being allocated a substantially equivalent portion of the range of identifiers for monitoring;

periodically send a status message from each of the event processors to be received by the other event processors in the set indicating that the event processor sending the status message is active;

in response to not receiving a status message from one of the event processors for at least a determined period of time, automatically reallocate the range of identifiers to the active event processors from which status messages were received; and

when a newly started event processor is activated, reallocate the portion of the range of identifiers to each of the set of event processors;

wherein each active event processor receives a different substantially equivalent portion of the range of identifiers based on the number of active event processors.

19. A system for monitoring components in a data environment, comprising:

at least one processor; and

memory including instructions that, when executed by the at least one processor, cause the system to;

determine a set of event processors in a control environment for monitoring a plurality of components in the data environment, the plurality of components each having an identifier over a range of identifiers;

allocate a portion of the range of identifiers to each of the set of event processors, each event processor being allocated a substantially equivalent portion of the range of identifiers for monitoring;

sort the identifiers and allocate the sorted identifiers substantially uniformly across the set of event processors;

periodically send a status message from each of the event processors to be received by the other event processors in the set indicating that the event processor sending the status message is active; and

in response to not receiving a status message from one of the event processors for at least a determined period of time, automatically reallocate the range of identifiers to the active event processors from which status messages were received,

wherein each active event processor receives a different substantially equivalent portion of the range of identifiers based on the number of active event processors.