Predicting infrastructure failures in a data center for hosted service mitigation actions

• US 10,048,996 B1
• Filed: 09/29/2015
• Issued: 08/14/2018
• Est. Priority Date: 09/29/2015
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Bell, Charles H., Minarik, Joseph Thomas
• Primary Examiner(s): Bonzo, Bryce P
• Assistant Examiner(s): Arcos, Jeison C
• Application Number:
  US14/869,910
• Time in Patent Office: 1,050 Days
• CPC Class Codes: G06F 11/004 Error avoidance G06F11/07 and subgroups take precedence
• CPC Class Codes: G06F 11/008 Reliability or availability analysis
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/076 by exceeding a count or rate limit, e.g. word- or bit count limit
• CPC Class Codes: G06F 11/0772 Means for error signaling, e.g. using interrupts, exception flags, dedicated error registers
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3058 Monitoring arrangements for monitoring environmental properties or parameters of the computing system or of the computing system component, e.g. monitoring of power, currents, temperature, humidity, position, vibrations thermal management in cooling arrangements of a computing system G06F1/206
• CPC Class Codes: G06F 11/3442 for planning or managing the needed capacity
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis

Abstract

A data center may predict infrastructure failures in order to perform mitigation actions at services hosted at the data center. Operational metrics for different infrastructure systems of a data center may be collected and analyzed to generate failure models. The failure models may be evaluated to predict infrastructure failure events. The predicted infrastructure failure events may be programmatically provided to the services. The services may evaluate the prediction and select mitigation actions to perform. For data centers implemented as part of a provider network with services hosted across multiple data centers, mitigation actions may be performed at multiple data centers for a service in response to a predicted failure event at one data center.

Claims

1. A data center, comprising:

one or computing devices comprising one or more respective hardware processors and memory and configured to implement;

a plurality of infrastructure management systems, configured to manage a plurality of different types of infrastructure resources for the data center;

a plurality of services hosted using the infrastructure resources at the data center, wherein each of the plurality of services provides a different network-based service to clients remote to the data center; and

an infrastructure monitor, configured to;

collect operational metrics from the infrastructure management systems for the plurality of different types of infrastructure resources;

generate one or more failure models from the operational metrics;

evaluate the one or more failure models to predict whether an infrastructure failure event is to occur at a future time; and

responsive to a prediction of an occurrence of the infrastructure failure event, send respective notifications of the predicted infrastructure failure event via a programmatic interface to at least some of the plurality of services;

each respective service of the plurality of services, configured to;

receive the respective notification of the predicted infrastructure failure event via the programmatic interface;

determine a respective mitigation action for the respective service; and

automatically perform the respective mitigation action responsive to receipt of the respective notification of the predicted infrastructure failure event to mitigate one or more consequences associated with occurrence of the predicted infrastructure failure event.

2. The data center of claim 1,

wherein the infrastructure monitor is further configured to send a prior notification including a probability of occurrence of the predicted failure event via the programmatic interface to at least one of the services before the notification is sent;

wherein the at least one service is further configured to;

evaluate the probability of occurrence of the predicted infrastructure failure event; and

based on the evaluation, delay performing the mitigation action.

3. The data center of claim 1, wherein the respective notifications of the predicted infrastructure failure event indicate a probability of occurrence for the predicted infrastructure failure event, and wherein at least one of the services is further configured to:

evaluate the probability of occurrence for the predicted infrastructure failure event with respect to one or more probability thresholds to select, from among a plurality of mitigation actions, the mitigation action to perform.

4. The data center of claim 1, wherein the data center is one of a plurality of data centers implementing a provider network, wherein the plurality of services hosted at the data center are hosted across the plurality of data centers of the provider network, wherein the respective mitigation action is performed at the data center and at least one other data center hosting the at least one service.

5. A method, comprising:

performing, by one or more computing devices;

receiving a stream of operational metrics from a plurality of different infrastructure systems of a data center, wherein the data center hosts a plurality of different services using the infrastructure resources, wherein each of the plurality of services provides a different network-based service to clients remote to the data center;

generating one or more failure models from the stream of operational metrics for the plurality of different types of infrastructure resources;

evaluating the one or more failure models to predict whether an infrastructure failure event is to occur at a future time; and

responsive to a prediction of an occurrence of the infrastructure failure event, reporting the prediction of the infrastructure failure event via a programmatic interface to at least some of the services hosted at the data center for determination of respective mitigation actions for the respective services to take to mitigate one or more consequences associated with occurrence of the predicted infrastructure failure event.

6. The method of claim 5, further comprising:

determining a respective mitigation action to perform for the predicted infrastructure failure event, wherein the prediction of the infrastructure failure event reported to at least one of the plurality of services includes the respective mitigation action.

7. The method of claim 5, further comprising providing a notification of the predicted infrastructure failure event to one or more clients of the at least one service.

8. The method of claim 5, wherein reporting the prediction of the infrastructure failure event comprises identifying the at least one service as subscribed to a type of infrastructure failure event that includes the predicted failure event.

9. The method of claim 5, wherein the predicted infrastructure failure event indicates:

an environmental failure;

a power failure;

a networking failure;

or

a hardware failure.

10. The method of claim 5, further comprising:

in response to receiving the prediction of the infrastructure failure event via the programmatic interface;

evaluating, at at least one service, the prediction of the infrastructure failure event with respect to one or more service mitigation actions; and

based, at least in part, on the evaluation, selecting the mitigation action from the one or more service mitigation actions for the predicted infrastructure failure event.

11. The method of claim of claim 10, wherein the receiving the stream of operational metrics, the generating the one or more failure models, the evaluating the failure models, and the reporting the prediction are performed by an infrastructure monitor for the data center and wherein the method further comprises:

sending, by the at least one service, response data indicating the selected mitigation action; and

updating, by the infrastructure monitor, the one or more failure models according to the response data.

12. The method of claim 10, further comprising:

upon determining that the predicted infrastructure failure event has not occurred, undoing the selected mitigation action.

13. The method of claim 10, wherein the data center is one of a plurality of data centers implementing a provider network, wherein the plurality of services hosted at the data center are hosted across the plurality of data centers of the provider network, wherein the selected mitigation action is performed at the data center and at least one other data center hosting the at least one service.

14. A non-transitory, computer-readable storage medium, storing program instructions that when executed by one or more computing devices cause the one or more computing devices to implement:

receiving, at a service hosted in one or more data centers using infrastructure resources of the one or more data centers, a prediction of an infrastructure failure event to occur at a future time, via a programmatic interface, wherein the service provides a network-based service to clients remote to the one or more data centers;

evaluating, at the service, the prediction of the infrastructure failure event to determine whether to perform a mitigation action;

based, at least in part, on the evaluation, selecting by the service, the mitigation action for the predicted infrastructure failure event; and

performing, at the service, the selected mitigation action to mitigate one or more consequences associated with occurrence of the predicted infrastructure failure event.

15. The non-transitory, computer-readable storage medium of claim 14, wherein the predicted infrastructure failure event indicates a probability of occurrence for the predicted infrastructure failure event and wherein, in evaluating, at the service, the prediction of the infrastructure failure event, the program instructions cause the one or more computing devices to implement:

comparing the probability of occurrence to a plurality of probability thresholds, wherein the mitigation action is selected according to a determination that the probability of occurrence is exceeds a probability threshold corresponding to the migration action.

16. The non-transitory, computer-readable storage medium of claim 14, wherein the program instructions cause the one or more computing devices to further implement providing a notification of the predicated failure event to one or more clients of the service.

17. The non-transitory, computer-readable storage medium of claim 16, wherein the notification to the one or more clients of the service is provided via a graphical user interface.

18. The non-transitory, computer-readable storage medium of claim 14, wherein the program instructions cause the one or more computing devices to further implement:

prior to receiving the prediction of the infrastructure failure event;

receiving another prediction of the same infrastructure failure event including a probability of occurrence of the infrastructure failure event,

wherein the selected mitigation action is not performed in response to receiving the other prediction.

19. The non-transitory, computer-readable storage medium of claim 14, wherein the program instructions cause the one or more computing devices to further implement:

prior to receiving the prediction of the infrastructure failure;

receiving, at an infrastructure monitor, a stream of operational metrics from a plurality of different infrastructure systems of the one or more data centers, wherein the one or more data centers host a plurality of different services including the service;

generating, by the infrastructure monitor, one or more failure models from the stream of operational metrics;

evaluating, by the infrastructure monitor, the failure models to predict the infrastructure failure event; and

reporting, by the infrastructure monitor, the prediction of the infrastructure failure event via the programmatic interface to the service.

20. The non-transitory, computer-readable storage medium of claim 19, wherein the program instructions cause the one or more computing devices to further implement sending, by the service, response data indicating the selected mitigation action to the infrastructure monitor.