Automated root cause analysis

• US 9,104,572 B1
• Filed: 02/11/2013
• Issued: 08/11/2015
• Est. Priority Date: 02/11/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Thompson, James Michael, Olson, Marc Stephen, Viswanathan, Ganesh, Hawks, Benjamin Arthur, Doshi, Bhavesh Anil
• Primary Examiner(s): Wilson, Yolanda L
• Application Number:
  US13/764,691
• Time in Patent Office: 911 Days
• Field of Search: 714/47.3, 714/26
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3055 Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction by redundancy in operation G06F11/14; error detection or correction by redundancy in hardware G06F11/16
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: H04L 41/0631 using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
• CPC Class Codes: H04L 41/142 using statistical or mathematical methods
• CPC Class Codes: H04L 41/145 involving simulating, designing, planning or modelling of a network
• CPC Class Codes: H04L 43/0817 by checking functioning
• CPC Class Codes: H04L 43/087 Jitter
• CPC Class Codes: H04L 43/0876 Network utilisation, e.g. volume of load or congestion level

Abstract

Various aspects of the performance of computing resources, such as storage volumes, are measured and used to train a probability model. The probability model is used in a query engine that is able to respond receive queries about a computing resource'"'"'s state. The queries may specify a state of the computing resource and provide a set of measurements of the computing resource'"'"'s performance. The query engine may use the probability model, which may be in the form of a contingency table, to provide information that indicates one or more most likely causes of the state.

Claims

1. A computer-implemented method, comprising:

under the control of one or more computer systems configured with executable instructions,

for a plurality of storage volumes in a distributed computing environment, obtaining performance metric values for each of a plurality of performance metrics;

analyzing the obtained performance metrics to determine a probability model that, for each storage volume health state of a plurality of storage volume health states, models probabilities that the volume health state is caused by corresponding issue;

obtaining, for a particular storage volume, a health state for the storage volume and a set of performance metric values for the particular storage volume;

using the determined probability model to determine, based at least in part on the obtained health state and set of performance metric values for the particular storage volume, an issue potentially causing the health state of the particular storage volume; and

provide information identifying the determined issue.

2. The computer-implemented method of claim 1, wherein at least some of the performance metrics are latency metrics for input/output communications between clients and corresponding data storage servers.

3. The computer-implemented method of claim 1, wherein communication between the clients and corresponding data storage servers comprises multiple processes and the plurality of performance metrics includes a performance metric for each of at least some of the plurality of performance metrics.

4. The computer-implemented method of claim 1, wherein using the determined probability model to determine the issue potentially causing the health state comprises:

for each performance metric value of at least two of the performance metric values, determining a probability that a storage volume having the health state is in a range of performance metric values that includes the performance metric value; and

using the determined probabilities to identify the issue.

5. The computer-implemented method of claim 1, wherein:

determining the issue potentially causing the health state includes determining a plurality of issues potentially causing the health state, the plurality of issues being ranked according to the probability model; and

the information identifying the determined issue identifies at least one other issue from the determined plurality of issues.

6. The computer-implemented method of claim 1, wherein using the determined probability model to determine the issue and providing the information are performed in response to a detected change in health state of the particular storage volume.

7. A computer-implemented method, comprising:

under the control of one or more computer systems configured with executable instructions, obtaining a probability model for computing resources, the probability model based at least in part on measurements of multiple processes involved in operation of the computing resources, wherein the computing resources are storage volumes;

for a particular computing resource, obtaining measurements taken during a performance degradation of the particular computing resource for at least some of the multiple processes; and

using the obtained measurements for the particular computing resource and the probability model to diagnose the performance degradation.

8. The computer-implemented method of claim 7, wherein the multiple processes involve multiple devices in a network supporting the computing resources.

9. The computer-implemented method of claim 7, wherein:

the computing resources communicate with other computing resources; and

the measurements of multiple processes include, for each computing resource of a plurality of the computing resources that communicates with another computing resource, latencies for different portions of a communications route between the computing resource and the other computing resource.

10. The computer-implemented method of claim 7, wherein:

the method further comprises;

obtaining the measurements;

using the measurements to generate a contingency table; and

the probability model is based at least in part on the contingency table.

11. The computer-implemented method of claim 7, wherein:

the method further comprises;

obtaining the measurements;

using the measurements to generate a logistic regression model; and

the probability model is based at least in part on the logistic regression model.

12. The computer-implemented method of claim 7, wherein:

the probability model associates each health state of a plurality of health states with a probability of being in the health state; and

the performance degradation corresponds to a change from a first health state of the plurality of health states to a second health state of the plurality of health states.

13. The computer-implemented method of claim 7, wherein:

the performance degradation corresponds to the particular computing resource being in a particular health state;

diagnosing the performance degradation includes, for each measurement of at least a plurality of the obtained measurements, determining a probability of being in the particular health state given a predetermined range, from a plurality of predetermined ranges, into which the measurement falls.

14. A system, comprising:

one or more processors; and

memory including executable instructions that, when executed by the one or more processors, cause the system to implement at least;

an interface configured to receive a query that specifies measurements for a computing resource, the measurements comprising a measurement for each of a plurality of processes involved in operation of a data storage volume; and

a query engine configured to;

use the measurements to determine, based at least in part on previously obtained measurements for the processes, a response to the query, the response including a probability model generated based at least in part on the previously obtained measurements; and

provide the determined response, wherein the determined response includes information indicating one or more potential causes of a performance degradation of the data storage volume.

15. The system of claim 14, wherein the system corresponds to a computing resource provider that provides use of the computing resource the system provides the interface to customers of the computing resource provider.

16. The system of claim 14, wherein the measurement for the data storage volume include at least one measurement that involves two separate device separated by a network.

17. The system of claim 14, wherein:

operation of the data storage volume involves communication with a computing resource;

at least a subset of the processes are part of the communication; and

one or more measurements of the processes that are part of the communication are timing measurements.

18. The system of claim 14, wherein using the measurements based at least in part on the previously obtained measurements for the processes includes using a contingency table generated based at least in part on the previously obtained measurements.

19. The system of claim 14, wherein the previously obtained measurements are from a fleet of computing resources of a same type as the data storage volume.

20. The system of claim 14, wherein the query specifies a health state of the computing resource from a plurality of predetermined health states.

21. One or more non-transitory computer-readable storage media having collectively stored thereon instructions that, when executed by one or more processors of a computer system, cause the computer system to:

obtain measurements of processes involved in the operation of a fleet of data storage volumes;

generate, based at least in part on the obtained measurements, a probability model that, for each process of at least a plurality of the processes, is usable to obtain probabilities as a function of at least a data storage volume health state and a measurement of the process;

use the generated probability model to provide information about data storage volumes; and

wherein the generated probability model includes information indicating one or more potential causes of a performance degradation of the data storage volume.

22. The one or more non-transitory computer-readable storage media of claim 21, wherein the probability model is a representation of conditional probabilities that is usable to determine a probability of being in a particular health state given a measurement of a process of the processes.

23. The one or more non-transitory computer-readable storage media of claim 21, wherein using the generated probability model to provide information includes:

receiving a query that specifies a health state of a particular data storage volume and plurality of measurements comprising a measurement for each of a plurality of processes involved in the particular data storage volume'"'"'s operation;

using the probability module to determine one or more potential causes of the health state for the particular data storage volume; and

providing information that is indicative of at least one of the determined one or more potential causes.

24. The one or more non-transitory computer-readable storage media of claim 21, wherein:

the computing resources include storage volumes logically attached to respective computer system instances; and

at least some of the processes are part of the computer system instances communicating with the storage volumes to perform input/output operations.

25. The one or more non-transitory computer-readable storage media of claim 21, wherein the obtained measurements include at least one of communication latencies or operational latencies.