Root cause detection and monitoring for storage systems

• US 10,282,245 B1
• Filed: 06/25/2015
• Issued: 05/07/2019
• Est. Priority Date: 06/25/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Viswanathan, Ganesh, Sasikumar, Vinayak, Pop, Artur, Chang, Shuai, Zeghers, Benjamin Ryan
• Primary Examiner(s): Wilson, Yolanda L
• Application Number:
  US14/751,028
• Time in Patent Office: 1,412 Days
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/0775 Content or structure details of the error report, e.g. specific table structure, specific error fields
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3072 where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 11/3485 for I/O devices
• CPC Class Codes: G06F 2201/81 Threshold
• CPC Class Codes: H04L 41/0636 based on a decision tree analysis
• CPC Class Codes: H04L 41/0681 Configuration of triggering conditions
• CPC Class Codes: H04L 41/0686 Additional information in the notification, e.g. enhancement of specific meta-data
• CPC Class Codes: H04L 41/142 using statistical or mathematical methods
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation
• CPC Class Codes: H04L 43/0817 by checking functioning
• CPC Class Codes: H04L 43/16 Threshold monitoring
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: Y02D 10/00 : Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

A storage system includes a monitoring service that identifies root causes of storage systems issues using relationships. The monitoring service can use thresholds associated with the relationships to detect the root causes. Relationships can be based on correlation relationships between the different levels of the storage system. In various embodiments, relationships can also be based on events that affect multiple storage volumes or on short-term events. Once a relationship is identified, a threshold for that relationship is generated or updated. The monitoring service can make that threshold accessible to other components of the monitoring service or an operator of the storage system to be used in detecting root causes.

Claims

1. A system comprising:

a host computing device configured to host one or more virtual computing device instances, the host computing device configured to transmit storage commands generated by the one or more virtual computing device instances via a communications network, the one or more virtual computing device instances executing on behalf of a client computing device;

a storage processing service, executed on one or more storage computing devices, the storage processing service configured to;

obtain a storage command request from a virtual computing device instance of the one or more virtual computing device instances;

process the storage command request to generate a storage command processing result associated with at least one storage volume, the storage volume associated with a storage computing device of the one or more storage computing devices;

collect storage command metric information based at least in part on the storage command processing result; and

a storage monitoring service, executed on one or more computing devices configured to;

obtain, from the storage processing service, the collected storage command metric information;

identify a relationship, among one or more storage volumes, based on the collected metric information, wherein at least one storage volume of the one or more storage volumes is executing on one of the one or more storage computing devices, wherein identifying the relationship further comprises identifying a first storage system event that affects multiple storage volumes;

determine that the identified relationship is different than a pre-determined relationship of the one or more storage volumes, wherein the pre-determined relationship corresponds to an association of storage system events for at least one of the one or more storage volumes, wherein the pre-determined relationship is associated with a metric threshold;

responsive to determining that the identified relationship is different than the pre-determined relationship, generate a new metric threshold for association with the identified relationship, the new metric threshold different from the metric threshold, wherein the new metric threshold comprises a number of active storage volumes and a time of usage associated with the first storage system event; and

make accessible the new metric threshold.

2. The system of claim 1, wherein the identified relationship is associated with an intermittent downtime at a faulty storage volume.

3. The system of claim 1, wherein the new metric threshold corresponds to aggregating an existing storage command metric to generate a cumulative metric threshold, wherein the existing storage command metric is associated with the metric threshold.

4. A computer-implemented method for generating metric thresholds for relationships associated with storage volumes in a storage system comprising:

obtaining storage command metric information from a storage service, the metric information based at least in part on a storage command request from a virtual computing device instance hosted on a host computing device, wherein the virtual computing device instance is executing the storage command request;

identifying a relationship among a plurality of storage volumes based on the obtained storage command metric information, wherein identifying the relationship further comprises identifying a first storage system event that affects multiple storage volumes;

determining that the identified relationship is different than a pre-determined relationship of the plurality of storage volumes, wherein the pre-determined relationship corresponds to an association of storage system events for at least one storage volume of the plurality of storage volumes, wherein the pre-determined relationship is associated with a metric threshold;

responsive to determining that the identified relationship is different than the pre-determined relationship, generating a new metric threshold for association with the identified relationship, the new metric threshold different from the metric threshold, wherein the new metric threshold comprises a number of active storage volumes and a time of usage associated with the first storage system event; and

make accessible the new metric threshold.

5. The computer-implemented method of claim 4 further comprising:

determining that the identified relationship is related to a temporal characteristic of at least one storage volume; and

aggregating an existing storage command metric to generate a cumulative metric threshold, wherein the existing storage command metric is associated with the metric threshold.

6. The computer-implemented method of claim 5, wherein the temporal characteristic corresponds to an event below a time threshold.

7. The computer-implemented method of claim 5, wherein the event below the time threshold corresponds to a storage volume being associated with a faulty state and a proper operational state.

8. The computer-implemented method of claim 7, wherein aggregating the existing storage command metric to generate the cumulative metric threshold comprises aggregating, for downtime events at the storage volume being associated with the faulty state and the proper operational state, a service downtime metric comprising the one or more downtimes for the storage volume.

9. The computer-implemented method of claim 4, wherein the first storage system event that affects multiple storage volumes corresponds to a service downtime in the storage system, and wherein the new metric threshold further comprises a start time of operation and a stop time of operation for the service downtime.

10. The computer-implemented method of claim 4, wherein the first storage system event that affects multiple storage volumes corresponds to an unauthorized instantiation of storage volumes by a customer account in the storage system, and wherein the time of usage is further associated with the customer account.

11. The computer-implemented method of claim 4, wherein the association of storage system events corresponds to at least one of an instantiation of storage volumes, a replication of a storage volume, reading and writing among the plurality of storage volumes with input/output storage command requests, or any storage system event initiated by a storage command request to a storage service comprising storage volumes.

12. The computer-implemented method of claim 4 further comprising:

receiving a request, from a user-interface of a monitoring service, for metrics regarding an event that affects multiple storage volumes from an operator of the storage system.

13. A non-transitory computer-readable storage medium including computer-executable instructions comprising:

computer-executable instructions that, when executed by a computing device associated with one or more client computing devices;

obtain metric information that is based at least in part on a storage command request from a virtual computing device instance hosted on a host computing device, wherein the virtual computing device instance is executing the storage command request, wherein the storage command request is executed on a plurality of storage volumes having a defined relationship;

identify, with the obtained metric information, an additional relationship among a plurality of storage volumes, wherein identifying the additional relationship further comprises identifying a first storage system event that affects multiple storage volumes;

determine that the identified relationship is different than the defined relationship of the plurality of storage volumes, wherein the defined relationship corresponds to a correlation of at least one storage volume of the plurality of storage volumes with a logical storage component comprising a portion of the plurality of storage volumes; and

responsive to determining that the identified relationship is different than the defined relationship, generate a new metric threshold for association with the identified relationship, the new metric threshold different from the metric threshold, wherein the new metric threshold comprises a number of active storage volumes and a time of usage associated with the first storage system event; and

make accessible the new metric threshold.

14. The non-transitory computer-readable storage medium of claim 13, wherein the first storage system event that affects multiple storage volumes corresponds to at least one of a power outage, a service downtime in a geographical region, or an unauthorized instantiation of storage volumes.

15. The non-transitory computer-readable storage medium of claim 14 further comprising:

determine whether a suppression rule corresponds to the first storage system event that affects multiple storage volumes based at least in part on the identified relationship information; and

responsive to determining that a suppression rule is to be applied, suppress notifications regarding the first storage system event that affects multiple storage volumes.

16. The non-transitory computer-readable storage medium of claim 15 further comprising:

identify a root cause based at least in part on the identified relationship, the root cause representing the first storage system event that affects multiple storage volumes.

17. The non-transitory computer-readable storage medium of claim 15, wherein the identified relationship corresponds to a storage volume downtime relationship representing an intermittent downtime at a faulty storage volume.