Root cause detection and monitoring for storage systems

• US 10,223,189 B1
• Filed: 06/25/2015
• Issued: 03/05/2019
• Est. Priority Date: 06/25/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Viswanathan, Ganesh, Sasikumar, Vinayak, Pop, Artur, Chang, Shuai, Zeghers, Benjamin Ryan
• Primary Examiner(s): Wilson, Yolanda L
• Application Number:
  US14/751,036
• Time in Patent Office: 1,349 Days
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0754 by exceeding limits
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/0772 Means for error signaling, e.g. using interrupts, exception flags, dedicated error registers
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: H04L 41/0622 based on time
• CPC Class Codes: H04L 41/0631 using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
• CPC Class Codes: H04L 41/0681 Configuration of triggering conditions
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Suppression routines are described for implementation by a monitoring service. The monitoring service uses collected data to identify faulty storage volumes. Advantageously, in some cases, the monitoring service can notify an operator of the storage system that certain storage volumes are faulty. In some embodiments, these notifications are to be suppressed because not all notifications of faulty volumes are necessary. Suppression rules can indicate that a faulty storage volume is at fault because it is a test volume, associated with a large power outage, or some other learned event from storage command metrics. The monitoring service can suppress notifications about these known system issues, among others.

Claims

1. A system comprising:

a host computing device configured to host one or more virtual computing device instances, the host computing device configured to transmit storage commands generated by the one or more virtual computing device instances via a communications network, the one or more virtual computing device instances executing on behalf of a client computing device;

a storage processing service, executed on one or more storage computing devices, the storage processing service configured to;

obtain a storage command request from a client computing device of the one or more virtual computing devices;

process the storage command request to generate a storage command processing result associated with at least one storage volume, the storage volume associated with the one or more storage computing devices; and

collect storage command metric information based at least in part on the storage command processing result; and

a storage monitoring service, executed on one or more computing devices, configured to;

obtain, from the storage processing service, the collected storage command metric information;

process the collected storage command metric information for the at least one storage volume;

identify a correlation relationship of a first storage volume and a second storage volume across the one or more storage computing devices, the correlation relationship further indicating a first fault of the least one storage volume and a second fault of a logical storage component, wherein the logical storage component includes a logical storage level of the first storage volume and the second storage volume including the at least one storage volume;

identify, based at least in part on the identified correlation relationship, one or more faulty storage volumes among the one or more storage computing devices;

obtaining suppression threshold information corresponding to at least one storage volume, the suppression threshold information indicating that a notification for a storage system issue is to be suppressed;

determine that one of the one or more faulty storage volumes corresponds to the suppression threshold information; and

suppress notifications regarding the one faulty storage volume.

2. The system of claim 1, wherein the suppression threshold information is associated with an identifier for the one faulty storage volume.

3. The system of claim 1, wherein the suppression threshold information is associated with suppressing storage volumes sharing a common logical storage level of the one or more storage computing devices.

4. The system of claim 1, wherein the storage monitoring service further:

receives information regarding a time period for suppression; and

wherein the storage monitoring service suppresses notifications regarding the faulty storage volume in accordance with the time period for suppression.

5. A computer-implemented method for managing fault notifications in a storage system comprising:

obtaining storage command metric information from a storage processing service, the storage command metric information based at least in part on a storage command request from a virtual computing device instance hosted on a host computing device;

obtaining correlation relationship information regarding the obtained storage command metric information, the correlation relationship information indicating a relationship between a first storage volume and a second storage volume, the correlation relationship information further indicating a first fault of at least one storage volume and a second fault of a logical storage component, wherein the logical storage component includes a logical storage level of the first storage volume and the second storage volume including the at least one storage volume;

identifying one or more storage volumes based at least in part on the correlation relationship information, the one or more storage volumes indicating a fault;

obtaining suppression threshold information corresponding to at least one storage volume indicating a fault, the suppression threshold information indicating that a notification for a storage system is to be suppressed;

determining that one of the identified one or more storage volumes corresponds to the suppression threshold information; and

suppressing notifications regarding the identified storage volume.

6. The method of claim 5, wherein determining that one of the identified one or more storage volumes corresponds to the suppression threshold information comprises determining whether a suppression threshold has been exceeded by the identified storage volume, the suppression threshold corresponding to a metric based on the correlation relationship information.

7. The computer-implemented method of claim 6, wherein the suppression threshold corresponds to storage volumes in a logical storage level of the one or more storage computing devices.

8. The computer-implemented method of claim 5 further comprising, responsive to determining that a suppression rule is to be applied to the storage volume, discontinuing monitoring of the identified storage volume for a period of time.

9. The computer-implemented method of claim 5, wherein identifying the one or more storage volumes based at least in part on the correlation relationship information comprises determining that at least one of the one or more storage volumes has exceeded a time threshold for being at fault.

10. The computer-implemented method of claim 5, wherein the identified storage volume further comprises at least one of a recovered storage volume, a deleted volume, a lost volume, or a storage volume not currently associated with the storage processing service.

11. The computer-implemented method of claim 5, wherein the first fault of the at least one storage volume further corresponds to a number of I/O write operations per second falling below an I/O write operation threshold or a queue length of the at least one storage volume exceeding a queue length threshold.

12. The computer-implemented method of claim 5, wherein the logical storage component is determined according to a location on a rack.

13. The computer-implemented method of claim 5, wherein the identified storage volume is associated with an identified system issue.

14. A non-transitory computer-readable storage medium including computer-executable instructions that, when executed by a computing device, cause the computing device to:

obtain storage command metric information from a storage processing service, the storage command metric it based at least in part on a storage command request from a virtual computing device instance hosted on a host computing device;

obtain correlation relationship information regarding the obtained storage command metric information, the correlation relationship information indicating a relationship between a first storage volume and a second storage volume, the correlation relationship information further indicating a first fault of at least one storage volume and a second fault of a logical storage component, wherein the logical storage component includes a logical storage level of the first storage volume and the second storage volume including the at least one storage volume;

identify one or more storage volumes based at least in part on the correlation relationship information, the one or more storage volumes indicating a fault;

obtain a suppression threshold information corresponding to at least one storage volume, the suppression threshold information indicating that a notification for a storage system is to be suppressed;

determining that the at least one storage volume corresponds to the suppression threshold information; and

suppress notifications regarding the at least one storage volume.

15. The non-transitory computer-readable storage medium of claim 14, wherein the at least one storage volume is a test volume.

16. The non-transitory computer-readable storage medium of claim 14, wherein identifying the one or more storage volumes further comprises:

obtaining information regarding an event that affects multiple storage volumes at the storage processing service.

17. The non-transitory computer-readable storage medium of claim 14 further comprising:

correlating a first characteristic of the first storage volume with a second characteristic of the second storage volume; and

identifying a common characteristic of the first storage volume and the second storage volume; and

identifying a suppression rule based at least in part on the common characteristic.

18. The non-transitory computer-readable storage medium of claim 17 further comprising:

determining that a group of the one or more storage volumes shares the identified common characteristic, the group comprising the first storage volume and the second storage volume;

determining that the group of the one or more storage volumes exceed a group threshold; and

applying the identified suppression rule to the group of the one or more storage volumes.

19. The non-transitory computer-readable storage medium of claim 18, wherein the group threshold corresponds to a metric based on the obtained correlation relationship information.

20. The non-transitory computer-readable storage medium of claim 14, wherein the suppression rule is associated with an identifier for characteristics of a server hosting the at least one storage volume, wherein the characteristics of the server correspond to at least one of the server being degraded or the server being a test server.