Root cause detection and monitoring for storage systems

• US 9,898,357 B1
• Filed: 06/25/2015
• Issued: 02/20/2018
• Est. Priority Date: 06/25/2015
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Viswanathan, Ganesh, Sasikumar, Vinayak, Pop, Artur, Chang, Shuai, Zeghers, Benjamin Ryan
• Primary Examiner(s): Mehrmanesh, Elmira
• Application Number:
  US14/751,047
• Time in Patent Office: 971 Days
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 2201/815 Virtual middleware or OS functionality using virtual machines to implement generic software techniques for error detection or fault masking G06F11/1484
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

Notification routines are described for implementation by a monitoring service. As part of an exemplary notification routine, a faulty storage volume is correlated at multiple logical storage levels of a storage system with other faulty storage volumes. The correlation pattern can follow a tree-based decision format, where each faulty storage volume is sequentially compared at a lower logical storage level. Advantageously, once a common logical storage component of a group of storage volumes is identified, a notification is issued about the group of faulty storage volumes sharing the common logical storage component. Additionally, notifications can be issued according to a severity level of the group of faulty storage volumes. In some embodiments, before issuing the notification, the group of faulty storage volumes can be compared to a time allowed for the group of faulty storage volume to be at fault.

Claims

1. A system comprising:

one or more virtual computing device instances for transmitting storage requests via a communications network, the one or more virtual computing device instances executing on behalf of a client computing device, and the one or more virtual computing device instances hosted on a host computing device;

a storage processing service, executed on one or more storage computing devices, the storage processing service configured to;

obtain a storage command request from a virtual computing device instance of the one or more virtual computing device instances;

process the storage command request to generate a storage command processing result associated with at least one storage volume, the at least one storage volume associated with a storage computing device of the one or more storage computing devices;

collect storage command metric information based at least in part on the storage command processing result; and

a storage monitoring service, executed on one or more computing devices, configured to;

obtain, from the storage processing service, the collected storage command metric information;

process the collected storage command metric information for the at least one storage volume to generate a correlation processing result for a logical storage component representing a logical storage level of the one or more storage computing devices;

identify, across the one or more storage computing devices, a correlation relationship; and

identify, based on the identified correlation relationship, one or more faulty storage volumes among the one or more storage computing devices;

for each faulty storage volume of the one or more faulty storage volumes, correlate one or more characteristics of a first faulty storage volume with corresponding characteristics of a second faulty storage volume, wherein the one or more characteristics represent logical storage levels of a storage system;

identify a common logical storage component of the first faulty storage volume and the second faulty storage volume;

determine that a quantity of the identified one or more faulty storage volumes satisfies a threshold; and

issue a notification regarding the quantity of the identified one or more faulty storage volumes satisfying the threshold.

2. The system of claim 1, wherein the one or more characteristics corresponds to a server hosting the first faulty storage volume and the second faulty storage volume, wherein the storage monitoring service is further configured to:

determine that the first faulty storage volume and the second faulty storage volume have exceeded a time threshold for being associated with a fault;

correlate, at lower logical levels of the storage system, the one or more characteristics of the first faulty storage volume with corresponding characteristics of the second faulty storage volume at a lower logical storage level; and

issue a second notification regarding the first faulty storage volume and second faulty storage volume.

3. The system of claim 1, wherein the identified common logical storage component corresponds to a shared network, wherein correlating the one or more characteristics of the first faulty storage volume with the corresponding characteristics of the second faulty storage volume comprises determining that the first faulty storage volume and the second faulty storage volume are on the shared network,

the notification further comprising information regarding the first faulty storage volume and the second faulty storage volume being on the shared network.

4. The system of claim 1, wherein the threshold corresponds to a maximum number of faulty volumes.

5. A computer-implemented method for identifying a common logical storage component of storage volumes, the computer-implemented method comprising:

obtaining metric information from a storage processing service, the metric information based at least in part on a storage command request;

obtaining correlated relationship information regarding the obtained metric information, the correlated relationship information indicating a relationship between a first fault of at least one storage volume and a second fault of a logical storage component, wherein the logical storage component includes a logical storage level of two or more storage volumes including the at least one storage volume;

identifying one or more storage volumes based at least in part on the correlated relationship information, each storage volume indicating a fault;

for each storage volume of the one or more storage volumes, correlating a first characteristic of a first storage volume with a second characteristic of a second storage volume;

identifying a common logical storage component of the first storage volume and the second storage volume;

determining that a quantity of the one or more storage volumes satisfies a threshold; and

issuing a notification regarding the quantity of the one or more storage volumes satisfying the threshold.

6. The computer-implemented method of claim 5 further comprising:

determining that the first storage volume and the second storage volume have exceeded a time threshold for being associated with a first fault.

7. The computer-implemented method of claim 5, wherein the first characteristic corresponds to the first storage volume being on a storage server, wherein the second characteristic corresponds to the second storage volume being on the storage server, wherein the identified common logical storage component corresponds to a storage component at a lower logical storage level than the storage server, further comprising issuing a second notification regarding the first storage volume and the second storage volume, the second notification comprising information about the lower logical storage level.

8. The computer-implemented method of claim 5 wherein the identified common logical storage component corresponds to a shared network, the computer-implemented method further comprising:

determining whether the first storage volume and the second storage volume share a lower logical storage level; and

responsive to the determination that the first storage volume and the second storage volume do not share a lower logical storage level, issuing a second notification regarding the first storage volume and second storage volume being on the shared network.

9. The computer-implemented method of claim 5, wherein the first fault of the at least one storage volume corresponds to a number of I/O read operations per second falling below an I/O read operation threshold, and wherein the second fault of the logical storage component corresponds to a network latency of the logical storage component, wherein a number of I/O read operations per second at the logical storage component affects the network latency of logical storage component.

10. The computer-implemented method of claim 5, wherein the first fault of the at least one storage volume further corresponds to a number of I/O write operations per second falling below an I/O write operation threshold and a queue length of the at least one storage volume exceeding a threshold defining a maximum queue length.

11. The computer-implemented method of claim 5, wherein the logical storage component is determined according to a location on a rack.

12. The computer-implemented method of claim 5 further comprising:

obtaining an index of storage volumes indicating a fault from a monitoring service.

13. The computer-implemented method of claim 5 further comprising:

determining whether one or more suppression rules correspond to the at least one storage volume based at least in part on the identified correlation relationship information; and

responsive to determining that the one or more suppression rules correspond to the at least one storage volume, suppressing notifications regarding the at least one storage volume.

14. The computer-implemented method of claim 5, wherein the storage command request is received from a virtual computing device instance hosted on a host computing device, wherein the virtual computing device instance is executing the storage command request on behalf of a client computing device.

15. A non-transitory computer-readable storage medium including computer-executable instructions comprising:

computer-executable instructions that, when executed by a computing device associated with one or more client computing devices;

obtain metric information from a storage processing service, the metric information based at least in part on a storage command request from a virtual computing device instance hosted on a host computing device, wherein the virtual computing device instance is executing the storage command request on behalf of a client computing device;

obtain correlated relationship information regarding the obtained metric information, the correlated relationship information indicating a relationship between a first fault of at least one storage volume and a second fault of a logical storage component, wherein the logical storage component includes a logical storage level of two or more storage volumes including the at least one storage volume; and

identify one or more storage volumes based at least in part on the correlated relationship information, each storage volume indicating a fault;

for each storage volume of the one or more storage volumes, correlate a first characteristic of a first storage volume with a second characteristic of a second storage volume;

identify a shared characteristic of the first storage volume and the second storage volume;

determine that a quantity of the one or more storage volumes satisfies a threshold; and

issue a notification regarding the quantity of the one or more storage volumes satisfying the threshold.

16. The non-transitory computer-readable storage medium of claim 15, wherein the threshold corresponds to a maximum number of faulty volumes.

17. The non-transitory computer-readable storage medium of claim 15 further comprising:

responsive to the identification of the shared characteristic being a shared network, determine whether the first storage volume and the second storage volume share a lower logical storage level; and

responsive to the determination that the first storage volume and the second storage volume do not share a lower logical storage level, issue a second notification regarding the first storage volume and second storage volume being on the shared network.

18. The non-transitory computer-readable storage medium of claim 15 further comprising, wherein the first fault of the at least one storage volume further corresponds to a queue length of the at least one storage volume exceeding a queue length threshold, the queue length having been generated by storage command requests from an account associated with the at least one storage volume.

19. The non-transitory computer-readable storage medium of claim 15 further comprising:

identify a root cause of a storage system issue based at least in part on the correlated relationship information.

20. The non-transitory computer-readable storage medium of claim 15 further comprising:

issue a second notification regarding the shared characteristic according to a severity level of the second notification.

21. The non-transitory computer-readable storage medium of claim 20, wherein the severity level of the second notification corresponds to a low severity level indicating that the first storage volume and the second storage volume have been at fault for less than ten minutes.

22. The non-transitory computer-readable storage medium of claim 20, wherein the severity level of the second notification corresponds to a high severity level indicating that the first storage volume and the second storage volume have been at fault for more than ninety minutes, wherein the second notification is issued directly to an operator of the storage system with the severity level indicated as part of the second notification.