Correctly identifying potential anomalies in a distributed storage system

• US 9,575,828 B2
• Filed: 07/08/2015
• Issued: 02/21/2017
• Est. Priority Date: 07/08/2015
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Agarwal, Amit, Dutta, Debojyoti, George, Johnu, Sharma, Manoj, Zhang, Kai, Tarre, Marc Solanas, Das Gupta, Aparupa, Udupi, Yathiraj B., Mason, Seth
• Primary Examiner(s): Iqbal, Nadeem
• Application Number:
  US14/794,676
• Publication Number:
  US 20170010931A1
• Time in Patent Office: 594 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0787 Storage of error reports, e.g. persistent data storage, storage using memory protection
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: H04L 67/1034 Reaction to server failures by a load balancer
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method for assisting evaluation of anomalies in a distributed storage system is disclosed. The method includes a step of monitoring at least one system metric of the distributed storage system. The method further includes steps of maintaining a listing of patterns of the monitored system metric comprising patterns which previously did not result in a failure within one or more nodes of the distributed storage system, and, based on the monitoring, identifying a pattern (i.e., a time series motif) of the monitored system metric as a potential anomaly in the distributed storage system. The method also includes steps of automatically (i.e. without user input) performing a similarity search to determine whether the identified pattern satisfies one or more predefined similarity criteria with at least one pattern of the listing, and, upon positive determination, excepting the identified pattern from being identified as the potential anomaly.

Claims

1. A method for assisting evaluation of anomalies in a distributed storage system, the method comprising:

monitoring at least one system metric of the distributed storage system;

maintaining a listing of patterns of the at least one system metric which previously did not result in a failure within one or more nodes of the distributed storage system;

identifying a pattern of the at least one system metric as a potential anomaly in the distributed storage system;

automatically performing a similarity search to determine whether the identified pattern satisfies one or more similarity criteria with at least one pattern of the listing; and

upon positive determination, excepting the identified pattern from being identified as the potential anomaly.

2. The method according to claim 1, wherein the pattern of the at least one system metric is identified as a potential anomaly based on comparison of values of the at least one system metric within a duration of the pattern with values of the at least one system metric within an earlier time interval of the duration of the pattern.

3. The method according to claim 1, wherein:

monitoring the at least one system metric of the distributed storage system comprises monitoring the at least one system metric for a first node of the distributed storage system,

the method further comprises monitoring of the at least one system metric for a second node of the distributed storage system, and

the pattern of the at least one system metric for the first node is identified as a potential anomaly based on comparison of values of the at least one system metric for the first node with values of the at least one system metric for the second node.

4. The method according to claim 1, wherein the pattern of the at least one system metric is identified as a potential anomaly based on comparison of values of the at least one system metric with values of at least one other system metric.

5. The method according to claim 1, further comprising:

upon negative determination, determining whether the identified pattern is associated with the failure within one or more nodes of the distributed storage system.

6. The method according to claim 5, wherein the determination of whether the identified pattern is associated with the failure is based on one or more logs generated by one or more services associated with the at least one system metric.

7. The method according to claim 1, wherein the patterns of the listing were previously identified as potential anomalies.

8. A system for assisting evaluation of anomalies in a distributed storage system, the system comprising:

at least one memory configured to store computer executable instructions, and

at least one processor coupled to the at least one memory and configured, when executing the instructions, to;

monitor at least one system metric of the distributed storage system;

maintain a listing of patterns of the at least one system metric which previously did not result in a failure within one or more nodes of the distributed storage system;

identify a pattern of the at least one system metric as a potential anomaly in the distributed storage system;

automatically perform a similarity search to determine whether the identified pattern satisfies one or more similarity criteria with at least one pattern of the listing; and

upon positive determination, except the identified pattern from being identified as the potential anomaly.

9. The system according to claim 8, wherein the pattern of the at least one system metric is identified as a potential anomaly based on comparison of values of the at least one system metric within a duration of the pattern with values of the at least one system metric within an earlier time interval of the duration of the pattern.

10. The system according to claim 8, wherein:

monitoring the at least one system metric of the distributed storage system comprises monitoring the at least one system metric for a first node of the distributed storage system, the method further comprises monitoring of the at least one system metric for a second node of the distributed storage system, and

the pattern of the at least one system metric for the first node is identified as a potential anomaly based on comparison of values of the at least one system metric for the first node with values of the at least one system metric for the second node.

11. The system according to claim 8, wherein the pattern of the at least one system metric is identified as a potential anomaly based on comparison of values of the at least one system metric with values of at least one other system metric.

12. The system according to claim 8, wherein the at least one processor is further configured to, upon negative determination, determine whether the identified pattern is associated with the failure within one or more nodes of the distributed storage system.

13. The system according to claim 12, wherein the determination of whether the identified pattern is associated with the failure is based on one or more logs generated by one or more services associated with the at least one system metric.

14. The system according to claim 8, wherein the patterns of the listing were previously identified as potential anomalies.

15. One or more computer readable storage media encoded with software comprising computer executable instructions and when the software is executed operable to perform a method for assisting evaluation of anomalies in a distributed storage system, the method comprising:

monitoring at least one system metric of the distributed storage system;

maintaining a listing of patterns of the at least one system metric which previously did not result in a failure within one or more nodes of the distributed storage system;

identifying a pattern of the at least one system metric as a potential anomaly in the distributed storage system;

automatically performing a similarity search to determine whether the identified pattern satisfies one or more similarity criteria with at least one pattern of the listing; and

upon positive determination, excepting the identified pattern from being identified as the potential anomaly.

16. The one or more computer readable media according to claim 15, wherein the pattern of the at least one system metric is identified as a potential anomaly based on comparison of values of the at least one system metric within a duration of the pattern with values of the at least one system metric within an earlier time interval of the duration of the pattern.

17. The one or more computer readable media according to claim 15, wherein:

monitoring the at least one system metric of the distributed storage system comprises monitoring the at least one system metric for a first node of the distributed storage system,

the method further comprises monitoring of the at least one system metric for a second node of the distributed storage system, and

the pattern of the at least one system metric for the first node is identified as a potential anomaly based on comparison of values of the at least one system metric for the first node with values of the at least one system metric for the second node.

18. The one or more computer readable media according to claim 15, wherein the pattern of the at least one system metric is identified as a potential anomaly based on comparison of values of the at least one system metric with values of at least one other system metric.

19. The one or more computer readable media according to claim 15, wherein the method further comprises:

upon negative determination, determining whether the identified pattern is associated with the failure within one or more nodes of the distributed storage system.

20. The one or more computer readable media according to claim 15, wherein the patterns of the listing were previously identified as potential anomalies.