Method and system for monitoring and analyzing quality of service in a metro-cluster

• US 9,658,778 B2
• Filed: 11/03/2014
• Issued: 05/23/2017
• Est. Priority Date: 01/14/2014
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Hrischuk, Curtis, Larson, Phil, Wu, Siu, Basit, Abdul, Faulkner, Kevin, Fairbanks, Leon, Pai, Vinod Vasant
• Primary Examiner(s): Schnee, Hal
• Application Number:
  US14/531,246
• Publication Number:
  US 20150199141A1
• Time in Patent Office: 932 Days
• CPC Class Codes: G06F 11/076 by exceeding a count or rate limit, e.g. word- or bit count limit
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3419 by assessing time
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 2201/815 Virtual
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0617 in relation to availability
• CPC Class Codes: G06F 3/0653 Monitoring storage devices or systems
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

Methods and systems for inter-cluster storage system monitoring and analysis are provided. The method includes monitoring a non-volatile memory delay center for a first storage cluster having a first node and a second node configured to operate as a first high availability pair, where data for a write request to write data to the first node is also written to the second node as well as to a second cluster having a third node and a fourth node, where the third node and the fourth node are also configured to operate as a second high availability pair to store the data for the write request at one or both of the third and fourth node. The non-volatile memory delay center is used to monitor and detect latency due to any delay caused by a non-volatile memory of the first node used as a write cache.

Claims

1. A machine implemented method, comprising:

monitoring a non-volatile memory delay center for a first cluster having a first node and a second node configured to operate as a first high availability pair, where data for a write request to write data to the first node is also written to the second node as well as to a second cluster having a third node and a fourth node, where the third node and the fourth node are also configured to operate as a second high availability pair to store the data for the write request at one or both of the third and fourth node;

wherein the non-volatile memory delay center is used to monitor and detect latency due to any delay caused by a non-volatile memory of the first node used as a write cache;

determining if a sum of write traffic for storage volumes associated with the first node and write traffic from either the third node or the fourth node has reached a threshold value, when the latency in processing write requests is unacceptable; and

notifying a user to check the non-volatile memory when the threshold value is reached;

wherein a management application collects response time for the non-volatile memory delay center and predicts an expected response time for the future and when the predicted response time reaches a certain value, then the sum of write traffic for the storage volumes associated with the first node and the write traffic from either the third node or the fourth node is compared with an overall storage capacity of the non-volatile memory to determine if the overall storage capacity has been reached.

2. The method of claim 1, wherein the threshold value indicates a maximum storage capacity for the non-volatile memory of the first node.

3. The method of claim 1, executing the monitoring, determining and notifying steps for non-volatile memory used by each of the second node, the third node and the fourth node.

4. The method of claim 1, wherein when the threshold value is not reached, notifying the user to check a link between the first cluster and the second cluster.

5. The method of claim 1, wherein the management application monitors non-volatile memory delay centers for both the first cluster and the second cluster to predict expected response times for each delay center.

6. The method of claim 1, further comprising:

identifying the non-volatile memory as a resource causing unacceptable latency in processing write requests by the first node, when the threshold value has been reached.

7. The method of claim 1, further comprising:

identifying a workload as a bully workload for reaching the threshold value based on an amount of data that is written by the first node for the workload using the non-volatile memory as the write cache.

8. A non-transitory, machine readable storage medium having stored thereon instructions for performing a method, comprising machine executable code which when executed by at least one machine, causes the machine to:

monitor a non-volatile memory delay center for a first cluster having a first node and a second node configured to operate as a first high availability pair, where data for a write request to write data to the first node is also written to the second node as well as to a second cluster having a third node and a fourth node, where the third node and the fourth node are also configured to operate as a second high availability pair to store the data for the write request at one or both of the third and fourth node;

wherein the non-volatile memory delay center is used to monitor and detect latency due to any delay caused by a non-volatile memory of the first node used as a write cache;

determine if a sum of write traffic for storage volumes associated with the first node and write traffic from either the third node or the fourth node has reached a threshold value, when the latency in processing write requests is unacceptable; and

notify a user to check the non-volatile memory when the threshold value is reached;

wherein a management application collects response time for the non-volatile memory delay center and predicts an expected response time for the future and when the predicted response time reaches a certain value, then the sum of write traffic for the storage volumes associated with the first node and the write traffic from either the third node or the fourth node is compared with an overall storage capacity of the non-volatile memory to determine if the overall storage capacity has been reached.

9. The storage medium of claim 8, wherein the threshold value indicates a maximum storage capacity for the non-volatile memory of the first node.

10. The storage medium of claim 8, executing the monitoring, determining and notifying steps for non-volatile memory used by each of the second node, the third node and the fourth node.

11. The storage medium of claim 8, wherein when the threshold value is not reached, notifying the user to check a link between the first cluster and the second cluster.

12. The storage medium of claim 8, wherein the management application monitors non-volatile memory delay centers for both the first cluster and the second cluster to predict expected response times for each delay center.

13. The storage medium of claim 8, wherein the machine executable code further causes the machine to:

identify the non-volatile memory as a resource causing unacceptable latency in processing write requests by the first node, when the threshold value has been reached.

14. The storage medium of claim 8, wherein the machine executable code further causes the machine to:

identify a workload as a bully workload for reaching the threshold value based on an amount of data that is written by the first node for the workload using the non-volatile memory as the write cache.

15. A system comprising:

a memory containing machine readable medium comprising machine executable code having stored thereon instructions; and

a processor module coupled to the memory, the processor module configured to execute the machine executable code to;

monitor a non-volatile memory delay center for a first cluster having a first node and a second node configured to operate as a first high availability pair, where data for a write request to write data to the first node is also written to the second node as well as to a second cluster having a third node and a fourth node, where the third node and the fourth node are also configured to operate as a second high availability pair to store the data for the write request at one or both of the third and fourth node;

wherein the pg,73 non-volatile memory delay center is used to monitor and detect latency due to any delay caused by a non-volatile memory of the first node used as a write cache;

determine if a sum of write traffic for storage volumes associated with the first node and write traffic from either the third node or the fourth node has reached a threshold value, when the latency in processing write requests is unacceptable; and

notify a user to check the non-volatile memory when the threshold value is reached;

wherein a management application collects response time for the non-volatile memory delay center and predicts an expected response time for the future and when the predicted response time reaches a certain value, then the sum of write traffic for the storage volumes associated with the first node and the write traffic from either the third node or the fourth node is compared with an overall storage capacity of the non-volatile memory to determine if the overall storage capacity has been reached.

16. The system of claim 15, wherein the threshold value indicates a maximum storage capacity for the non-volatile memory of the first node.

17. The system of claim 15, executing the monitoring, determining and notifying steps for non-volatile memory used by each of the second node, the third node and the fourth node.

18. The system of claim 15, wherein when the threshold value is not reached, notifying the user to check a link between the first cluster and the second cluster.

19. The system of claim 15, wherein the management application monitors non-volatile memory delay centers for both the first cluster and the second cluster to predict expected response times for each delay center.

20. The system of claim 15, wherein the machine executable code further causes the machine to:

identify a workload as a bully workload for reaching the threshold value based on an amount of data that is written by the first node for the workload using the non-volatile memory as the write cache.