Method and system for monitoring and analyzing quality of service in a storage system

• US 9,547,445 B2
• Filed: 11/07/2014
• Issued: 01/17/2017
• Est. Priority Date: 01/14/2014
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Faulkner, Kevin, Fairbanks, Leon, Wu, Siu, Pai, Vinod Vasant, Hrischuk, Curtis
• Primary Examiner(s): Yi, David X
• Assistant Examiner(s): PUCHE, TAHILBA O
• Application Number:
  US14/535,565
• Publication Number:
  US 20150199136A1
• Time in Patent Office: 802 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3419 by assessing time
• CPC Class Codes: G06F 11/3442 for planning or managing the needed capacity
• CPC Class Codes: G06F 11/3485 for I/O devices
• CPC Class Codes: G06F 2201/81 Threshold
• CPC Class Codes: G06F 2201/875 Monitoring of systems including the internet
• CPC Class Codes: G06F 2201/88 Monitoring involving counting
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0653 Monitoring storage devices or systems
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0673 Single storage device
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD

Abstract

Methods and systems for monitoring quality of service (QOS) data for a plurality of storage volumes are provided. QOS data is collected for the plurality of storage volumes and includes a response time in which each of the plurality of storage volumes respond to an input/output (I/O) request. An expected range for future QOS data based on the collected QOS data is generated. The process then determines a deviation of each potential bully storage volume of a resource used by any victim storage volume, where the deviation of each bully storage volume is based on a number of current I/O requests (IOPS) that are processed by each potential bully storage volume, a forecasted value of TOPS and a predicted upper threshold TOPS value for each potential bully storage volume; and filters the potential bully storage volumes based on an impact of each potential bully storage volume.

Claims

1. A machine implemented method, comprising:

collecting quality of service (QOS) data for a plurality of storage volumes from a storage operating system of a storage system, each storage volume associated with a QOS policy;

wherein the QOS data includes a response time in which each of the plurality of storage volumes respond to an input/output (I/O) request;

generating an expected range for future QOS data based on the collected QOS data, where the expected range predicts future response times for the plurality of storage volumes and is a range of measured performance activity of a workload over a period of time;

monitoring QOS data for each storage volume for determining whether a current QOS data for each storage volume is within the expected range;

identifying a plurality of potential victim storage volumes when a response time of the storage volumes has reached a predefined, static threshold value and a dynamic threshold value that is based on the expected range;

filtering the plurality of potential victim storage volumes to determine victim storage volumes based on a deviation of each potential victim storage volume'"'"'s response time, where the deviation is based on a current response time for each potential victim storage volume, a predicted response time that is based on the expected range and a predicted upper threshold response time for each potential victim storage volume that changes dynamically as more QOS data is collected for the plurality of storage volumes;

identifying a plurality potential bully storage volumes based on a visit rate, a service time and utilization of a resource that is in contention and used by any victim storage volume;

determining a deviation of each potential bully storage volume of the resource in contention, where the deviation of each bully storage volume is based on a number of current I/O requests (IOPS) that are processed by each potential bully storage volume, a forecasted value of TOPS based on the collected data and a predicted upper threshold TOPS value for each potential bully storage volume; and

filtering the potential bully storage volumes based on an impact of each potential bully storage volume, where the impact of each potential bully storage volume depends on the deviation and the current number of IOPs for each bully storage volume.

2. The method of claim 1, wherein the expected range provides the dynamic threshold value to a processor executable performance manager for predicting an incident associated with any of the plurality of storage volumes.

3. The method of claim 1, wherein the expected range provides the dynamic threshold value to a processor executable performance manager for determining resource contention between any of the plurality of storage volumes resulting in a storage volume becoming a victim due to overuse by a bully storage volume.

4. The method of claim 1, wherein the QOS data includes a wait time for each I/O request at each of the resources of the storage system.

5. The method of claim 1, wherein the QOS data includes a number of visits for each I/O request at each of the resources within the storage system.

6. The method of claim 1, wherein the resources at the storage system are categorized as a service center that services a request and a delay center that only moves the request to a next destination.

7. The method of claim 6, wherein the resources include a processor for a network module that interfaces with clients and operates as a service center and a processor for a storage module that operates as a service center and manages storage devices where information is stored.

8. A non-transitory, machine readable storage medium having stored thereon instructions for performing a method, comprising machine executable code which when executed by at least one machine, causes the machine to:

collect quality of service (QOS) data for a plurality of storage volumes from a storage operating system of a storage system, each storage volume associated with a QOS policy;

wherein the QOS data includes a response time in which each of the plurality of storage volumes respond to an input/output (I/O) request;

generate an expected range for future QOS data based on the collected QOS data, where the expected range predicts future response times for the plurality of storage volumes and is a range of measured performance activity of a workload over a period of time;

monitor QOS data for each storage volume for determining whether a current QOS data for each storage volume is within the expected range;

identify a plurality of potential victim storage volumes when a response time of the storage volumes has reached a predefined, static threshold value and a dynamic threshold value that is based on the expected range;

filter the plurality of potential victim storage volumes to determine victim storage volumes based on a deviation of each potential victim storage volume'"'"'s response time, where the deviation is based on a current response time for each potential victim storage volume, a predicted response time that is based on the expected range and a predicted upper threshold response time for each potential victim storage volume that changes dynamically as more QOS data is collected for the plurality of storage volumes;

identify a plurality potential bully storage volumes based on a visit rate, a service time and utilization of a resource that is in contention and used by any victim storage volume;

determine a deviation of each potential bully storage volume of the resource in contention, where the deviation of each bully storage volume is based on a number of current I/O requests (IOPS) that are processed by each potential bully storage volume, a forecasted value of TOPS based on the collected data and a predicted upper threshold TOPS value for each potential bully storage volume; and

filter the potential bully storage volumes based on an impact of each potential bully storage volume, where the impact of each potential bully storage volume depends on the deviation and the current number of IOPs for each bully storage volume.

9. The storage medium of claim 8, wherein the expected range provides the dynamic threshold value to a processor executable performance manager for predicting an incident associated with any of the plurality of storage volumes.

10. The storage medium of claim 8, wherein the expected range provides the dynamic threshold value to a processor executable performance manager for determining resource contention between any of the plurality of storage volumes resulting in a storage volume becoming a victim due to overuse by a bully storage volume.

11. The storage medium of claim 8, wherein the QOS data includes a wait time for each I/O request at each of the resources of the storage system.

12. The storage medium of claim 8, wherein the QOS data includes a number of visits for each I/O request at each of the resources within the storage system.

13. The storage medium of claim 8, wherein the resources at the storage system are categorized as a service center that services a request and a delay center that only moves the request to a next destination.

14. The storage medium of claim 13, wherein the resources include a processor for a network module that interfaces with clients and operates as a service center and a processor for a storage module that operates as a service center and manages storage devices where information is stored.

15. A system, comprising:

a memory containing machine readable medium comprising machine executable code having stored thereon instructions; and

a processor module coupled to the memory, the processor module configured to execute the machine executable code to;

collect quality of service (QOS) data for a plurality of storage volumes from a storage operating system of a storage system;

wherein the QOS data includes a response time in which each of the plurality of storage volumes respond to an input/output (I/O) request;

generate an expected range for future QOS data based on the collected QOS data, where the expected range predicts future response times for the plurality of storage volumes and is a range of measured performance activity of a workload over a period of time;

monitor QOS data for each storage volume for determining whether a current QOS data for each storage volume is within the expected range;

identify a plurality of potential victim storage volumes when a response time of the storage volumes has reached a predefined, static threshold value and a dynamic threshold value that is based on the expected range;

filter the plurality of potential victim storage volumes to determine victim storage volumes based on a deviation of each potential victim storage volume'"'"'s response time, where the deviation is based on a current response time for each potential victim storage volume, a predicted response time that is based on the expected range and a predicted upper threshold response time for each potential victim storage volume that changes dynamically as more QOS data is collected for the plurality of storage volumes;

identify a plurality potential bully storage volumes based on a visit rate, a service time and utilization of a resource that is in contention and used by any victim storage volume;

determine a deviation of each potential bully storage volume of the resource in contention, where the deviation of each bully storage volume is based on a number of current I/O requests (IOPS) that are processed by each potential bully storage volume, a forecasted value of TOPS based on the collected data and a predicted upper threshold TOPS value for each potential bully storage volume; and

filter the potential bully storage volumes based on an impact of each potential bully storage volume, where the impact of each potential bully storage volume depends on the deviation and the current number of IOPs for each bully storage volume.

16. The system of claim 15, wherein the expected range provides the dynamic threshold value to a processor executable performance manager for predicting an incident associated with any of the plurality of storage volumes.

17. The system of claim 15, wherein the expected range provides dynamic threshold value to a processor executable performance manager for determining the resource contention between any of the plurality of storage volumes resulting in a storage volume becoming a victim due to overuse by a bully storage volume.

18. The system of claim 15, wherein the QOS data includes a wait time for each I/O request at each of the resources of the storage system.

19. The system of claim 15, wherein the QOS data includes a number of visits for each I/O request at each of the resources within the storage system.

20. The system of claim 15, wherein the resources at the storage system are categorized as a service center that services a request and a delay center that only moves the request to a next destination.