Methods and systems for filtering collected QOS data for predicting an expected range for future QOS data

• US 9,870,330 B2
• Filed: 03/04/2015
• Issued: 01/16/2018
• Est. Priority Date: 01/14/2014
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Faulkner, Kevin, Weihs, Joseph, Wang, Rui
• Primary Examiner(s): Shin, Christopher
• Application Number:
  US14/638,220
• Publication Number:
  US 20160259742A1
• Time in Patent Office: 1,049 Days
• Field of Search: 710301
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 11/3485 for I/O devices
• CPC Class Codes: G06F 13/36 for access to common bus or bus system
• CPC Class Codes: G06F 13/4221 being an input/output bus, e.g. ISA bus, EISA bus, PCI bus, SCSI bus
• CPC Class Codes: G06F 2201/81 Threshold
• CPC Class Codes: G06F 2201/88 Monitoring involving counting
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0653 Monitoring storage devices or systems
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

Methods and systems for monitoring quality of service (QOS) data for a plurality of storage volumes from a storage operating system of a storage system are provided. A performance manager collects QOS data for a storage volume from among the plurality of storage volumes and the QOS data includes a response time in which the storage volume responds to an input/output (I/O) request; determines that the collected QOS data is noisy by comparing an average number of I/O requests processed within a time duration for the storage volume with a first threshold value; uses comparable QOS data of another storage volume for generating an expected range for future QOS data; and monitors QOS data for the storage volume for determining whether a current QOS data for the storage volume is within the expected range.

Claims

1. A machine implemented method, comprising:

collecting quality of service (QOS) data for a storage volume from among a plurality of storage volumes used for storing data for a storage operating system of a storage system having a plurality of storage devices, wherein a processor executable performance manager interfacing with the storage system collects the QOS data from the storage operating system of the storage system using a network connection, and wherein the QOS data includes a response time in which the storage volume responds to input/output (I/O) requests for storing and retrieving data at the plurality of storage devices and an average number of I/O requests that are processed by the storage operating system using the storage volume within a certain duration;

identifying by the performance manager, a portion of the collected QOS data that is noisy by comparing the average number of I/O requests processed within the certain duration for the storage volume with a first threshold value;

wherein the first threshold value is set to detect unreliability in the collected QOS data due to a measurement error by the storage system;

selecting reliable QOS data to replace the portion of the collected QOS data by the performance manager;

wherein the reliable data is based on QOS data for one or more other storage volumes of the storage system that have a QOS policy similar to a QOS policy for the storage volume and share a resource of the storage system with the storage volume associated with the portion of the collected QOS data;

replacing the portion of the collected QOS data with the reliable QOS data by the performance manager for generating an expected range for evaluating future QOS collected data for the storage volume;

using the expected range and the future collected QOS data by the performance manager to identify resource contention associated with one or more resources of the storage system used by the storage volume and identifying a bully workload contributing to the resource contention; and

executing a remediation task for reducing the resource contention, the remediation task depending on a computing resource type of the one or more resources in contention.

2. The method of claim 1, wherein the portion of the collected QOS data is noisy when the average number of I/O requests processed within the certain duration is greater than the first threshold value and less than a second threshold value that is set to offset variation due to any random error.

3. The method of claim 1, wherein the reliable QOS data is selected for the one or more other storage volumes that share a resource of a same cluster of the storage system with the storage volume associated with the portion of collected QOS data.

4. The method of claim 1, wherein the expected range is generated using a linear prediction model executed by the performance manager.

5. The method of claim 1, wherein the expected range provides a dynamic threshold value to the performance manager for predicting an incident associated with the storage volume for identifying the storage volume as the victim.

6. The method of claim 1, wherein the expected range is generated when the collected QOS data meets a minimal quantity and when the minimal quantity is not met, then one or more statistical values based on the collected QOS data is used for generating the expected range.

7. The method claim 1, wherein the reliable QOS data of the other one or more storage volumes is selected when the one and more of the other storage volumes and the storage volume associated with the noisy portion of the collected QOS data have similar read/write operation ratios for reading and writing data.

8. A non-transitory, machine readable storage medium having stored thereon instructions for performing a method, comprising machine executable code which when executed by at least one machine, causes the machine to:

collect quality of service (QOS) data for a storage volume from among a plurality of storage volumes used for storing data for a storage operating system of a storage system having a plurality of storage devices, wherein a processor executable performance manager interfacing with the storage system collects the QOS data from the storage operating system of the storage system using a network connection, and wherein the QOS data includes a response time in which the storage volume responds to input/output (I/O) requests for storing and retrieving data at the plurality of storage devices and an average number of I/O requests that are processed by the storage operating system using the storage volume within a certain duration;

identify by the performance manager, a portion of the collected QOS data that is noisy by comparing the average number of I/O requests processed within the certain duration for the storage volume with a first threshold value;

wherein the first threshold value is set to detect unreliability in the collected QOS data due to a measurement error by the storage system;

select reliable QOS data to replace the portion of the collected QOS data by the performance manager;

wherein the reliable data is based on QOS data for one or more other storage volumes of the storage system that have a QOS policy similar to a QOS policy for the storage volume and share a resource of the storage system with the storage volume associated with the portion of the collected QOS data;

replace the portion of the collected QOS data with the reliable QOS data by the performance manager for generating an expected range for evaluating future QOS collected data for the storage volume;

use the expected range and the future collected QOS data by the performance manager to identify resource contention associated with one or more resources of the storage system used by the storage volume and identify a bully workload contributing to the resource contention; and

execute a remediation task for reducing the resource contention, the remediation task depending on a computing resource type of the one or more resources in contention.

9. The storage medium of claim 8, wherein the portion of the collected QOS data is noisy when the average number of I/O requests processed within the certain duration is greater than the first threshold value and less than a second threshold value that is set to offset variation due to any random error.

10. The storage medium of claim 8, wherein the reliable QOS data is selected for the one or more other storage volumes that share a resource of a same cluster of the storage system with the storage volume associated with the portion of collected QOS data.

11. The storage medium of claim 8, wherein the expected range is generated using a linear prediction model executed by the performance manager.

12. The storage medium of claim 8, wherein the expected range provides a dynamic threshold value to the performance manager for predicting an incident associated with the storage volume for identifying the storage volume as the victim.

13. The storage medium of claim 8, wherein the expected range is generated when the collected QOS data meets a minimal quantity and when the minimal quantity is not met, then one or more statistical values based on the collected QOS data is used for generating the expected range.

14. The storage medium of claim 8, wherein the reliable QOS data of the other one or more storage volumes is selected when the one and more of the other storage volumes and the storage volume associated with the noisy portion of the collected QOS data have similar read/write operation ratios for reading and writing data.

15. A system, comprising:

a processor of a storage system executing a storage operating system using a plurality of storage volumes for storing information at a plurality of storage devices; and

a processor for a performance manager that interfaces with the storage system, executing instructions out of a memory device to;

collect quality of service (QOS) data maintained by the storage system for a storage volume from among the plurality of storage volumes wherein the QOS data is maintained by the storage system and includes a response time in which the storage volume responds to input/output (I/O) requests for storing and retrieving data at the plurality of storage devices and an average number of I/O requests that are processed by the storage operating system using the storage volume within a certain duration;

identify a portion of the collected QOS data that is noisy by comparing the average number of I/O requests processed within the certain duration for the storage volume with a first threshold value;

wherein the first threshold value is set to detect unreliability in the collected QOS data due to a measurement error by the storage system;

select reliable QOS data to replace the portion of the collected QOS data;

wherein the reliable data is based on QOS data for one or more other storage volumes of the storage system that have a QOS policy similar to a QOS policy for the storage volume and share a resource of the storage system with the storage volume associated with the portion of the collected QOS data;

replace the portion of the collected QOS data with the reliable QOS data for generating an expected range for evaluating future QOS collected data for the storage volume;

use the expected range and the future collected QOS data to identify resource contention associated with one or more resources of the storage system used by the storage volume and identify a bully workload contributing to the resource contention; and

execute a remediation task for reducing the resource contention, the remediation task depending on a computing resource type of the one or more resources in contention.

16. The system of claim 15, wherein the portion of the collected QOS data is noisy when the average number of I/O requests processed within the certain duration is greater than the first threshold value and less than a second threshold value that is set to offset variation due to any random error.

17. The system of claim 15, wherein the reliable QOS data is selected for the one or more other storage volumes that share a resource of a same cluster of the storage system with the storage volume associated with the portion of collected QOS data.

18. The system of claim 15, wherein the expected range is generated using a linear prediction model executed by the performance manager.

19. The system of claim 15, wherein the expected range is generated when the collected QOS data meets a minimal quantity and when the minimal quantity is not met, then one or more statistical values based on the collected QOS data is used for generating the expected range.

20. The system of claim 15, wherein the reliable QOS data of the other one or more storage volumes is selected when the one and more of the other storage volumes and the storage volume associated with the noisy portion of the collected QOS data have similar read/write operation ratios for reading and writing data.