Techniques for monitoring and managing wait queues

• US 8,015,327 B1
• Filed: 09/17/2007
• Issued: 09/06/2011
• Est. Priority Date: 09/17/2007
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Hadavi, M. Michael, Zahavi, William, Klinger, Wolfgang, Davidson, James L., Dunfey, Alexander V.
• Primary Examiner(s): Hafiz; Tariq
• Assistant Examiner(s): LEWIS-TAYLOR, DAYTON A.
• Application Number:
  US11/901,335
• Time in Patent Office: 1,450 Days
• Field of Search: 710/52, 710/53, 710/57
• US Class Current: 710/52
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 2201/88 Monitoring involving counting

Abstract

Described are techniques for managing a wait queue in a system. A plurality of buckets associated with the wait queue are defined. Each of the plurality of buckets is associated with one of more queue depth values and one or more counters. For each received request for service, a current depth of the wait queue indicating a number of other requests included in the wait queue waiting to be serviced is determined, a bucket in accordance with the current depth of the wait queue is selected and information is recorded by updating said one or more counters of the bucket selected. The received request is placed in the wait queue if there is another request currently being serviced or if there is at least one other request currently in the wait queue.

Claims

1. A method for managing a wait queue in a system comprising:

defining a plurality of buckets associated with the wait queue, each of the plurality of buckets being associated with one or more queue depth values and one or more counters, each of the one or more queue depth values associated with said each bucket representing a possible value for a depth of the wait queue at a point in time indicating a total number of requests in the wait queue waiting to be serviced at said point in time, wherein said defining the plurality of buckets and associating the one or more queue depth values and the one or more counters with each of the plurality of buckets are performed prior to receiving requests for service;

receiving said requests for service; and

for each of said requests for service received, performing;

determining a current depth of the wait queue indicating all requests currently included in the wait queue waiting to be serviced;

selecting a bucket from said plurality of buckets based on the current depth of the wait queue and a first of one or more queue depth values associated with the bucket selected, wherein said bucket selected is associated with one or more queue depth values including the first queue depth value equal to the current depth of the wait queue;

recording information by updating said one or more counters of the bucket selected; and

placing said each request in the wait queue if there is another request currently being serviced or if there is at least one other request currently in the wait queue.

2. The method of claim 1, wherein the system is a data storage system and the wait queue is associated with all incoming I/O requests received by the data storage system.

3. The method of claim 1, wherein the one or more counters associated with each bucket include an event count and a cumulative queue depth, the event count representing a number of events associated with said each bucket as selected in said selecting step, the cumulative queue depth representing the sum of queue depths recorded for said each bucket in accordance with each occurrence of said selecting step selectin said each bucket.

4. The method of claim 3, wherein said selecting step selecting a first bucket further comprises:

incrementing by one the event count associated with the first bucket; and

incrementing the cumulative queue depth associated with the first bucket by the current depth of the wait queue.

5. The method of claim 1, further comprising:

reporting, in accordance with a polling interval, collected data, said collected data including values associated with said one or more counters for each of said plurality of buckets and a value indicating an amount of time the system is busy servicing requests.

6. The method of claim 5, further comprising:

determining, for each of said plurality of buckets, an average queue depth using the event count and cumulative queue depth associated with said each bucket.

7. The method of claim 6, further comprising:

determining an average service time for said polling interval, said average service time being determined in accordance with the elapsed time of said polling interval and a total number of requests received during the polling interval, the total number of requests determined by adding the event counts associated with said plurality of buckets.

8. The method of claim 7, further comprising:

determining, for each of said plurality of buckets, an average response time in accordance with the average queue depth for said each bucket and the average service time for said polling interval.

9. The method of claim 8, further comprising:

determining, for each of said plurality of buckets, a percentage of requests included in said each bucket in accordance with said event count for said each bucket and the total number of requests.

11. The method of claim 1, wherein the system is a data storage system and the wait queue is associated with at least one component of the data storage system in connection with servicing I/O requests received by the data storage system which are serviced by the at least one component.

10. A method for managing a wait queue in a system comprising:

defining a plurality of buckets associated with the wait queue, each of the plurality of buckets being associated with one or more queue depth values and one or more counters, each of the one or more queue depth values representing a number of requests in the wait queue waiting to be serviced; and

for each received request for service, performing;

determining a current depth of the wait queue indicating a number of other requests included in the wait queue waiting to be serviced;

selecting a bucket from said plurality of buckets, said bucket being associated with a first of said one or more queue depth values equal to the current depth of the wait queue;

recording information by updating said one or more counters of the bucket selected; and

placing the received request in the wait queue if there is another request currently being serviced or if there is at least one other request currently in the wait queue, wherein the system is a data storage system and the wait queue is associated with all incoming I/O requests received by the data storage system, the one or more counters associated with each bucket include an event count and a cumulative queue depth, the event count representing a number of events associated with said each bucket as selected in said selecting step, the cumulative queue depth representing the sum of queue depths recorded for said each bucket in accordance with each received request selecting said each bucket, and wherein said selecting step selects a first bucket and further comprises;

incrementing by one the event count associated with the first bucket; and

incrementing the cumulative queue depth associated with the first bucket by the current depth of the wait queue; and

wherein the method further comprising;

reporting, in accordance with a polling interval, collected data, said collected data including values associated with said one or more counters for each of said plurality of buckets and a value indicating an amount of time the system is busy servicing requests;

determining, for each of said plurality of buckets, an average queue depth using the event count and cumulative queue depth associated with said each bucket;

determining an average service time for said polling interval, said average service time being determined in accordance with the elapsed time of said polling interval and a total number of requests received during the polling interval, the total number of requests determined by adding the event counts associated with said plurality of buckets;

determining, for each of said plurality of buckets, an average response time in accordance with the average queue depth for said each bucket and the average service time for said polling interval;

determining, for each of said plurality of buckets, a percentage of requests included in said each bucket in accordance with said event count for said each bucket and the total number of requests; and

determining a cumulative percentage value based on a sum of percentages of requests included in two or more buckets representing a range of queue depths associated with the wait queue, a first response time being the average response time associated with a first of said two or more buckets having a maximum queue depth of said range; and

monitoring whether said system is performing in accordance with at least one quality of service level associated with a service agreement response time, said monitoring including comparing said cumulative percentage value to said at least one quality of service level associated with said service agreement response time.

12. A method for monitoring performance of a data storage system comprising:

receiving configuration information for a wait queue, said configuration information defining a plurality of buckets associated with the wait queue, each of the plurality of buckets being associated with one or more queue depth values indicating a number of I/O requests and one or more counters, each of the one or more queue depth values associated with said each bucket representing a possible value for a depth of the wait queue at a point in time indicating a total number of I/O requests in the wait queue waiting to be serviced at said point in time, the wait queue including received I/O requests waiting to be serviced by at least one component of the data storage system;

defining the plurality of buckets and associating the one or more queue depth values and the one or more counters with each of the plurality of buckets prior to receiving I/O requests for service;

receiving said I/O requests for service;

for each of the I/O requests for service received, performing by the data storage system;

determining a current depth of the wait queue indicating all I/O requests currently included in the wait queue waiting to be serviced;

selecting a bucket from said plurality of buckets based on the current depth of the wait queue and a first of one or more queue depth values associated with the bucket selected, wherein said bucket that is selected is associated with one or more queue depth values including the first queue depth value equal to the current depth of the wait queue;

recording information by updating said one or more counters of the bucket selected; and

placing said each I/O request in the wait queue if there is another I/O request currently being serviced, or if there is at least one other I/O request currently in the wait queue;

reporting, by the data storage system in accordance with a polling interval, collected data to a management system, said collected data including values associated with said one or more counters for each of said plurality of buckets and a value indicating an amount of time the at least component is busy servicing I/O requests; and

determining, by the management system for each of said plurality of buckets, an average response time for said polling interval using said collected data for said polling interval.

13. A method for monitoring performance of a data storage system comprising:

receiving configuration information for a wait queue, said configuration information defining a plurality of buckets associated with the wait queue, each of the plurality of buckets being associated with one or more queue depth values indicating a number of I/O requests in the wait queue waiting to be serviced and one or more counters, the wait queue including received I/O requests waiting to be serviced by at least one component of the data storage system;

for each received I/O request to be serviced, performing by the data storage system;

determining a current depth of the wait queue indicating a number of other I/O requests included in the wait queue waiting to be serviced;

selecting a bucket from said plurality of buckets, said bucket being associated with a first of said one or more queue depth values equal to the current depth of the wait queue;

recording information by updating said one or more counters of the bucket selected; and

placing the received I/O request in the wait queue if there is another I/O request currently being serviced, or if there is at least one other I/O request currently in the wait queue;

reporting, by the data storage system in accordance with a polling interval, collected data to a management system, said collected data including values associated with said one or more counters for each of said plurality of buckets and a value indicating an amount of time the at least component is busy servicing I/O requests; and

determining, by the management system for each of said plurality of buckets, an average response time for said polling interval using said collected data for said polling interval;

determining a percentage value indicating a percentage of I/O requests included in two or more buckets for said polling interval, said two or more buckets representing a range of queue depth values associated with the wait queue, a first response time being the average response time associated with a first of said two or more buckets having a maximum queue depth of said range; and

monitoring whether said at least one component of the data storage system is performing in accordance with at least one quality of service level associated with a service agreement response time, said monitoring including comparing said percentage value to said at least one quality of service level associated with said service agreement response time.

14. The method of claim 13, wherein, for said polling interval, said average response time for each of said plurality of buckets and a percentage of I/O requests associated with each of said plurality of buckets is displayed in graphical form at the management system.

15. A computer readable medium comprising code stored thereon for managing a wait queue in a data storage system, the computer readable medium comprising code stored thereon for:

defining a plurality of buckets associated with the wait queue, each of the plurality of buckets being associated with one or more queue depth values and one or more counters, each of the one or more queue depth values associated with said each bucket representing a possible value for a depth of the wait queue at a point in time indicating a total number of requests in the wait queue waiting to be serviced at said point in time, wherein said defining the plurality of buckets and associating the one or more queue depth values and the one or more counters with each of the plurality of buckets are performed prior to receiving requests for service;

receiving said requests for service; and

for each of said requests for service received, performing;

determining a current depth of the wait queue indicating all requests currently included in the wait queue waiting to be serviced;

selecting a bucket from said plurality of buckets based on the current depth of the wait queue and a first of one or more queue depth values associated with the bucket selected, wherein said bucket selected is associated with one or more queue depth values including the first queue depth value equal to the current depth of the wait queue;

recording information by updating said one or more counters of the bucket selected; and

placing said each request in the wait queue if there is another request currently being serviced or if there is at least one other request currently in the wait queue.

16. The computer readable medium of claim 15, wherein the wait queue is associated with incoming I/O requests received by the data storage system.

17. The computer readable medium of claim 15, wherein the one or more counters associated with each bucket include an event count and a cumulative queue depth, the event count representing a number of events associated with said each bucket as selected in said selecting step, the cumulative queue depth representing the sum of queue depths recorded for said each bucket in accordance with each occurrence of said selecting that selects said each bucket, and said code for selecting selects a first bucket and further comprises code for;

incrementing by one the event count associated with the first bucket; and

incrementing the cumulative queue depth associated with the first bucket by the current depth of the wait queue.

18. The computer readable medium of claim 15, further comprising code for:

reporting, in accordance with a polling interval, collected data, said collected data including values associated with said one or more counters for each of said plurality of buckets and a value indicating an amount of time the system is busy servicing requests.

19. A computer readable medium comprising code stored thereon for managing a wait queue in a data storage system, the computer readable medium comprising code stored thereon for:

defining a plurality of buckets associated with the wait queue, each of the plurality of buckets being associated with one or more queue depth values and one or more counters, each of the one or more queue depth values representing a number of requests in the wait queue waiting to be serviced; and

for each received request for service, performing;

determining a current depth of the wait queue indicating a number of other requests included in the wait queue waiting to be serviced;

selecting a bucket from said plurality of buckets, said bucket being associated with a first of said one or more queue depth values equal to the current depth of the wait queue;

recording information by updating said one or more counters of the bucket selected; and

placing the received request in the wait queue if there is another request currently being serviced or if there is at least one other request currently in the wait queue;

reporting, in accordance with a polling interval, collected data, said collected data including values associated with said one or more counters for each of said plurality of buckets and a value indicating an amount of time the system is busy servicing requests;

determining, for each of said plurality of buckets, an average queue depth using the event count and cumulative queue depth associated with said each bucket;

determining an average service time for said polling interval, said average service time being determined in accordance with the elapsed time of said polling interval and a total number of requests received during the polling interval, the total number of requests determined by adding the event counts associated with said plurality of buckets; and

determining, for each of said plurality of buckets, an average response time in accordance with the average queue depth for said each bucket and the average service time for said polling interval.

20. The computer readable medium of claim 19, further comprising code for:

determining, for each of said plurality of buckets, a percentage of requests included in said each bucket in accordance with said event count for said each bucket and the total number of requests;

determining a cumulative percentage value based on a sum of percentages of requests included in two or more buckets representing a range of queue depths associated with the wait queue, a first response time being the average response time associated with a first of said two or more buckets having a maximum queue depth of said range; and

monitoring whether said system is performing in accordance with at least one quality of service level associated with a service agreement response time, said monitoring including comparing said cumulative percentage value to said at least one quality of service level associated with said service agreement response time.