Low-impact application-level performance monitoring with minimal and automatically upgradable instrumentation in a storage system

• US 10,152,232 B1
• Filed: 06/29/2016
• Issued: 12/11/2018
• Est. Priority Date: 06/29/2016
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Kleiner, Vladimir, Shoikhet, Kirill, Miron, Amir, Kucherov, Anton, Veprinsky, Alexandr
• Primary Examiner(s): Chappell, Daniel C
• Application Number:
  US15/196,674
• Time in Patent Office: 895 Days
• CPC Class Codes: G06F 11/14 Error detection or correction of the data by redundancy in operation G06F11/16 takes precedence
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 2201/88 Monitoring involving counting

Abstract

Described embodiments provide methods and systems for tracking performance of a storage system including one or more system resources. One or more threads may be generated to operate the storage system. Each of the one or more threads may be associated with one of a plurality of tasks of the storage system. When an operating state of one of the one or more threads is changed, one or more performance counters of the storage system may be updated. A performance snapshot may be generated by capturing a state of the tasks associated with the threads and the performance counters.

Claims

1. A method comprising:

generating one or more threads to operate a storage system, the storage system comprising one or more system resources, and each of the one or more threads associated with one of a plurality of tasks of the storage system;

updating one or more performance counters of the storage system in response to a change in operating state of one of the one or more threads; and

generating a performance snapshot by capturing a state of the tasks associated with the threads and the performance counters wherein generating the performance snapshot comprises;

grouping one or more of the threads that are associated with a given task;

grouping counter values of the one or more performance counters into containers associated with the given task; and

generating a performance snapshot based upon the grouped one or more threads that are associated with the given task, and the containers associated with the given task,

wherein generating the performance snapshot further comprises generating a directed-graph associated with the given task, the directed-graph comprising a plurality of nodes and at least one edge joining associated ones of the nodes,

wherein each of the nodes is associated with a synchronization object of an operating system of the storage system,

wherein each of the nodes is identified by a two-tuple node key based upon a task identifier associated with the node, and a resource identifier associated with the node, and

wherein each of the edges is associated with a respective sequence of instructions.

2. The method of claim 1 wherein the one or more threads to operate a storage system comprises more than one thread associated with one of the plurality of tasks.

3. The method of claim 1, further comprising identifying each edge by a three-tuple edge key based upon a task identifier associated with the edge, a first resource identifier

associated with a first node associated with the edge, and a second resource identifier associated with a second node associated with the edge.

4. The method of claim 1, wherein the operating state comprises one of running, ready and suspended.

5. The method of claim 1, further comprising:

collecting, by one or more performance counters, application-level performance data with low-impact instrumentation, and

updating firmware associated with the one or more performance counters by updating software associated with the storage system.

6. The method of claim 1, further comprising associating the one or more performance counters with the given task, the one or more performance counters comprising one or more of:

a counter for accumulating a number of accesses of an associated resource by the given task, a counter for accumulating a number of requested units from the associated resource by the given task, a counter for accumulating a time of the given task in a suspended state, a counter for determining a number of suspended requests to the associated resource by the given task, and performance counters associated with sequences of processing instructions comprising one or more of;

a counter for accumulating a number of passes of the given task, a counter for accumulating a time of one or more operating states of the given task, and a counter for accumulating a number of cooperative yields of the given task.

7. The method of claim 1, wherein generating a performance snapshot further comprises generating the performance snapshot at a periodic snapshot interval while concurrently operating the storage system, wherein the snapshot interval is based upon a target time-precision of the performance snapshot and available resources of the storage system.

8. A system comprising:

a processor; and

memory storing computer program code that when executed on the processor causes the processor to operate a storage system, the storage system operable to perform the operations of;

generating one or more threads to operate a storage system, the storage system comprising one or more system resources, and each of the one or more threads associated with one of a plurality of tasks of the storage system;

updating one or more performance counters of the storage system in response to a change in operating state of one of the one or more threads; and

generating a performance snapshot by capturing a state of the tasks associated with the threads and the performance counters, wherein generating the performance snapshot comprises;

grouping one or more of the threads that are associated with a given task;

grouping counter values of the one or more performance counters into containers associated with the given task; and

generating a performance snapshot based upon the grouped one or more threads that are associated with the given task and the containers associated with the given task,

wherein generating the performance snapshot further comprises generating a directed-graph associated with the given task, the directed-graph comprising a plurality of nodes and at least one edge joining associated ones of the nodes,

wherein each of the nodes is associated with a synchronization object of an operating system of the storage system,

wherein each of the nodes is identified by a two-tuple node key based upon a task identifier associated with the node, and a resource identifier associated with the node, and

wherein each of the edges is associated with a respective sequence of instructions.

9. The system of claim 8, wherein the storage system is further operable to perform the operations of:

collecting, by one or more performance counters, application-level performance data with low-impact instrumentation, and

updating firmware associated with the one or more performance counters by updating software associated with the storage system.

10. The system of claim 8, wherein the storage system is further operable to perform the operations of:

associating the one or more performance counters with the given task, the one or more performance counters comprising one or more of;

a counter for accumulating a number of accesses of an associated resource by the given task, a counter for accumulating a number of requested units from the associated resource by the given task, a counter for accumulating a time of the given task in a suspended state, a counter for determining a number of suspended requests to the associated resource by the given task, and performance counters associated with sequences of processing instructions comprising one or more of;

a counter for accumulating a number of passes of the given task, a counter for accumulating a time of one or more operating states of the given task, and a counter for accumulating a number of cooperative yields of the given task; and

generating the performance snapshot at a periodic snapshot interval while concurrently operating the storage system, wherein the snapshot interval is based upon a target time-precision of the performance snapshot and available resources of the storage system.

11. A computer program product including a non-transitory computer readable storage medium having computer program code encoded thereon that when executed on a processor of a computer causes the computer to operate a storage system, the computer program product comprising:

computer program code for generating one or more threads to operate a storage system, the storage system comprising one or more system resources, and each of the one or more threads associated with one of a plurality of tasks of the storage system;

computer program code for updating one or more performance counters of the storage system in response to a change in operating state of one of the one or more threads; and

computer program code for generating a performance snapshot by capturing a state of the tasks associated with the threads and the performance counters,

wherein generating the performance snapshot comprises;

grouping one or more of the threads that are associated with a given task;

grouping counter values of the one or more performance counters into containers associated with the given task; and

generating a performance snapshot for each task based upon the grouped one or more threads and the containers associated with the given task,

wherein generating a performance snapshot further comprises generating a directed-graph associated with the given task, the directed-graph comprising a plurality of nodes and at least one edge joining associated ones of the nodes,

wherein each of the nodes is associated with a synchronization object of an operating system of the storage system,

wherein each of the nodes is identified by a two-tuple node key based upon a task identifier associated with the node, and a resource identifier associated with the node, and

wherein each of the edges is associated with a respective sequence of instructions.

12. The computer program product of claim 11, wherein the one or more performance counters are operable to collect application-level performance data with low-impact instrumentation, and wherein firmware associated with the one or more performance counters is updated with updates to software associated with the storage system, and wherein the computer program code for generating a performance snapshot further comprises computer program code for generating the performance snapshot at a periodic snapshot interval while concurrently operating the storage system, wherein the snapshot interval is based upon a target time-precision of the performance snapshot and available resources of the storage system.