DIAGNOSIS OF APPLICATION PERFORMANCE PROBLEMS VIA ANALYSIS OF THREAD DEPENDENCIES

• US 20090320021A1
• Filed: 06/19/2008
• Published: 12/24/2009
• Est. Priority Date: 06/19/2008
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Cao, Jiaxin, Wang, Jiajie, Li, Zituo, Zhu, Bin Benjamin, Pan, Aimin
• Application Number:
  US12/141,948
• Publication Number:
  US 20090320021A1
• Time in Patent Office: Days
• US Class Current: 718/100
• CPC Class Codes: G06F 11/0715 in a system implementing multitasking multitasking per se G06F9/46
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/3612 by runtime analysis performance monitoring G06F11/3466
• CPC Class Codes: G06F 2201/865 Monitoring of software

Abstract

A “Performance Evaluator” provides various techniques for tracking system events to diagnose root causes of application performance anomalies. In general, traces of system events involved in inter-thread interactions are collected at application runtime. These traces are then used to construct inter-thread dependency patterns termed “control patterns.” Control patterns are then evaluated to determine root causes of performance anomalies. Where an application terminates abnormally or full traces cannot be collected for some reason, partial control patterns are constructed for that application. In various embodiments, “fingerprints” are then generated from full or partial control patterns and are matched to fingerprints corresponding to operations in other control patterns extracted from reference traces collected on the same or similar systems. Matched fingerprints or control patterns are then used to deduce the root cause of application performance anomalies associated with full or partial traces.

Claims

1. A system for constructing control patterns for use in diagnosing root causes of performance problems in applications, comprising:

a device for monitoring events and inter-thread interactions of a task during execution of the task using instrumentation techniques;

a device for recording trace data including the events and inter-thread interactions of the task for a period of interest; and

a device for evaluating the recorded trace data to construct control patterns, said control patterns including identifications of all critical participant threads and causal relations of operations that happened in those threads during the period of interest.

2. The system of claim 1 wherein the instrumentation techniques further comprise a device for instrumenting a set of kernel functions for collecting the trace data for the task.

3. The system of claim 1 further comprising a device for evaluating the control patterns to determine one or more root causes of a performance problem that occurred during the execution of the task.

4. The system of claim 3 wherein evaluating the recorded trace data to construct the control patterns further comprises:

identifying as “

significant operations”

all operations represented in the trace data whose duration exceed a predetermined operation time threshold;

recursively evaluating all significant operations to identify all causal operations of each significant operation; and

determining a total period of delay for each causal operation.

5. The system of claim 4 wherein determining the root causes as being responsible for the performance problem further comprises:

identifying the causal operation being responsible for the largest total period of delay as a root cause; and

identifying one or more additional causal operations if there are other large delay periods which are not covered in the largest total period of delay.

6. The system of claim 1 further comprising:

comparing one or more of the control patterns to a database of pre-evaluated control patterns to determine whether a match to the control patterns is in the database, said database including one or more root causes of performance problems for each pre-evaluated control pattern in the database; and

outputting root causes of any performance problem corresponding to a match to the control patterns.

7. The system of claim 1 where the recorded trace data contains only a partial control pattern, and further comprising:

comparing the partial control pattern with control patterns constructed from subsequently recorded trace data; and

wherein if the partial control pattern matches a control pattern from subsequently recorded trace data, the matched control pattern from the subsequently recorded trace data is further evaluated to diagnose one or more root causes of a performance problem that occurred during the execution of the task.

8. The system of claim 1 further comprising:

constructing a first control pattern fingerprint for one or more of the control patterns or partial control patterns;

comparing the first control pattern fingerprint to a database of control pattern fingerprints to determine whether there is a match to the first control pattern fingerprint; and

outputting root causes of any performance problem corresponding to the any match to the first control pattern fingerprint.

9. A method for constructing control patterns for use identifying critical participant threads and causal relations of operations during execution of an application, comprising steps for:

using instrumentation elements to monitor events and inter-thread interactions associated with a task during execution of the task;

identifying a period of interest during execution of the task;

recording trace data including the events and inter-thread interactions of the task for the period of interest;

constructing one or more control patterns from the recorded trace data, said control patterns including identifications of all critical participant threads and causal relations of operations that happened in those threads during the period of interest; and

evaluating the control patterns to determine one or more root causes of a performance problem that occurred during the execution of the task.

10. The method of claim 9 wherein the instrumentation elements include steps for instrumenting a set of kernel functions for collecting the trace data for the task during the period of interest.

11. The method of claim 9 wherein constructing the control patterns from the recorded trace data further comprises:

identifying as “

significant operations”

all operations represented in the trace data whose duration exceed a predetermined operation time threshold;

recursively evaluating all significant operations to identify all causal operations of each significant operation; and

determining a total period of delay for each causal operation.

12. The method of claim 11 wherein determining the root causes as being responsible for the performance problem further comprises:

identifying the causal operation being responsible for the largest total period of delay as a root cause; and

identifying one or more additional causal operations as additional root causes if those causal operations have corresponding delay periods which are longer than a predetermined threshold and which are not covered in the largest total period of delay.

13. The method of claim 9 further comprising:

comparing one or more of the control patterns to a database of pre-evaluated control patterns to determine whether a match to the control patterns is in the database, said database including one or more root causes of performance problems for each pre-evaluated control pattern in the database; and

outputting root causes of any performance problem corresponding to a match to the control patterns.

14. The method of claim 9 wherein the recorded trace data contains only enough information to construct a partial control pattern, and further comprising:

comparing the partial control pattern with control patterns constructed from subsequently recorded trace data; and

wherein if the partial control pattern matches a control pattern from the subsequently recorded trace data, the matched control pattern from the subsequently recorded trace data is further evaluated to diagnose one or more root causes of the performance problem that occurred during the execution of the task.

15. The method of claim 9 wherein starting the recording of the trace data for the period of interest is manually triggered via a user interface.

16. A computer-readable medium having computer-executable instructions stored thereon for diagnosing root causes of performance problems in applications, said instructions comprising:

buffering system-level and application-level trace data including events and inter-thread interactions related to execution of a particular task;

automatically initiating a recording of the trace data, including any buffered trace data whenever any operation associated with the task does not complete within a predetermined time period;

constructing one or more control patterns from the recorded trace data, said control patterns including identifications of all critical participant threads and causal relations of operations that happened in those threads during the period of interest; and

evaluating the control patterns to determine one or more root causes of a performance problem that occurred during the execution of the task.

17. The computer-readable medium of claim 16 wherein the recording is manually initiated via a user interface when a user suspects that a performance anomaly is occurring, whether or not any operation associated with the task does not complete within the predetermined time period.

18. The computer-readable medium of claim 16 wherein constructing the control patterns from the recorded trace data further comprises:

identifying as “

significant operations”

all operations represented in the trace data whose duration exceed a predetermined operation time threshold;

recursively evaluating all significant operations to identify all causal operations of each significant operation; and

determining a total period of delay for each causal operation.

19. The computer-readable medium of claim 18 wherein determining the root causes as being responsible for the performance problem further comprises:

identifying the causal operation being responsible for the largest total period of delay as a root cause; and

identifying one or more additional causal operations as additional root causes if those causal operations have corresponding delay periods which are longer than a predetermined threshold and which are not covered in the largest total period of delay.

20. The computer-readable medium of claim 16 wherein the recorded trace data contains only enough information to construct a partial control pattern, and further comprising:

comparing the partial control pattern with control patterns constructed from subsequently recorded trace data; and

wherein if the partial control pattern matches a control pattern from the subsequently recorded trace data, the matched control pattern from the subsequently recorded trace data is further evaluated to diagnose one or more root causes of the performance problem that occurred during the execution of the task.