Modularized collaborative performance issue diagnostic system

• US 10,565,045 B2
• Filed: 06/28/2017
• Issued: 02/18/2020
• Est. Priority Date: 06/28/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kirshenbaum, Alexander, Dietrich, Aaron Edward, Cohen, Jason L.
• Primary Examiner(s): Contino, Paul
• Application Number:
  US15/636,259
• Publication Number:
  US 20190004884A1
• Time in Patent Office: 965 Days
• CPC Class Codes: G06F 11/0706 the processing taking place on a specific hardware platform or in a specific software environment
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0778 Dumping, i.e. gathering error/state information after a fault for later diagnosis
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/3003 Monitoring arrangements specially adapted to the computing system or computing system component being monitored
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3055 Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction by redundancy in operation G06F11/14; error detection or correction by redundancy in hardware G06F11/16
• CPC Class Codes: G06F 11/3409 for performance assessment

Abstract

A collaborative diagnostic system monitors events in a system and identifies a causality chain from a detected performance issue to the root cause of that performance issue. The collaborative diagnostic system includes multiple issue detectors, multiple analysis core modules, and multiple scenario modules that work together to identify the causality chain and root cause. Each issue detector is a module or component that includes logic to detect known behaviors in the system, such as performance issues in the system. Each analysis core module includes logic to analyze and correlate low level system behavior within a conceptual area. Within each analysis core module are one or more diagnostic modules that are specific to that analysis core module to help determine what is happening in the system. Each scenario module includes logic to take an appropriate responsive action in response to the root cause of a performance issue being determined.

Claims

1. A method, comprising:

detecting, in a computing system, a performance issue in the computing system based on a set of rules or criteria;

in response to detecting the performance issue,

analyzing, in the computing system, one or more corresponding system behaviors to identify a root cause for the detected performance issue in the computing system, wherein the analyzing one or more corresponding system behaviors includes;

receiving a query identifying the detected performance issue;

in response to receiving the query identifying the detected performance issue, determining a corresponding system behavior at a time of the detected performance issue based on an event log containing records of tasks performed in the computing system at various times;

based on the determined corresponding system behavior, issuing another query to determine another corresponding system behavior that causes the determined system behavior based on the event log; and

repeating the determining and issuing operations using the another corresponding system behavior until the root cause for the detected performance issue is identified; and

performing, in the computing system, a responsive action in response to the identified root cause for the detected performance issue.

2. The method of claim 1, wherein performing the responsive action comprising presenting an indication of the root cause of the detected performance issue.

3. The method of claim, 2, wherein performing the responsive action further comprising presenting a causality chain for the detected performance issue.

4. The method of claim 1, wherein the responsive action comprises turning on logging of event data for an application running in the computing system.

5. The method of claim 1, wherein analyzing the one or more corresponding system behavior includes performing behavior-specific analysis for a conceptual area of the computing system.

6. A method performed in a computing system having a processor executing multiple processes and providing multiple analysis modules individually configured to analyze and correlate system behaviors in the computing system, the method comprising:

detecting, in a computing system, a performance issue of executing the one or more processes in the computing system based on a set of rules or criteria;

in response to detecting the performance issue in the computing system,

issuing a query to an analysis module for identifying a system behavior related to an operation performed by one of the processes at a time of the detected performance issue in the computing system based on an event log having records individually containing metadata describing system events occurred in the computing system at various times;

receiving, from the analysis module, the identified system behavior of the operation performed by one of the processes at the time of the detected performance issue in the computing system;

based on the identified system behavior of the operation performed by the one of the processes, issuing another query to the analysis module or another analysis module for determining another system behavior of another operation that causes the identified system behavior; and

repeating the issuing and receiving operations using the determined another system behavior to determine one or more additional system behaviors until a root cause for the detected performance issue is identified; and

performing, in the computing system, a responsive action in response to the identified root cause for the detected performance issue.

7. The method of claim 6, wherein:

the system behavior is a first system behavior; and

the method further includes;

identifying another operation performed by another one of the processes at the time of the detected performance issue in the computing system based on the event log; and

based on the identified another operation, determining a second system behavior that caused the first system behavior.

8. The method of claim 6, wherein the individual system events include an inter-process call, a file I/O access by a process, a call by a process to an operating system in the computing system, or a network access by a process in the computing system.

9. The method of claim 6, further comprising:

forming a causality chain having one end representing the detected performance issue, another end representing the root cause, and one or more intermediate nodes individually representing one of the determined system behaviors.

10. The method of claim 6, further comprising:

forming a causality chain having one end representing the detected performance issue, another end representing the root cause, and one or more intermediate nodes individually representing one of the determined system behaviors, at least one of the nodes being connected to multiple branches of the causality chain.

11. The method of claim 6, wherein detecting the performance issue includes analyzing metadata of a sequence of system events in the event log based on one or more conditions.

12. The method of claim 6, wherein determining the system behavior includes determining the system behavior that causes the operation performed by the process in the computing system to be late.

13. The method of claim 6, wherein determining the system behavior includes determining the system behavior that causes the operation performed by the process in the computing system to be late, and wherein the method further includes identifying another one of the processes that owned the identified operation and determining another system behavior that causes execution of the another one of the processes to be late.

14. A computing system, comprising:

a processor; and

a memory having instructions executable by the processor to provide multiple processes and analysis modules individually configured to analyze and correlate system behaviors in the computing system and to cause the computing system to;

upon detecting, in the computing system, a performance issue of executing the one or more processes in the computing system based on a set of rules or criteria;

issue a query to an analysis module for determining a corresponding system behavior at a time of the detected performance issue in the computing system based on an event log having records individually containing metadata regarding system events occurred in the computing system at various times;

receive, from the analysis module, the determined corresponding system behavior;

based on the received corresponding system behavior, issue another query to another analysis module to determine another corresponding system behavior that causes the determined system behavior;

repeat the determining and issuing operations using the determined another corresponding system behavior until a root cause for the detected performance issue is identified; and

perform, in the computing system, a responsive action in response to the identified root cause for the detected performance issue.

15. The computing system of claim 14, wherein:

the system behavior is a first system behavior; and

the memory includes additional instructions executable by the processor to cause the computing system to;

identify another operation performed by another one of the processes at the time of the detected performance issue in the computing system based on the event log; and

based on the identified another operation, determine a second system behavior that caused the first system behavior.

16. The computing system of claim 14, wherein the individual system events include an inter-process call, a file I/O access by a process, a call by a process to an operating system in the computing system, or a network access by a process in the computing system.

17. The computing system of claim 14, the memory includes additional instructions executable by the processor to cause the computing system to:

form a causality chain having one end representing the detected performance issue, another end representing the root cause, and one or more intermediate nodes individually representing one of the determined system behaviors.

18. The computing system of claim 14, the memory includes additional instructions executable by the processor to cause the computing system to:

form a causality chain having one end representing the detected performance issue, another end representing the root cause, and one or more intermediate nodes individually representing one of the determined system behaviors, at least one of the nodes being connected to multiple branches of the causality chain.

19. The computing system of claim 14, wherein to detect the performance issue includes to analyze metadata of a sequence of system events in the event log based on one or more conditions.

20. The computing system of claim 14, wherein to determine the system behavior includes to determine the system behavior that causes the operation performed by the process in the computing system to be late, and wherein the memory includes additional instructions executable by the processor to cause the computing system to identify another one of the processes that owned the identified operation and determining another system behavior that causes execution of the another one of the processes to be late.