Diagnosing Abnormalities Without Application-Specific Knowledge

• US 20110087924A1
• Filed: 10/14/2009
• Published: 04/14/2011
• Est. Priority Date: 10/14/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Kandula, Srikanth, Agarwal, Sharad, Bahl, Paramivir, Verkaik, Patrick D., Mahajan, Ratul, Padhye, Jitendra D.
• Granted Patent: US 8,392,760 B2
• Time in Patent Office: Days
• US Class Current: 714/26
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36

Abstract

Methods, articles, and systems for determining a probable cause of a component'"'"'s abnormal behavior are described. To determine the probable cause, a computing device computes, for one or more pairs of components having dependency relationships, a likelihood that behavior of one component of a pair is impacting behavior of the other component of the pair. This computing is based on joint historical behavior of the pair of components. The computing device then determines that one of a plurality of components is a probable cause of the abnormal behavior based on the computed likelihoods.

Claims

1. A system comprising:

a processor; and

a graph component configured to be operated by the processor to generate a dependency graph for a plurality of network components, each network component represented as a node in the dependency graph and each dependency relationship between a pair of network components represented as an edge between two nodes;

a diagnostic component configured to be operated by the processor to perform operations including;

detecting abnormal behavior of a first network component of the plurality of network components based on behavior metrics of the first network component and historical values of those behavior metrics;

for each pair of network components of the plurality of network components having a dependency relationship, computing a likelihood that behavior of one network component of the pair is impacting behavior of the other network component of the pair, the computing being based on joint historical behavior of the pair of network components; and

determining that a second network component of the plurality of network components is a probable cause of the detected abnormal behavior of the first network component based on the computed likelihoods.

2. The system of claim 1, wherein the computing is agnostic with respect to semantics of behavior metrics comprising the joint historical behavior.

3. A method implemented by a computing device, the method comprising:

for each pair of components of a plurality of components having a dependency relationship, computing a likelihood that behavior of one component of the pair is impacting behavior of the other component of the pair, the computing being based on joint historical behavior of the pair of components; and

determining that one of the plurality of components is a probable cause of abnormal behavior of at least one other component of the plurality of components based on the computed likelihoods.

4. The method of claim 3, further comprising detecting abnormal behavior of the at least one other component based on behavior metrics of the component and historical values of those behavior metrics.

5. The method of claim 3, further comprising periodically receiving a plurality of behavior metrics for one or more of the plurality of components, the one or more of the components each having a state represented as a multi-variable vector of behavior metrics.

6. The method of claim 5, wherein the behavior metrics include computer processing unit utilization, memory usage, disk usage, amount of network traffic and other input and output, response times, loss rate, delay, and key-value pairs.

7. The method of claim 3, wherein the components include application processes, operating systems, network paths, configuration files, and virtual components that represent collective behavior of communication peers of other components that are network components.

8. The method of claim 3, further comprising generating a dependency graph for the plurality of components, each component represented as a node in the dependency graph and each dependency relationship between a pair of components represented as an edge between two nodes.

9. The method of claim 8, wherein the generating further comprises automatically generating the dependency graph using a plurality of templates, each template associated with a different component type.

10. The method of claim 8, wherein the computed likelihoods are assigned as edge weights to edges of the dependency graph, and each edge weight is defined as:

11. The method of claim 10, further comprising, if either or both of S and D is performing normally at the time when the abnormality is detected, assigning a low weight to E(S→

D).

12. The method of claim 10, wherein the difference between two states is the difference between vectors associated with the two states, each vector having L variables, the difference being defined as:

Σ

_i=1^L|dⁱ|/L

where dⁱ is the difference of the i-th variable normalized over the joint historical behavior by performing;

dⁱ=(v_tkⁱ−

v_nowⁱ)/(v_maxⁱ−

v_minⁱ)

where v_tkⁱ and v_nowⁱ are values of a variable v at two different times and v_maxⁱ and v_minⁱ are maximum and minimum observed values of v found in the joint historical behavior.

13. The method of claim 3, wherein the determining further comprises determining a ranking of components that are probable causes of the abnormal behavior.

14. The method of claim 13, wherein the ranking of the components is defined as:

Rank(c→

e)∝

(I(c→

e)×

S(c))<sup>−

1</sup>

where c and e are components having a direct or indirect dependency relationship, I(c→

e) is the maximum weight W(p) of acyclic paths p from c to e, and W(p) is further defined as;

15. The method of claim 3, further comprising utilizing results of the determination to remedy the abnormal behavior or inform an operator, the utilizing including one or more of:

providing an indication of the determined component to the operator;

providing an indication of the determined component to a rules-based engine configured to perform remedial measures;

providing a user interface to the operator that is configured to enable the operator to view details associated with the computing and the determining;

or

providing an indication of the determined component to a knowledge base of remedies to enable the knowledge base to suggest a remedy to the operator.

16. An article of manufacture comprising:

one or more computer-readable storage media storing computer executable instructions that, when executed by a computing device, cause the computing device to perform operations including;

detecting abnormal behavior of a first network component of a plurality of network components based on behavior metrics of the first network component and historical values of those behavior metrics;

for each pair of the network components having a dependency relationship, computing a likelihood that behavior of one network component of the pair is impacting behavior of the other network component of the pair, the computing being based on joint historical behavior of the pair of network components; and

determining that a second network component of the network components is a probable cause of abnormal behavior of the first network component based on the computed likelihoods.

17. The article of claim 16, wherein the detecting comprises calculating, for each behavior metric, a mean and standard deviation of the historical values of that behavior metric, and utilizing the behavior metric, the mean, and the standard deviation in an error function defined as:

18. The article of claim 16, wherein the computing further comprises:

utilizing states of each of the pair of network components at different points in time to calculate the likelihood, each state defined by a multi-variable vector that includes multiple variables associated with multiple behavior metrics of one of the network components; and

filtering out or de-emphasizing a portion of the variables the multi-variable vectors.

19. The article of claim 18, wherein a variable is de-emphasized if the variable indicates normal behavior.

20. The article of claim 18, wherein a variable is filtered out if the variable is redundant with respect to other variables, is irrelevant with respect to interaction between network components, or is an aggregate variable that aggregates other variables.