Automated root cause analysis of problems associated with software application deployments

• US 8,001,527 B1
• Filed: 12/21/2005
• Issued: 08/16/2011
• Est. Priority Date: 12/21/2004
• Status: Active Grant

• Current Assignee: Citrix Systems, Inc. (Cloud Software Group)
• Original Assignee: Zenprise, Inc. (Cloud Software Group)
• Inventors: Bala, Gregory Paul, Hassan, Tanvir, Roach, Kelly Brian, Qureshi, Waheed
• Primary Examiner(s): Puente; Emerson
• Assistant Examiner(s): Labud; Jonathan R
• Application Number:
  US11/316,367
• Time in Patent Office: 2,064 Days
• Field of Search: 717/100, 717/104, 717/107, 717/108, 717/120, 717/121, 717/124, 717/126, 717/127, 717/130, 717/131, 714/20, 714/25, 714/40, 714/48, 707/103
• US Class Current: 717/120
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06N 5/048 Fuzzy inferencing

Abstract

Computer systems and methods are disclosed for managing a deployment of a software application. One system includes an application model describing the deployment, the application model comprising a representation of physical and logical objects in a domain of the deployment, configuration data about the objects, and information about relationships between the objects. The system also includes a root cause analysis module configured to identify one or more problematic objects of the application model, and to use pattern-recognition on the application model to find root cause candidates that may be a root cause of one or more problems associated with the problematic objects. The root cause analysis module can be further configured to apply diagnostic unit tests on one or more objects associated with the root cause candidates, the diagnostic unit tests configured to narrow down a list of possible root causes of the problems. Various pattern-recognition techniques are disclosed, including looking for recent property or configurational changes in the application model, clustering of problematic objects, examining links between objects in the application model, comparisons between pairs of non-problematic objects as well as between problematic objects and non-problematic objects, temporal comparisons of the state of the application model, and examining objects that are near the problematic objects in the application model.

Claims

1. A computer-implemented method of finding a root cause of one or more problems with a deployment of a software application within a managed computer system, comprising:

discovering, after installation of the deployment, information about the deployment and software services that are not provided by the deployment, by sending software probes onto a communication network;

using the discovered information to create an application model describing the deployment, the application model comprising;

an object graph representing hardware and software elements of the managed computer system, the elements represented as objects of the graph;

configuration data about the elements; and

information about relationships between the elements;

dynamically modifying the application model on an ongoing basis in response to detected changes of configuration of the deployment;

accessing a repository of logic rules generated from analyses of reference materials for the software application and empirical analyses done on a plurality of different deployments of the software application, each rule corresponding to a known problem associated with the software application, each rule configured to be used to test for satisfaction of one or more possible conditions of the deployment;

using at least one of the rules to determine that one or more of the possible conditions are satisfied by the deployment;

marking objects of the application model that are associated with the satisfied conditions; and

using pattern-recognition on a portion of the application model that includes some of the marked objects to identify one or more root cause candidates associated with the marked objects, the root cause candidates comprising elements of the application model, at least some of the root cause candidates not being marked objects;

wherein said discovering, using the discovered information, dynamically modifying, accessing, using at least one of the rules, marking, and using pattern-recognition are executed based on code that is separate from code of the software application of the managed computer system.

2. The method of claim 1, further comprising applying diagnostic unit tests on one or more objects associated with the root cause candidates, the diagnostic unit tests configured to narrow down a list of possible root causes of the problems.

3. The method of claim 2, further comprising:

providing an encoded representation of a source of knowledge associated with the software application, the encoded representation mapping a combination of possible features of the deployment of the software application to a known problem;

from said use of pattern-recognition or from said diagnostic unit tests, detecting one or more of the features in the deployment; and

based on the detected one or more features and the encoded representation of the knowledge source, programmatically evaluating whether the problem exists within the deployment.

4. The method of claim 2, further comprising using results of said pattern-recognition or said diagnostic unit tests to report elements of the application model that have been ruled out as possible root causes.

5. The method of claim 1, wherein said marking comprises:

identifying an object of the application model by mathematically modeling normal behavior of the object to detect anomalous behavior of a hardware or software element represented by the object; and

marking the identified object.

6. The method of claim 1, wherein using pattern-recognition comprises:

comparing one of the marked objects of the object graph with other objects of the object graph that are of identical type to that of the marked object, to find a feature of the marked object that is dissimilar to corresponding features of the other objects with which the marked object is compared; and

identifying the dissimilar feature as a root cause candidate.

7. The method of claim 1, wherein using pattern-recognition comprises:

determining that one of the marked objects has had a recent configurational change compared to nearby objects in the object graph; and

identifying one of said hardware or software elements corresponding to the object having had the recent configurational change, and designating the identified element as a root cause candidate.

8. The method of claim 1, wherein using pattern-recognition comprises grouping the marked objects into clusters by k-means clustering, using numerical measurements from the deployment as cluster analysis dimensions.

9. The method of claim 8, wherein using pattern-recognition further comprises looking for configurational relationships between the objects of one of the clusters.

10. The method of claim 8, wherein using pattern-recognition further comprises looking for temporal relationships between the objects of one of the clusters.

11. The method of claim 8, wherein using pattern-recognition further comprises conducting diagnostic troubleshooting unit tests on objects within the clusters.

12. The method of claim 1, wherein using pattern-recognition comprises:

finding an upstream-most object of a dependency chain of marked objects, such that the upstream-most object does not depend on an adjacent marked object in the chain; and

identifying one of said hardware or software elements corresponding to the upstream-most object, and designating the identified element as a root cause candidate.

13. The method of claim 1, wherein using pattern-recognition comprises:

finding a candidate marked object having a directed link to a plurality of marked objects;

determining that there are no marked objects having directed links to the candidate marked object; and

identifying one of said hardware or software elements corresponding to the candidate marked object, and designating the identified element as a root cause candidate.

14. The method of claim 1, wherein using pattern-recognition comprises:

finding a candidate object having a directed link to one or more marked objects;

determining that there are no marked objects having directed links to the one or more marked objects to which the candidate object has directed links; and

identifying one of said hardware or software elements corresponding to the candidate object, and designating the identified element as a root cause candidate.

15. The method of claim 1, wherein using pattern-recognition comprises:

finding a first candidate object having a directed link to one or more marked objects;

finding a second candidate object having directed links to the one or more marked objects and directed links to one or more non-marked objects; and

identifying one of said hardware or software elements corresponding to the first candidate object, and designating the identified element as a root cause candidate.

16. The method of claim 1, wherein the application model further comprises a temporal history of a state of a marked object of the object graph, and wherein using pattern-recognition comprises using the temporal history to compare a current state of the marked object with one or more previous states of the marked object.

17. A computer program which embodies the method of claim 1, represented in computer storage.

18. A meta-application system for managing a computer system including a deployment of a software application, comprising:

a repository of logic rules generated from analyses of reference materials for the software application and empirical analyses done on a plurality of different deployments of the software application, each rule corresponding to a known problem associated with the software application, each rule configured to be used to test for satisfaction of one or more possible conditions of the deployment;

an application model describing the deployment, the application model comprising;

an object graph representing hardware and software elements of the managed computer system, the elements represented as objects of the object graph;

configuration data about the elements; and

information about relationships between the elements;

a discovery component configured to conduct, after installation of the deployment, automated discovery of information about at least the deployment, by sending software probes onto a communication network, the discovery component configured to discover software services that are not provided by the deployment or the meta-application system, the discovery component configured to use the discovered information to create the application model, wherein the discovery component is configured to, after creating the application model, dynamically modify the application model on an ongoing basis in response to detected changes of configuration of the deployment;

an analysis subsystem comprising;

a problem detector configured to use one or more of the rules to determine that one or more of the possible conditions of the rules is satisfied by the deployment; and

a root cause analysis module configured to;

mark application model objects that (1) are associated with the satisfied conditions of the rules, and/or (2) correspond to elements in the deployment whose behavior has deviated from mathematical models created by the meta-application system; and

use pattern-recognition on a portion of the application model that includes the marked objects to find a root cause candidate that may be a root cause of one or more problems associated with one or more of the marked objects of the object graph, at least some of the root cause candidates not being marked objects; and

one or more computer systems operative to implement the application model, the discovery component, and the root cause analysis module;

wherein the application model, discovery component, and root cause analysis module comprise code that is separate from code of the software application of the managed computer system.

19. The meta-application system of claim 18, wherein the root cause analysis module is further configured to apply diagnostic unit tests on hardware or software elements that correspond to one or more of the graph'"'"'s objects that are associated with the root cause candidates, the diagnostic unit tests configured to narrow down a list of possible root causes of the problems.

20. The meta-application system of claim 19, further comprising an encoded representation of a source of knowledge associated with the software application, the encoded representation mapping a logical combination of possible features of the deployment of the software application to a known problem, wherein the root cause analysis module is configured to detect, from said use of pattern-recognition or from said diagnostic unit tests, one or more of the features in the deployment or additional marked objects of the application model, the meta-application system configured to programmatically evaluate whether the problem exists within the deployment, based on the detected one or more features and the encoded representation of the knowledge source.

21. The meta-application system of claim 19, wherein the root cause analysis module is configured to use results of said pattern-recognition or said diagnostic unit tests to report elements of the application model that have been ruled out as possible root causes.

22. The meta-application system of claim 18, wherein the root cause analysis module is configured to use a pattern-recognition method comprising:

comparing one of the marked objects of the object graph with other objects of the object graph that are of identical type to that of the marked object, to find a feature of the marked object that is dissimilar to corresponding features of the other objects with which the marked object is compared; and

identifying the dissimilar feature as a root cause candidate.

23. The meta-application system of claim 18, wherein the root cause analysis module is configured to use a pattern-recognition method comprising:

determining that one of the marked objects has had a recent configurational change compared to nearby objects in the object graph; and

identifying one of said represented hardware or software elements corresponding to the object having had the recent configurational change, and designating the identified element as a root cause candidate.

24. The meta-application system of claim 18, wherein the root cause analysis module is configured to use a pattern-recognition method comprising grouping the marked objects into clusters by k-means clustering, wherein numerical measurements from the deployment are used as cluster analysis dimensions.

25. The meta-application system of claim 18, wherein the root cause analysis module is configured to use a pattern-recognition method comprising:

finding an upstream-most object of a dependency chain of marked objects, such that the upstream-most object does not depend on an adjacent marked object in the chain; and

identifying one of said represented hardware or software elements corresponding to the upstream-most object, and designating the identified element as a root cause candidate.

26. The meta-application system of claim 18, wherein the root cause analysis module is configured to use a pattern-recognition method comprising:

finding a candidate marked object having a directed link to a plurality of marked objects;

determining that there are no marked objects having directed links to the candidate marked object; and

identifying one of said represented hardware or software elements corresponding to the candidate marked object, and designating the identified element as a root cause candidate.

27. The meta-application system of claim 18, wherein the root cause analysis module is configured to use a pattern-recognition method comprising:

finding a candidate object having a directed link to one or more marked objects;

determining that there are no marked objects having directed links to the one or more marked objects to which the candidate object has directed links; and

identifying one of said represented hardware or software elements corresponding to the candidate object, and designating the identified element as a root cause candidate.

28. The meta-application system of claim 18, wherein the root cause analysis module is configured to use a pattern-recognition method comprising:

finding a first candidate object having a directed link to one or more marked objects;

finding a second candidate object having directed links to the one or more marked objects and directed links to one or more non-marked objects; and

identifying one of said represented hardware or software elements corresponding to the first candidate object, and designating the identified element as a root cause candidate.

29. The meta-application system of claim 18, wherein the application model further comprises a temporal history of a state of a marked object of the object graph, and wherein the root cause analysis module is configured to use a pattern-recognition method comprising using the temporal history to compare a current state of the marked object with one or more previous states of the marked object.

30. The meta-application system of claim 29, wherein the pattern-recognition method further comprises:

finding a feature of the object in the current state that is dissimilar to said previous states of the object; and

identifying the dissimilar feature as a root cause candidate.

31. The meta-application system of claim 18, wherein the root cause analysis module is configured to identify the one or more suspect objects of the application model prior to using pattern-recognition.

32. The meta-application system of claim 31, wherein the root cause analysis module determines which of the objects to mark by identifying application model objects associated with detected features that map to specific known problems with the software application.

33. The meta-application system of claim 18, wherein the root cause analysis module is configured to use a pattern-recognition method comprising:

determining a quality or value of a first aspect of a first non-marked object of the application model;

determining a quality or value of said first aspect of a second non-marked object of the application model;

determining a quality or value of said first aspect of a marked object of the application model;

determining that the qualities or values of the first aspect of the first and second non-suspect objects are substantially the same;

determining that the quality or value of the first aspect of the first non-marked object is substantially different than the quality or value of the first aspect of the marked object; and

identifying an object of the application model associated with the first aspect of the marked object, and designating the identified object as a root cause candidate.

34. The meta-application system of claim 33, wherein the pattern-recognition method further comprises, prior to said identifying:

determining a quality or value of a second aspect of the first non-marked object;

determining a quality or value of the second aspect of the second non-marked object;

determining a quality or value of the second aspect of the marked object; and

determining that the qualities or values of the second aspect of the first and second non-marked objects are substantially different.

35. The meta-application system of claim 18, wherein the one or more computer systems includes the managed computer system.