Automated detection of problems in software application deployments

• US 7,788,536 B1
• Filed: 12/21/2005
• Issued: 08/31/2010
• Est. Priority Date: 12/21/2004
• Status: Active Grant

• Current Assignee: Citrix Systems, Inc. (Cloud Software Group)
• Original Assignee: Zenprise, Inc. (Cloud Software Group)
• Inventors: Hassan, Tanvir, Roach, Kelly Brian, Qureshi, Waheed, McGinty, John M.
• Primary Examiner(s): Beausoliel; Robert
• Assistant Examiner(s): Ehne; Charles
• Application Number:
  US11/316,395
• Time in Patent Office: 1,714 Days
• Field of Search: 714/26, 714/38, 714/48
• US Class Current: 714/38.14
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06N 5/048 Fuzzy inferencing

Abstract

A system for monitoring and analyzing the operation of a deployment of a software application is disclosed. The system includes a monitoring component, a repository that stores a plurality of rules, and an analysis engine. The monitoring component collects data values of each of a plurality of state metrics associated with the deployment of the software application. At least some of the rules in the repository map respective combinations of states associated with the state metrics to corresponding problems. The analysis engine programmatically detects instances of said problems, at least in-part, by using the rules to analyze the state metric data values collected by the monitoring component. The rules can also include remedies associated with the detected problems, and the system may include remedy selection and execution modules.

Claims

1. A computer-implemented meta-application system for monitoring and analyzing the operation of a deployment of a software application, comprising:

a computer system comprising one or more computers, said computer system programmed to implement at least;

a monitoring component that collects data values of each of a plurality of state metrics associated with the deployment of the software application;

a repository that stores a plurality of rules, at least some of which map respective combinations of states associated with the state metrics to corresponding problems; and

an analysis engine that programmatically detects instances of said problems, at least in-part, by using the rules to analyze the state metric data values collected by the monitoring component;

wherein at least one of the rules is configured to be used by the analysis engine to generate a virtual circuit comprising;

a plurality of atomic gates each configured to be used by the analysis engine to test for whether collected data values for one or more of the state metrics indicate the existence of one of a plurality of possible conditions of the deployment of the software application, and to produce an output bead if the condition exists within the deployment; and

a plurality of operator gates each having one or more inputs, each of the inputs configured to receive output beads from one of the atomic gates or from another one of the operator gates, each operator gate configured to be used by the analysis engine to test for a condition associated with received beads and to produce an output bead if the condition is met;

wherein the rule is configured to be applied by the analysis engine to automatically detect the existence of a problem within the deployment based on output beads generated from use of the operator gates.

2. The meta-application system of claim 1, wherein each of the atomic gates comprises an open atomic formula comprising a predicate signature and one or more arguments, each argument comprising one of an unbound variable and a constant term, each predicate signature and atomic formula corresponding to one of the possible conditions of the deployment of the software application.

3. The meta-application system of claim 2,

wherein the analysis engine is adapted to compare the open atomic formula of each atomic gate to a closed atomic formula comprising a predicate signature and one or more arguments, each argument of the closed atomic formula being a constant term, each closed atomic formula indicative of a current condition of the deployment;

wherein a match between an open atomic formula of an atomic gate and a closed atomic formula that is compared to the open atomic formula implies;

that a predicate signature of the closed atomic formula matches a predicate signature of the open atomic formula;

that a total number of arguments of the closed atomic formula matches a total number of arguments of the open atomic formula;

that all of the constant term arguments of the closed atomic formula match corresponding constant term arguments of the open atomic formula; and

that a deployment component associated with at least one argument of the closed atomic formula meets the possible condition associated with the atomic gate;

wherein the analysis engine is configured to respond to a match between the open atomic formula of the atomic gate and a received closed atomic formula by producing a bead, the bead including variable bindings comprising assignments of constant term arguments of the closed atomic formula to corresponding unbound variables of the open atomic formula, the analysis engine configured to send the bead to an input of one of the operator gates.

4. The meta-application system of claim 3, wherein each constant term of the closed atomic formula comprises one of a numerical value and a string.

5. The meta-application system of claim 3, wherein one of the operator gates has a first input that receives output beads from a first atomic gate, and a second input that receives output beads from a second atomic gate, the first atomic gate having a first open atomic formula with a particular unbound variable argument, the second atomic gate having a second open atomic formula also with said particular unbound variable argument, said analysis engine configured to use said one operator gate to generate an output bead only if a constant term argument of a first closed atomic formula received by the first atomic gate matches a constant term argument of a second closed atomic formula received by the second atomic gate, wherein said constant term argument of the first closed atomic formula corresponds to said particular unbound variable argument in the first open atomic formula, and said constant term argument of the second closed atomic formula corresponds to said particular unbound variable argument in the second open atomic formula.

6. The meta-application system of claim 1, wherein at least one of the operator gates is adapted to be used by the analysis engine to generate an output bead only if a temporal measure associated with a first bead received by the operator gate precedes a temporal measure associated with a second bead received by the operator gate.

7. The meta-application system of claim 1, wherein at least one of the operator gates is adapted to be used by the analysis engine to generate an output bead only if an input bead is received on each of the operator gate'"'"'s inputs.

8. The meta-application system of claim 1, wherein at least one of the operator gates is adapted to be used by the analysis engine to generate an output bead if an input bead is received on any of the operator gate'"'"'s inputs.

9. The meta-application system of claim 1, wherein said at least one of the rules comprises an abstract remedy for treating the problem described by the rule, the remedy comprising a set of one or more plans, each plan comprising steps and decision factors for correcting the problem, each plan comprising parameters, the system further comprising an automation subsystem configured to determine bindings for the parameters based on satisfaction of the conditions of the rule.

10. The meta-application system of claim 9, wherein the automation subsystem includes a planning module configured to convert each plan into a tree of nodes comprising said steps and decision factors for correcting the problem.

11. The meta-application system of claim 10, wherein at least one of the nodes comprises a scope node containing one or more of (1) said bindings for the parameters of the plan, (2) metadata about the problem that the plan addresses, and (3) a logical formula of conditions corresponding to the problem.

12. The meta-application system of claim 10, wherein at least one of the nodes comprises an interface node configured to query for information needed to customize the plan to the deployment.

13. The meta-application system of claim 12, wherein the interface node is configured to solicit input from a human user of the system.

14. The meta-application system of claim 12, wherein the interface node is configured to query for information from a computer-implemented application model of the deployment, the application model comprising:

a representation of a plurality of hardware and software entities of the deployment; and

a representation of dependencies between the entities.

15. The meta-application system of claim 10, wherein at least one of the nodes comprises an interface node configured to execute an action against the deployment.

16. The meta-application system of claim 9, wherein the automation subsystem includes an execution engine configured to at least partially automate the execution of the plan against the deployment, based on the bindings for the parameters.

17. The meta-application system of claim 16, wherein the automation subsystem detects and logs outcomes of plans that are executed at least partially by the execution engine.

18. The meta-application system of claim 9, wherein each of the rules includes a numerical severity representing a degree to which the problem associated with the rule adversely impacts the deployment, the automation subsystem being configured to optimize an order of execution of a set of remedies associated with problems detected by the analysis engine, based on the severities of the rules associated with the detected problems.

19. The meta-application system of claim 1, wherein at least one of the operator gates specifies contribution weights for its inputs and has a first input with a first contribution weight and a second input with a second contribution weight that is less than the first contribution weight, the analysis engine being adapted to give more weight to the first input than to the second input for purposes of producing an output bead from the operator gate.

20. The meta-application system of claim 1, wherein at least one of the operator gates specifies contribution weights for its inputs and has a first input with a first contribution weight and a second input with a second contribution weight that is less than the first contribution weight, the analysis engine being adapted to give more weight to the first input than to the second input for purposes of producing a confidence value representing a degree of confidence that the deployment meets all of the conditions for which the operator gate tests.

21. The meta-application system of claim 20, wherein the analysis engine is adapted to give more weight to the first input than to the second input for purposes of producing a confidence value representing a degree of confidence that the problem associated with said at least one of the rules exists within the deployment.

22. The meta-application system of claim 1, wherein the analysis engine, in applying said at least one of the rules to the deployment, is configured to receive a condition confidence value representing a confidence that a particular condition exists, and to use said condition confidence value to evaluate a confidence that the particular problem described by the rule exists, wherein the tested-for conditions of the gates of the rule include static significance values indicative of the importance of the different conditions, the analysis engine configured to use the significance values to decline to report the problem when (1) one or more of the tested-for conditions are found to be true, (2) the true conditions indicate that the problem is currently present in the deployment, and (3) a condition confidence value of at least one of the true conditions does not exceed the significance value of that particular condition.

23. The meta-application system of claim 1, wherein at least one of the operator gates designates at least one of its inputs as a required input, and designates at least another one of its inputs as an optional input, such that the analysis engine, in using the operator gate, requires that the required input receive a bead for the operator gate to be used to produce an output bead, and such that the analysis engine does not require that the optional input receive a bead for the operator gate to be used to produce an output bead.

24. The meta-application system of claim 23, wherein said at least one of the operator gates is adapted to cause the system to query the deployment for data corresponding to the required input before querying the deployment for data corresponding to the optional input.

25. The meta-application system of claim 23, wherein said at least one of the operator gates includes a confidence calculation procedure for calculating a confidence value representing a degree of confidence that the deployment meets all of the conditions for which the operator gate tests, the confidence calculation procedure configured to produce a greater confidence value if the optional input receives a bead than if the optional input does not receive a bead.

26. The meta-application system of claim 25, wherein the analysis engine is configured to use the calculated confidence value in calculating a confidence value in the existence of the problem associated with the rule containing the operator gate.

27. The meta-application system of claim 23, wherein said at least one of the operator gates includes a confidence calculation procedure for calculating a confidence value representing a degree of confidence that the deployment meets all of the conditions for which the operator gate tests, the confidence calculation procedure configured to compute a non-zero confidence value only if the required input receives a bead.

28. The meta-application system of claim 27, wherein the analysis engine is configured to use the calculated confidence value in calculating a confidence value in the existence of the problem associated with the rule containing the operator gate.

29. The meta-application system of claim 1, further comprising a computer-implemented application model of the deployment, the application model comprising:

a representation of a plurality of hardware and software entities of the deployment; and

a representation of dependencies between the entities.

30. The meta-application system of claim 29, wherein the application model is stored in an SQL database that supports SQL queries.

31. The meta-application system of claim 29, wherein at least one of the gates is configured to cause the analysis engine to query the application model for specific deployment state data to be used by the analysis engine in determining whether to produce an output bead from the operator gate.

32. The meta-application system of claim 29, wherein the analysis engine is configured to use said at least one of the rules to produce identifications of entities of the application model that cause or are affected by the problem described by the rule.

33. The meta-application system of claim 29, further comprising a root cause analysis module configured to use pattern-matching to correlate a detected problem with one of the entities of the application model.

34. The meta-application system of claim 29, wherein at least one of the rules comprises an encoded diagnostic troubleshooting method associated with entities of the application model.

35. The meta-application system of claim 29,

wherein said at least one of the rules comprises an abstract remedy for treating the problem described by the rule, the remedy comprising a set of one or more plans, each plan comprising steps and decision factors for correcting the problem, each plan comprising one or more parameters, the system further comprising an automation subsystem configured to determine bindings for the one or more parameters based on satisfaction of the conditions of the rule;

wherein the automation subsystem is configured to analyze the application model in order to evaluate whether a plan is suitable for executing on the deployment.

36. The meta-application system of claim 29,

wherein said at least one of the rules comprises an abstract remedy for treating the problem described by the rule, the remedy comprising a set of one or more plans, each plan comprising steps and decision factors for correcting the problem, each plan comprising one or more parameters, the system further comprising an automation subsystem configured to determine bindings for the one or more parameters based on satisfaction of the conditions of the rule;

wherein the automation subsystem is configured to customize a plan based on information obtained from the application model.

37. The meta-application system of claim 1, wherein a first of the operator gates comprises a confidence calculation formula that differs from a confidence calculation formula of a second operator gate having the same number of inputs, the analysis engine configured to apply the confidence calculation formula of the first operator gate to at least one of the first operator gate'"'"'s inputs for purposes of producing a confidence value associated with an output bead of the first operator gate.

38. The meta-application system of claim 37, wherein the confidence calculation formula of the first operator gate comprises a sigmoidal function.

39. The meta-application system of claim 38, wherein the sigmoidal function is multivariate and substantially continuous, smooth, and monotonic.

40. The meta-application system of claim 37, wherein the confidence calculation formula of the first operator gate includes a variable for a confidence value associated with a bead received by one of the inputs of the first operator gate.

41. The meta-application system of claim 1, further comprising a user interface that displays descriptions of problems detected by the analysis engine, as well as descriptions of remedies associated with the problems.

42. The meta-application system of claim 1, wherein at least one of the operator gates comprises cost estimates for acquiring missing inputs, the cost estimate of acquiring one input being distinct from the cost of acquiring another input, wherein the analysis engine is configured to acquire said one input before acquiring said another input.

43. The meta-application system of claim 1, wherein said at least one of the rules is an encoded representation of a human-readable text document associated with the software application.

44. The meta-application system of claim 1, wherein said at least one of the rules further comprises one or more wires, each wire adapted to be used by the analysis engine to receive a bead from one of the gates, each wire also adapted to be used by the analysis engine to deliver a received bead to another one of the gates.

45. A computer-implemented meta-application system for monitoring and analyzing the operation of a deployment of a software application, comprising:

a computer system comprising one or more computers, said computer system programmed to implement at least;

a monitoring component that collects data values of each of a plurality of state metrics associated with the deployment of the software application;

a repository that stores a plurality of rules, at least some of which map respective logical combinations of states associated with the state metrics to corresponding problems; and

an analysis engine that programmatically detects instances of said problems, at least in-part, by using the rules to analyze the state metric data values collected by the monitoring component;

wherein at least one rule stored in the repository includes at least one query for a new data value for at least one of the state metrics, and wherein the analysis engine is configured to initiate said query and use the data value obtained via said query in cooperation with the rule to evaluate whether the problem associated with said at least one rule exists within the deployment, the rules and the query being represented substantially in a logic programming language;

wherein the rules collectively comprise logical formulae including predicates, logical operators, mathematical operators, free variables, and constant terms, the logical operators including AND and OR operators.

46. The meta-application system of claim 45, wherein the at least one rule comprises an abstract remedy for treating the problem described by the rule, the remedy comprising a set of one or more plans, each plan comprising steps and decision factors for correcting the problem, each plan comprising at least one parameter, the system configured to determine a binding for the parameter based on the data value obtained via said query.

47. The meta-application system of claim 45, wherein the at least one rule comprises an abstract remedy for treating the problem described by the rule, the remedy comprising a set of one or more plans, each plan comprising steps and decision factors for correcting the problem, the system configured to customize a plan based on the data obtained via said query.

48. The meta-application system of claim 45, wherein each predicate represents a type of deployment state and has a value of TRUE or FALSE, at least one of the rules comprising a logical combination of predicates joined by one or more of the logical operators.

49. The meta-application system of claim 48, wherein each predicate comprises an open atomic formula representing a deployment condition, the open atomic formula comprising a predicate signature and one or more arguments, each argument of the open atomic formula comprising a free variable or a constant term, the analysis engine configured to determine that predicates are TRUE by detecting pattern-matches between the open atomic formulae of the predicates and closed atomic formulae indicative of specific deployment states, each closed atomic formula comprising a predicate signature and one or more constant term arguments.

50. The meta-application system of claim 49, wherein each pattern-match between an open atomic formula and a closed atomic formula indicates that:

a predicate signature of the closed atomic formula matches a predicate signature of the open atomic formula;

a total number of arguments of the closed atomic formula matches a total number of arguments of the open atomic formula;

all of the constant term arguments of the closed atomic formula match corresponding constant term arguments of the open atomic formula; and

a deployment component associated with at least one argument of the closed atomic formula meets the deployment condition represented by the open atomic formula.

51. The meta-application system of claim 50, wherein the analysis engine is configured to respond to a pattern-match between an open atomic formula and a closed atomic formula by:

binding at least one free variable of the open atomic formula with a corresponding constant term argument of the closed atomic formula; and

evaluating whether a logical combination of predicates of the rule containing the pattern-matched open atomic formula is satisfied when sufficient instances of the free variable within the predicates is bound to the constant term argument of the closed atomic formula;

wherein the meta-application is configured to respond to the analysis engine'"'"'s evaluation that the logical combination of predicates of the rule containing the pattern-matched open atomic formula is satisfied, by using the bound free variable to customize a remedy for solving a problem associated with the rule.

52. The meta-application system of claim 45, wherein the logical operators include at least one THEN operator that detects whether a time associated with a first condition is earlier than a time associated with a second condition.

53. A computer-implemented meta-application system for monitoring and analyzing the operation of a deployment of a software application, comprising:

a computer system comprising one or more computers, said computer system programmed to implement at least;

a monitoring component that collects data values of each of a plurality of state metrics associated with the deployment of the software application;

a repository that stores a plurality of rules, at least some of which map respective combinations of states associated with the state metrics to corresponding problems; and

an analysis engine that programmatically detects instances of said problems, at least in-part, by using the rules to analyze the state metric data values collected by the monitoring component;

wherein at least one rule stored in the repository includes a plurality of gates each configured to be used by the analysis engine to test for whether collected data values for one or more of the state metrics indicate the existence of one of a plurality of possible conditions of the deployment of the software application, at least some of the gates configured to propagate temporal information reflective of times that particular conditions were detected, the analysis engine being configured to use the temporal information to detect a problem with the deployment.

54. A computer-implemented meta-application system for monitoring and analyzing the operation of a deployment of a software application, comprising:

a computer system comprising one or more computers, said computer system programmed to implement at least;

a monitoring component that collects data values of each of a plurality of state metrics associated with the deployment of the software application;

a repository that stores a plurality of rules, at least some of which map respective combinations of states associated with the state metrics to corresponding problems, the rules written in a logical programming language and collectively comprising logical formulae including predicates, logical operators, mathematical operators, free variables, and constant terms, the logical operators including AND and OR operators; and

an analysis engine that programmatically detects instances of said problems, at least in-part, by using the rules to analyze the state metric data values collected by the monitoring component, the analysis engine including a feature detection component that analyzes a stream of data values of a time-varying state metric to detect a predefined type of feature of said stream, at least some of the rules taking the feature into consideration in evaluating whether a corresponding problem exists;

wherein the feature detection component analyzes the stream, at least in part, by using a mathematical model associated with the time-varying state metric to evaluate whether a behavior of the state metric includes the feature.

55. The meta-application system of claim 54, wherein one of the rules comprises an abstract remedy for treating the problem described by the rule, the remedy comprising a set of one or more plans, each plan comprising steps and decision factors for correcting the problem, each plan comprising at least one parameter, the system being configured to determine a binding for the parameter based on a feature detected by the feature detection component.

56. The meta-application system of claim 54, wherein one of the rules comprises an abstract remedy for treating the problem described by the rule, the remedy comprising a set of one or more plans, each plan comprising steps and decision factors for correcting the problem, the system being configured to customize a plan based on a feature detected by the feature detection component.

57. The meta-application system of claim 54, wherein the mathematical model detects a feature comprising a rising or descending trend in the behavior of the state metric.

58. The meta-application system of claim 54, wherein the feature detection component implements a seasonality algorithm to account for seasonal variations in the time-varying state metric.

59. The meta-application system of claim 54, wherein the feature detection component applies a control chart algorithm to the stream to detect a deviation from mathematically modeled behavior of the state metric.

60. A computer-implemented meta-application system for monitoring and analyzing the operation of a deployment of a software application, comprising:

a computer system comprising one or more computers, said computer system programmed to implement at least;

a monitoring component that collects data values of each of a plurality of state metrics associated with the deployment of the software application;

a repository that stores a plurality of rules, at least some of which map respective combinations of states associated with the state metrics to corresponding problems, each rule expressed in a logic programming language; and

an analysis engine that programmatically detects instances of said problems, at least in-part, by interpreting the rules to analyze the state metric data values collected by the monitoring component;

wherein the rules collectively comprise logical formulae including predicates, logical operators, mathematical operators, free variables, and constant terms, the logical operators including AND and OR operators.

61. The meta-application system of claim 60, wherein the logic programming language is a substantially first-order logic programming language.

62. The meta-application system of claim 60, wherein at least one rule stored in the repository includes at least one query for a new data value for at least one of the state metrics, and wherein the analysis engine is configured to initiate said query and use the data value obtained via said query in cooperation with the rule to evaluate whether the problem associated with said at least one rule exists within the deployment.

63. A computer-implemented system for monitoring and analyzing the operation of a deployment of a software application, comprising:

a monitoring component that collects data values of each of a plurality of state metrics associated with the deployment of the software application;

a repository that stores a plurality of rules, at least some of which map respective combinations of states associated with the state metrics to corresponding problems; and

an analysis engine that programmatically detects instances of said problems, at least in-part, by using the rules to analyze the state metric data values collected by the monitoring component to detect features of the deployment and convert the features into closed atomic formulas; and

one or more computers configured to implement the monitoring component and analysis engine;

wherein each rule corresponds to a potential problem with the deployment and comprises a logical combination of open atomic formulas containing free variables, the analysis engine configured to match closed atomic formulas of detected features against open atomic formulas of the rule to produce output value binding substitutions which logically combine to supply information to a remedy for the problem, each substitution comprising an assignment of a value to a variable, the substitutions including one or more variable assignments derived from closed atomic formulas of detected features.

64. The system of claim 63, wherein the variables and the features of the deployment are subject to unification to produce the output value binding substitutions.

65. The system of claim 63, wherein the analysis engine, in matching the closed formula with the open formula of free logical variables, is configured to produce a resultant formula representing at least a partial satisfaction of the open formula.