Apparatus and method for event correlation and problem reporting

• US 5,528,516 A
• Filed: 05/25/1994
• Issued: 06/18/1996
• Est. Priority Date: 05/25/1994
• Status: Expired due to Term

• Current Assignee: VMware, Inc. (Broadcom, Inc.)
• Original Assignee: System Management Arts, Inc. (Dell Technologies Inc.)
• Inventors: Kliger, Shmuel, Yemini, Yechiam, Yemini, Shaula
• Primary Examiner(s): Ramirez, Ellis B.
• Assistant Examiner(s): Wachsman, Hal D.
• Application Number:
  US08/249,282
• Time in Patent Office: 755 Days
• Field of Search: 364/550, 364/422, 364/191, 364/551.01, 364/554, 364/578-580, 364/500, 364/424.03, 364/424.04, 341/106, 341/107, 341/109, 341/78, 341/94, 370/100.1, 370/104.1, 370/13, 395/911-919, 395/922, 395/924, 395/931, 395/909, 395/905, 395/2.25-2.27, 395/2.56, 395/51, 395/52, 395/2, 395/2.1, 340/823.79, 340/0.8, 340/0.83, 340/0.16, 371/29.1, 371/29.5
• US Class Current: 702/181
• CPC Class Codes: G06F 11/2257 using expert systems
• CPC Class Codes: G06F 11/2273 Test methods
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 2201/86 Event-based monitoring

Abstract

An apparatus and method is provided for efficiently determining the source of problems in a complex system based on observable events. The problem identification process is split into two separate activities of (1) generating efficient codes for problem identification and (2) decoding the problems at runtime. Various embodiments of the invention contemplate creating a causality matrix which relates observable symptoms to likely problems in the system, reducing the causality matrix into a minimal codebook by eliminating redundant or unnecessary information, monitoring the observable symptoms, and decoding problems by comparing the observable symptoms against the minimal codebook using various best-fit approaches. The minimal codebook also identifies those observable symptoms for which the greatest benefit will be gained if they were monitored as compared to others. By defining a distance measure between symptoms and codes in the codebook, the invention can tolerate a loss of symptoms or spurious symptoms without failure. Changing the radius of the codebook allows the ambiguity of problem identification to be adjusted easily. The invention also allows probabilistic and temporal correlations to be monitored.

Claims

1. A method for detecting problems in a system which generates a plurality of symptoms, the method comprising the steps of:

(1) providing a computer-accessible codebook comprising a matrix of values each corresponding to a mapping between one of said plurality of symptoms and one of a plurality of likely problems in said system;

(2) monitoring a plurality of symptom data values representing said plurality of symptoms generated by said system over time;

(3) determining a mismatch measure between each of a plurality of groups of said values in said codebook and said plurality of symptom data values through the use of a computer, and selecting one of said plurality of likely problems corresponding to one of said plurality of groups having the smallest mismatch measure; and

(4) generating a report comprising said one selected likely problem from said codebook.

2. The method of claim 1, wherein step (3) comprises the step of determining a Hamming distance between each of said plurality of groups and said plurality of symptom data values.

3. The method of claim 1, wherein step (3) comprises the step of adding individual mismatch measures across a plurality of pairs, each pair comprising one of the plurality of symptom data values and one of the plurality of values.

4. The method of claim 3, wherein step (3) comprises the step of using a mismatch measure which gives a different weight to absence of a symptom data value than to presence of a symptom data value.

5. The method of claim 3, wherein step (3) comprises the step of outputting as likely problems all likely problems in said codebook which fall within a predetermined tolerance from a best fit match.

6. The method of claim 1, wherein step (1) comprises the step of specifying each of said values as a probability, said probability reflecting a likelihood that a corresponding symptom was caused by a corresponding problem.

7. The method of claim 6, wherein step (1) comprises the step of specifying each of said values as a pair of data, said pair comprising a first datum designating said probability and a second datum designating a temporal indicator corresponding to a time frame within which said probability holds.

8. The method of claim 6, wherein step (1) comprises the step of specifying each said probability as a discrete value.

9. The method of claim 1, wherein said system comprises a network of computer nodes, and wherein step (2) comprises the step of receiving messages from said computer nodes, said messages comprising said plurality of symptom data values.

10. The method of claim 1, wherein said system comprises a telecommunication network, and wherein step (2) comprises the step of receiving signals from equipment in said telecommunication network, said signals comprising said plurality of symptom data values.

11. The method of claim 1, wherein said system comprises a computer having peripherals, and wherein step (2) comprises the step of receiving signals from said peripherals, said signals comprising said plurality of symptom data values.

12. The method of claim 1, wherein said system comprises a plurality of satellites, and wherein step (2) comprises the step of receiving signals from said satellites, said signals comprising said plurality of symptom data values.

13. The method of claim 1, wherein said system comprises a human patient, and wherein step (2) comprises the step of receiving signals from sensors coupled to said human patient, said signals comprising said symptom data values.

14. The method of claim 1, wherein step (3) comprises the step of determining said mismatch measure by looking up a predetermined measure from a pre-computed table.

15. The method of claim 1, further comprising the step of, prior to step (1), providing a causality matrix comprising a larger set of values than said codebook, said larger set of values also corresponding to mappings between said plurality of symptom data values and said plurality of likely problems corresponding thereto;

and

wherein step (1) comprises the step of generating said codebook by reducing said larger set of values contained in said causality matrix into said codebook.

16. The method of claim 15, wherein step (1) comprises the step of eliminating redundant rows and columns from said causality matrix.

17. The method of claim 15, wherein step (1) comprises the step of reducing the number of rows in said causality matrix in accordance with a desired degree of distinction between groups of said plurality of symptoms.

18. The method of claim 1, further comprising the step of, prior to step (1), providing a causality graph comprising a plurality of nodes each corresponding to an event, a plurality of directed edges each pointing from one of the plurality of nodes to another of the plurality of nodes and corresponding to a causal relation between two or more of said events, wherein certain of said nodes are marked as problem nodes and others are marked as symptom nodes;

and wherein step (1) comprises the step of generating said codebook by traversing said directed edges leading from problem nodes to symptom nodes.

19. The method of claim 18, wherein step (1) comprises the steps of:

eliminating from said causality graph redundant symptom nodes that may be reached via directed edges from the same set of problem nodes; and

eliminating indistinguishable problem nodes that lead via directed edges to the same set of symptom nodes.

20. The method of claim 18, wherein step (1) comprises the step of eliminating from said causality graph symptom nodes in accordance with a desired degree of distinction between groups of said plurality of symptoms.

21. The method according to claim 1, wherein step (1) comprises the step of providing a computer-accessible codebook comprising a matrix of values which has been reduced from a causality matrix by eliminating redundant information from the causality matrix.

22. A method for detecting problems in a system which generates a plurality of symptoms, the method comprising the steps of:

(1) generating a causality matrix comprising a first matrix of values each corresponding to a mapping between one of said plurality of symptoms and one of a plurality of likely problems in said system;

(2) reducing said causality matrix into a codebook comprising a second matrix of values fewer in number than said first matrix of values by eliminating duplicative sets of values from said first matrix of values;

(3) monitoring a plurality of symptom data values representing said plurality of symptoms generated by said system over time;

(4) determining a mismatch measure between each of a plurality of groups of said values in said codebook and said plurality of symptom data values through the use of a computer, and selecting one of said plurality of likely problems corresponding to one of said plurality of groups having the smallest mismatch measure; and

(5) reporting said one selected likely problem from said codebook.

23. The method of claim 22, wherein step (2) comprises the step of eliminating redundant rows and columns from said first matrix.

24. The method of claim 22, further comprising the step of selecting a desired degree of distinction between each of said groups of said plurality of symptoms, each group corresponding to a different likely problem, and wherein step (2) comprises the step of deleting values from said first matrix which do not satisfy said desired level, said deletions made on the basis of comparisons between one or more of said values from said first matrix with said desired degree of distinction.

25. The method of claim 24, wherein said comparisons are made with respect to a Hamming distance determined with respect to one or more of said values from said first matrix.

26. The method of claim 24, further comprising the step of specifying each of said values in said first matrix as a probability, said probability reflecting a likelihood that a corresponding symptom was caused by a corresponding problem.

27. The method of claim 26, wherein said specifying step comprises the step of specifying each of said values as a discrete probability value.

28. The method of claim 24, further comprising the step of making said comparisons by using a mismatch measure which gives a different weight to absence of a symptom than to presence of a symptom.

29. A method of generating a codebook for use in a process of detecting problems in a system which generates a plurality of symptoms, the method comprising the steps of:

(1) preparing a causality matrix comprising a matrix of values each corresponding to a mapping between one of said plurality of symptoms and one of a plurality of likely problems in said system;

(2) making said causality matrix well-formed by deleting redundant sets of values from said matrix of values;

(3) selecting a desired degree of distinction between groups of said plurality of symptoms, each group corresponding to a different likely problem;

(4) generating, through the use of a computer, an optimal codebook from said well-formed causality matrix by selecting minimal groups of symptoms from said well-formed causality matrix such that selected groups of symptoms corresponding to any two likely problems satisfy the desired degree of distinction; and

(5) storing said optimal codebook in a computer storage device.

30. The method of claim 29, wherein step (1) comprises the step of preparing a formal specification of an event model which defines relationships between events in said system and causes thereof.

31. The method of claim 30, wherein said preparing step comprises the step of inputting to a compiler compilable statements which use probabilities to define said relationships.

32. The method of claim 31, further comprising the steps of:

compiling said compilable statements into methods and data structures; and

using said methods and data structures to generate said causality matrix by determining a causality closure of problems contained in a configuration specification.

33. The method of claim 29, wherein step (1) comprises the steps of:

(a) logging events occurring in said system over a period of time to a computer storage device;

(b) analyzing said logged events for statistical correlations;

(c) filtering said analyzed events based on a correlation threshold and producing a filtered set of data comprising symptoms and likely problems; and

(d) generating said causality matrix using said filtered set of data.

34. An optimal codebook generated in accordance with the method of claim 29.

35. A method of generating a codebook for use in a process of detecting problems in a system which generates a plurality of symptoms, the method comprising steps of:

(1) preparing a causality graph comprising a plurality of nodes each corresponding to a problem or a symptom, and a plurality of directed edges each pointing from one of the plurality of nodes to another of the plurality of nodes, and corresponding to a causal relation between two or more of said plurality of nodes;

(2) making said causality graph well-formed by deleting redundant nodes;

(3) selecting a desired degree of distinction between groups of said plurality of symptoms, each group corresponding to a different likely problem;

(4) generating, through the use of a computer, an optimal codebook from said well-formed causality graph by selecting a minimal set of symptom nodes such that the minimal set of symptom nodes caused by any two problem nodes satisfy the desired degree of distinction; and

(5) storing said optimal codebook in a computer storage device.

36. The method of claim 35, wherein step (1) comprises the step of inputting a formal specification of an event model into a computer, said formal specification defining relationships between events in said system and causes thereof.

37. The method of claim 36, wherein step (1) comprises the step of inputting compilable statements into a compiler, said compilable statements comprising probabilities to define said relationships.

38. The method of claim 36, further comprising the steps of:

compiling said formal specification into methods and data structures; and

using said methods and data structures to generate said causality graph by determining a causality closure of problems in a configuration specification.

39. An optimal codebook generated in accordance with the method of claim 35.

40. Apparatus for detecting problems in a system which generates a plurality of symptoms, the apparatus comprising:

a storage device for storing a codebook comprising a matrix of values each corresponding to a mapping between one of said plurality of symptoms and one of a plurality of likely problems in said system;

monitoring means for monitoring a plurality of symptom data values representing said plurality of symptoms generated by said system over time;

means for determining a mismatch measure between each of a plurality of groups of said values in said codebook and said plurality of symptom data values, and selecting one of said plurality of likely problems corresponding to one of said plurality of groups having the smallest mismatch measure; and

generating means for generating a report comprising said one selected likely problem.

41. The apparatus of claim 40, wherein said decoding means comprises means for determining a Hamming distance between each of said plurality of groups and said plurality of symptom data values.

42. The apparatus of claim 40, wherein said means for determining a mismatch measure adds individual mismatch measures across a plurality of pairs, each pair comprising one of the plurality of symptom data values and one of the plurality of values.

43. The apparatus of claim 42, wherein said best fit match determination uses a mismatch measure which gives a different weight to absence of a symptom data value than to presence of a symptom data value.

44. The apparatus of claim 42, wherein said decoding means outputs as likely problems all likely problems in said codebook which fall within a predetermined tolerance from a best fit match.

45. The apparatus of claim 40, wherein each of said values comprises a probability reflecting a likelihood that a corresponding symptom was caused by a corresponding problem.

46. The apparatus of claim 45, wherein each of said values comprises a pair of data, said pair comprising a first datum designating said probability and a second datum designating a temporal indicator corresponding to a time frame within which said probability holds.

47. The apparatus of claim 45, wherein each of said probability values is specified as a discrete value.

48. The apparatus of claim 40, wherein said system comprises a network of computer nodes, and wherein said monitoring means comprises means for receiving messages from said computer nodes, said messages comprising said plurality of symptom data values.

49. The apparatus of claim 40, wherein said system comprises a telecommunication network, and wherein said monitoring means comprises means for receiving signals from equipment in said network, said signals comprising said plurality of symptom data values.

50. The apparatus of claim 40, wherein said system comprises a computer having peripherals, and wherein said monitoring means comprises means for receiving signals from said peripherals, said signals comprising said plurality of symptom data values.

51. The apparatus of claim 40, wherein said system comprises a plurality of satellites, and wherein said monitoring means comprises means for receiving signals from said satellites, said signals comprising said plurality of symptom data values.

52. The apparatus of claim 40, wherein said system comprises a human patient, and wherein said monitoring means comprises means for receiving signals from sensors coupled to said human patient, said signals comprising said plurality of symptom data values.

53. The apparatus of claim 40, wherein said mismatch measure is determined by looking up a predetermined measure from a pre-computed table.

54. The apparatus of claim 40, further comprising:

means for storing a causality matrix comprising a larger set of values than said codebook, said values also corresponding to mappings between said symptom data values and said plurality of likely problems corresponding thereto; and

means for generating said codebook by reducing said larger set of values contained in said causality matrix into values for said codebook.

55. The apparatus of claim 54, wherein said codebook is generated by eliminating redundant rows and columns from said causality matrix.

56. The apparatus of claim 54, wherein said codebook is generated by reducing the number of rows in said causality matrix in accordance with a desired degree of distinction between groups of said plurality of symptoms.

57. The apparatus of claim 40, further comprising means for storing a causality graph comprising a plurality of nodes each corresponding to an event, and a plurality of directed edges each pointing from one of the plurality of nodes to another of the plurality of nodes and corresponding to a causal relation between two of said events, wherein certain nodes are marked as problems and certain nodes are marked as symptoms;

and

wherein said codebook is generated by traversing said directed edges in said causality graph leading from nodes marked as problems to nodes marked as symptoms.

58. The apparatus of claim 57, wherein said codebook is generated by eliminating from said causality graph redundant symptom nodes that may be reached via directed edges from the same set of problem nodes, and by eliminating indistinguishable problem nodes that lead via directed edges to the same set of symptom nodes.

59. The apparatus of claim 57, wherein said codebook is generated by eliminating from said causality graph symptom nodes in accordance with a desired degree of distinction between groups of said plurality of symptoms.

60. The apparatus according to claim 40, wherein the values in the codebook have been reduced from a causality matrix by eliminating redundant information from the causality matrix.

61. Apparatus for detecting problems in a system which generates a plurality of symptoms, the apparatus comprising:

generating means for generating a causality matrix comprising a first matrix of values each corresponding to a mapping between one of said plurality of symptoms and one of a plurality of likely problems in said system;

reducing means for reducing said causality matrix into a computer-accessible codebook comprising a second matrix of values fewer in number than said first matrix of values by eliminating duplicative sets of values from said first matrix;

monitoring means for monitoring, through the use of a computer, a plurality of symptom data values representing said plurality of symptoms generated by said system over time;

means for determining a mismatch measure between each of a plurality of groups of said values in said codebook and said plurality of symptom data values, and selecting one of said plurality of likely problems corresponding to one of said plurality of groups having the smallest mismatch measure; and

a report generator for reporting said one selected likely problem.

62. The apparatus of claim 61, wherein said reducing means eliminates redundant rows and columns from said first matrix.

63. The apparatus of claim 61, wherein said reducing means comprises means for inputting a desired degree of distinction between each of said groups of said plurality of symptoms, each group corresponding to a different likely problem, and wherein said reducing means deletes values from said first matrix which do not satisfy said desired degree of distinction, said deletions made on the basis of comparisons between one or more of said values from said first matrix with said desired degree of distinction.

64. The apparatus of claim 63, wherein said comparisons are made with respect to a Hamming distance determined with respect to one or more of said values from said first matrix.

65. The apparatus of claim 63, wherein each of said values in said first matrix comprises a probability reflecting a likelihood that a corresponding symptom was caused by a corresponding problem.

66. The apparatus of claim 65, wherein said probabilities comprise a discrete value.

67. The apparatus of claim 63, wherein said mismatch measure gives a different weight to absence of a symptom than to presence of a symptom.

68. Apparatus for generating a codebook for use in detecting problems in a system which generates a plurality of symptoms, the apparatus comprising:

preparing means for preparing a causality matrix comprising a matrix of values each corresponding to a mapping between one of said plurality of symptoms and one of a plurality of likely problems in said system;

means for making said causality matrix well-formed by deleting redundant sets of values from said matrix of values;

inputting means for inputting a desired degree of distinction between groups of said plurality of symptoms, each group corresponding to a different likely problem;

generating means for generating a computer-accessible optimal codebook from said well-formed causality matrix by selecting minimal groups of symptoms from said well-formed causality matrix such that selected groups of symptoms corresponding to any two likely problems satisfy the desired degree of distinction; and

a storage device for storing said computer-accessible optimal codebook.

69. The apparatus of claim 68, wherein said preparing means comprises:

means for inputting a specification of an event model defining relationships between events in said system and causes thereof; and

a compiler for compiling said specification into data structures.

70. The apparatus of claim 69, wherein said specification comprises statements which define said relationships using probability values.

71. The apparatus of claim 69, further comprising a matrix generator for transforming said data structures into said causality matrix by determining a causality closure of problems contained in a configuration specification.

72. The apparatus of claim 68, wherein said preparing means comprises:

means for logging events occurring in said system over a period of time to a computer storage device;

means for analyzing said logged events for statistical correlations;

means for filtering said analyzed events based on a correlation threshold and producing a filtered set of data comprising symptoms and likely problems; and

means for generating said causality matrix using said filtered set of data.

73. An optimal codebook generated using the apparatus of claim 68.

74. Apparatus for generating a codebook for use in detecting problems in a system which generates a plurality of symptoms, the apparatus comprising:

preparing means for preparing a causality graph comprising a plurality of nodes each corresponding to a problem or a symptom, and a plurality of directed edges each pointing from one of the plurality of nodes to another of the plurality of nodes and corresponding to a causal relation between two or more of said plurality of nodes;

means for making said causality graph well-formed by deleting redundant nodes;

specifying means for specifying a desired degree of distinction between groups of said plurality of symptoms, each group corresponding to a different likely problem;

generating means for generating, through the use of a computer, an optimal codebook from said well-formed causality graph by selecting a minimal group of symptom nodes such that selected groups satisfy the degree of distinction; and

a computer storage device for storing said optimal codebook.

75. The apparatus of claim 74, wherein said preparing means comprises means for inputting a specification of an event model which defines relationships between events in said system and causes thereof.

76. The apparatus of claim 75, wherein said specification comprises compilable statements which use probabilities to define said relationships.

77. The apparatus of claim 75, wherein said preparing means comprises means for compiling said specification into methods and data structures, and wherein said methods and data structures are used to generate said causality graph by determining a causality closure of problems contained in a configuration specification.

78. The apparatus of claim 74, wherein said preparing means comprises:

means for logging events occurring in said system over a period of time to a computer storage device;

means for analyzing said logged events for statistical correlations;

means for filtering said analyzed events based on a correlation threshold and producing a filtered set of data comprising symptoms and likely problems; and

means for generating said causality graph using said filtered set of data.

79. An optimal codebook generated using the apparatus of claim 74.

80. A method of preparing a data structure for use in detecting or isolating problems in a system having a plurality of components of various classes, said system generating a plurality of observable events, the method comprising the steps of:

(1) preparing compilable statements which define, for each class of component in said system,

(a) the relationships the component can participate in with respect to other classes of components;

(b) problems associated with the component and events caused therefrom;

(c) a causal propagation of the problems and the events to other components along the relationships;

(2) preparing a configuration specification which defines component instances in the system, their classes and their relationships to other components;

(3) translating, through the use of a computer, the compilable statements and the configuration specification into the data structure by determining a causality closure of the events in the system; and

(4) storing the data structure in a computer storage device.