Methods and systems for performing root cause analysis

• US 20090313198A1
• Filed: 06/17/2008
• Published: 12/17/2009
• Est. Priority Date: 06/17/2008
• Status: Active Grant

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Morimura, Tomohiro, Kudo, Yutaka, Fujita, Takahiro, Masuishi, Tetsuya
• Granted Patent: US 8,112,378 B2
• Time in Patent Office: Days
• US Class Current: 706/47
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: H04L 41/046 comprising network management agents or mobile agents therefor
• CPC Class Codes: H04L 41/0631 using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
• CPC Class Codes: H04L 41/0681 Configuration of triggering conditions
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 41/12 Discovery or management of network topologies
• CPC Class Codes: H04L 41/16 using machine learning or artificial intelligence
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation

Abstract

A root cause analysis engine uses event durations and gradual deletion of events to improve analysis accuracy and reduce the number of required calculations. Matching ratios of relevant rules are recalculated every time notification of an event is received. The calculation results are held in a rule memory in the analysis engine. Each event has a valid duration, and when the duration has expired, that event is deleted from the rule memory. Events held in the rule memory can be deleted without affecting other events held in the rule memory. The analysis engine can then re-calculate the matching ratio of each rule by only performing the re-calculation with respect to affected rules related to the deleted event. The calculation cost can be reduced because analysis engine processes events incrementally or decrementally. Analysis engine can determine the most possible conclusion even if one or more condition elements were not true.

Claims

1. A method of determining a cause of an occurrence in an information system made up of a plurality of monitored nodes, the method comprising:

storing a plurality of rules, each said rule setting forth one or more conditions to be satisfied for indicating a particular cause of a particular occurrence;

receiving a first event message regarding a first event related to one of said nodes;

assigning a valid duration to the first event;

determining which of said rules have conditions corresponding to the first event;

calculating a matching ratio for any rules having conditions corresponding to the first event;

storing a matching state of conditions; and

specifying the cause of the occurrence based upon calculated matching ratios of said rules.

2. The method according to claim 1, further including steps of

invalidating the first event when the valid duration has reached a final expiration point.

3. The method according to claim 2, further including steps of

receiving a plurality of earlier event messages regarding a plurality of earlier events prior to receiving the first event message, each of said earlier events having an earlier valid duration assigned thereto;

and

wherein the step of calculating the matching ratio is performed by including the earlier events in the matching ratio calculation when the earlier events correspond to conditions in the same rules as the first event.

4. The method according to claim 3,

wherein, when a particular earlier valid duration of a particular one of said earlier events reaches a first expiration point, an attrition value is assigned to a weighting value used in the matching ratio calculation, so that importance of the particular earlier event is gradually diminished in the matching ratio calculations involving the particular event until the final expiration point is reached, whereby the event is no longer included in the matching ratio calculation.

5. The method according to claim 4, further including a step of

setting the attrition value based upon an event type, such that different event types have predetermined different attrition values.

6. The method according to claim 4, further including a step of

setting the attrition value based upon a type of node that is associated with the event, such that different node types have predetermined different attrition values.

7. The method according to claim 1,

following the step of calculating the matching ratio and when matching ratios of two or more rules are calculated in relation to the first event, further including a step of

identifying the cause of the occurrence based upon a conclusion provided by a rule having the highest calculated matching ratio from among calculated matching ratios of said two or more rules.

8. The method according to claim 1, further including a step of

setting a length of the valid duration based upon an event type, such that different event types have predetermined different valid duration lengths.

9. The method according to claim 1,

setting a length of the valid duration based upon a type of node that is associated with an event, such that different node types have predetermined different valid duration lengths.

10. The method according to claim 1,

wherein the matching ratio is calculated as a function of

a number of events corresponding to conditions of a particular rule and whose valid duration has not yet finally expired, and

a total number of conditions required for satisfying the particular rule,

wherein the calculated matching ratio indicates a probability that a conclusion of the particular rule identifies the cause of the occurrence.

11. The method according to claim 1, further comprising:

displaying the cause of the occurrence on a display for viewing by an administrator.

12. The method according to claim 1, further comprising:

storing the cause of the occurrence and the calculated matching ratio in a database for later analysis.

13. The method according to claim 1, further comprising:

determining which conditions have not been satisfied among conditions required for satisfying a particular rule.

14. The method according to claim 13, further comprising:

displaying the cause of the occurrence and any conditions that have not been satisfied among conditions required for satisfying a particular rule on a display for viewing by an administrator.

15. The method according to claim 13, further comprising:

storing the cause of the occurrence, the calculated matching ratio and any conditions that have not been satisfied among conditions required for satisfying a particular rule in a database for later analysis.

16. The method according to claim 1, further comprising:

invalidating one or more events by a timer or a manual operation of an administrator.

17. An information system comprising:

a first computer having a first display, said first computer being in communication via a network with a plurality of monitored nodes in the information system;

a plurality of rules accessible by said first computer, each said rule setting forth one or more conditions to be satisfied for indicating a cause when an occurrence takes place in one or more of said monitored nodes,

wherein said first computer is configured to receive event messages regarding events related to occurrences at one or more of said monitored nodes and assign a valid duration to each event,

wherein said first computer is configured to determine which of said rules have conditions corresponding to the events, calculate a matching ratio for any rules having conditions corresponding to the received events, and store a matching state of conditions, and

wherein said first computer is configured to specify the cause of the occurrence based upon calculated matching ratios of said rules.

18. The system according to claim 17,

wherein the first computer is configured to invalidate the first event when the valid duration has reached a final expiration point.

19. The system according to claim 18,

wherein, when the first computer receives a plurality of earlier event messages regarding a plurality of earlier events prior to receiving the first event message, each of said earlier events having an earlier valid duration assigned thereto, the first computer is configured to calculate the matching ratio by including the earlier events in the matching ratio calculation when the earlier events correspond to conditions in the same rules as the first event.

20. The system according to claim 19,

wherein, when a particular earlier valid duration of a particular one of said earlier events reaches a first expiration point, the first computer is configured to assign an attrition value to a weighting value used in the matching ratio calculation, so that importance of the particular earlier event is gradually diminished in the matching ratio calculations involving the particular event until the final expiration point is reached, whereby the event is no longer included in the matching ratio calculation.

21. The system according to claim 20, further comprising:

stored attrition setting information accessible by the first computer, wherein the first computer is configured to set the attrition value based upon an event type, such that different event types have predetermined different attrition values.

22. The system according to claim 20, further comprising:

stored attrition setting information accessible by the first computer, wherein the first computer is configured to set the attrition value based upon a type of node that is associated with the event, such that different node types have predetermined different attrition values.

23. The system according to claim 17,

wherein, when matching ratios of two or more rules are calculated in relation to the first event, the first computer is configured to identify the cause of the occurrence based upon a conclusion provided by a rule having the highest calculated matching ratio from among calculated matching ratios of said two or more rules.

24. The system according to claim 17, further comprising:

stored valid duration setting information accessible by the first computer, wherein the first computer is configured to set a length of the valid duration based upon an event type, such that different event types have predetermined different valid duration lengths.

25. The system according to claim 17, further comprising:

stored valid duration setting information accessible by the first computer, wherein the first computer is configured to set a length of the valid duration based upon a type of node that is associated with an event, such that different node types have predetermined different valid duration lengths.

26. The system according to claim 17, further comprising:

wherein the first computer is configured to calculate the matching ratio as a function of

a number of events corresponding to conditions of a particular rule and whose valid duration has not yet finally expired, and

a total number of conditions required for satisfying the particular rule,

wherein the calculated matching ratio indicates a probability that a conclusion of the particular rule identifies the cause of the occurrence.

27. The system according to claim 17, further comprising:

a display in communication with said first computer for displaying the cause of the occurrence for viewing by an administrator.

28. The system according to claim 17, further comprising:

a storage in communication with said first computer for storing the cause and the calculated matching ratio in a database for later analysis.

29. The system according to claim 17,

wherein the first computer is configured to determine which conditions have not been satisfied among conditions required for satisfying a particular rule.

30. The system according to claim 29, further comprising:

a display in communication with said first computer for displaying the cause of the occurrence and any conditions that have not been satisfied among conditions required for satisfying a particular rule for viewing by an administrator.

31. The system according to claim 29, further comprising:

a storage in communication with the first computer for storing the cause, the calculated matching ratio and any conditions that have not been satisfied among conditions required for satisfying a particular rule in a database for later analysis.

32. The system according to claim 17, further comprising:

wherein the first computer is configured to invalidate one or more events by a timer or a manual operation of an administrator.

33. The system according to claim 17, further comprising:

one or more monitoring agents running on one or more of said monitored nodes for monitoring conditions of the monitored nodes, and for reporting the events of any occurrences at said monitored nodes to said first computer.

34. The system according to claim 17, further comprising:

wherein said monitored nodes comprise one or more server computers in operative communication with one or more storage systems.

35. A method of determining a cause of an occurrence in an information system made up of a plurality of monitored nodes, the method comprising:

storing a plurality of rules, each said rule setting forth one or more conditions and one conclusion;

generating expanded rules based upon said rules and a topology of a monitored portion of the information system so that each condition of said expanded rules corresponds to an event which can occur in the monitored portion of the information system;

instantiating a plurality of conditions of a plurality of said expanded rules without repeat as condition objects in a memory;

instantiating a plurality of conclusions of a plurality of said expanded rules as conclusion objects in said memory;

associating a plurality of said condition objects with a plurality of said conclusion objects in said memory based upon a structure of said expanded rules; and

executing a process which specifies the cause of the occurrence by activating or deactivating one or more of said condition objects when an event which affects satisfaction of the one or more condition objects occurs.

36. An information system comprising:

a first computer in communication via a network with a plurality of monitored nodes in the information system;

a plurality of rules accessible by said first computer, each said rule setting forth one or more conditions and one conclusion;

wherein said first computer is configured to generate expanded rules based upon said rules and a topology of a monitored portion of the information system so that each condition of said expanded rules corresponds to an event which can occur in the monitored portion of the information system;

wherein said first computer is configured to instantiate a plurality of conditions of a plurality of said expanded rules without repeating as condition objects in a memory;

wherein said first computer is configured to instantiate a plurality of conclusions of a plurality of said expanded rules as conclusion objects in said memory;

wherein said first computer is configured to associate a plurality of said condition objects with a plurality of said conclusion objects in said memory based upon a structure of said expanded rules; and

wherein said first computer is configured to execute a process which specifies the cause of the occurrence by activating or deactivating one or more of said condition objects when an event which affects satisfaction of the one or more condition objects occurs.