Causal ladder mechanism for proactive problem determination, avoidance and recovery

• US 7,349,826 B2
• Filed: 05/23/2006
• Issued: 03/25/2008
• Est. Priority Date: 05/23/2006
• Status: Active Grant

• Current Assignee: Trend Micro Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Wesley, Ajamu A., Subramanian, Balan, Chalasani, Nanchariah Raghuveera, Rahman, Javed
• Primary Examiner(s): Barlow, John
• Assistant Examiner(s): LE, JOHN H
• Application Number:
  US11/438,875
• Publication Number:
  US 20070276631A1
• Time in Patent Office: 672 Days
• Field of Search: 702/57, 702/58, 702182-185, 706/15, 706/20, 705/8, 703/14, 709/201, 709/232, 714/11, 714/26, 714/38, 714/43, 714/728, 455/431
• US Class Current: 702/185
• CPC Class Codes: G06F 11/008 Reliability or availability analysis

Abstract

A plurality of causal ladder is assembled in advance from component system events taken from previous system failures. The ladders classify the various transitions the system goes through from one set of observed states to another in multiple stages representing issues of differing urgency, importance and need for remediation. These stages are used at runtime determine the criticality of any abnormal system activity and to accurately predict the component failure prior to the system crashing. Each ladder comprises a plurality of elevated stages representing criticality of the problem. At runtime, the causal ladder engine correlates real-time events received from the system to stages of one or more pre-constructed causal ladders and identifies a probable problem (and/or the faulty component) from the corresponding causal ladder. The causal ladder engine also determines the stage of the problem from event occurrences. At each stage, a different potential solution is identified for the problem.

Claims

1. A method for proactive problem determination and component failure avoidance comprising:

assembling a plurality of causal ladders, each of the plurality of causal ladders representing a plurality of sequential stages of criticality of a component problem that resulted in a system failure, wherein assembling a plurality of causal ladders comprises;

receiving a plurality of failure events that resulted in a previous failure of a component;

identifying a component problem related to an occurrence of each of the plurality of failure events;

correlating each of the plurality of failure events to a discrete sequential stage of criticality in a causal ladder representing a respective identified component problem; and

assigning at least one remedy to each discrete sequential stage of criticality of the causal ladder representing the respective identified component problem;

for each of the plurality of sequential stages of criticality in each of the respective causal ladders, associating at least one remedy for a component problem;

identifying a potential component failure from one of the plurality of causal ladders representing a component problem relating to the potential component failure, wherein identifying a potential component failure from one of the plurality of causal ladders comprises;

receiving a runtime system event;

correlating the received runtime system event to the at least one of the plurality of failure events correlated to a discrete sequential stage of criticality in one of the plurality of causal ladders representing the identified component problem; and

determining a criticality of the identified component problem on the correlated discrete sequential stage of criticality in the causal ladder; and

determining a problem solution for the component problem based on the one of the plurality of causal ladders representing the component problem, wherein determining a problem solution for the component problem comprises;

accessing policy associated with the correlated discrete sequential stage of criticality in the causal ladder;

ranking the at least one remedy for the correlated discrete sequential stage of criticality in the causal ladder representing the identified component problem based on policy associated with the correlated discrete sequential stage; and

selecting a highest ranking solution for the identified component problem.

2. The method recited in claim 1, further comprising:

ranking the at least one remedy for the correlated discrete sequential stage of criticality in the causal ladder representing the identified component problem based on a previous performance in remedying the component problem.

3. The method recited in claim 2, wherein correlating the received runtime system event to the at least one of the plurality of failure events correlated to a discrete sequential stage of criticality in one of the plurality of causal ladders representing the identified component problem, further comprises:

determining an estimated time until failure for each of the plurality of sequential discrete stages of criticality in each of the plurality of causal ladders from a previous component problem; and

associating the respective estimated time until failure with each of the plurality of sequential discrete stages of criticality in each of the plurality of causal ladders.

4. The method recited in claim 3, wherein assigning at least one remedy to each discrete sequential stage of criticality of the causal ladder representing the respective identified component problem, further comprises:

associating an estimated repair time to each of the at least one remedy for each discrete sequential stage of criticality of the causal ladder.

5. The method recited in claim 4, wherein determining a problem solution for the component problem based on the one of the plurality of causal ladders representing the component problem, further comprises:

for each remedy assigned to the correlated discrete sequential stage of criticality of the causal ladder;

identify all remedies having an estimated repair time that is greater than an to estimated time until failure; and

excluding the identified remedies from ranking.

6. The method recited in claim 5, further comprising:

receiving a second runtime system event;

correlating the received second runtime system event to at least a second of the plurality of failure events assigned to second discrete sequential stage of criticality in one of the plurality of causal ladders representing the component problem, wherein at least one other runtime system event has correlated to the one of the plurality of causal ladders representing the component problem;

determining a second criticality of the component problem based the correlated second discrete sequential stage of criticality from the correlated causal ladder; and

elevating criticality of the component problem based the correlated another discrete sequential stage of criticality from the correlated causal ladder.

7. The method recited in claim 6, further comprising:

accessing policy associated with the correlated second discrete sequential stage of criticality from the correlated causal ladder;

retrieving an estimated time until failure for the correlated second discrete sequential stage of criticality from the correlated causal ladder;

retrieving a plurality of remedies associated with the second correlated discrete sequential stage of criticality of the causal ladder representing the identified component problem;

retrieving an estimated repair time associated with each of the plurality of remedies associated with the second correlated discrete sequential stage of criticality;

identifying any remedies of the plurality of remedies with an associated estimated repair time that is greater than the estimated time until failure for the correlated second discrete sequential stage of criticality from the correlated causal ladder;

excluding the identified any remedies from the plurality of remedies;

ranking the plurality of remedies based on policy associated with the correlated second discrete sequential stage of criticality; and

selecting a highest ranking of the plurality of remedies for the identified component problem.

8. The method recited in claim 1, further comprising:

constructing a hidden Markov model for each causal ladder.

9. A method for proactive problem determination and component failure avoidance comprising:

assembling a plurality of causal ladders, each of the plurality of causal ladders representing a plurality of sequential stages of criticality of a component problem that resulted in a system failure, wherein assembling a plurality of causal ladders comprises;

receiving a plurality of failure events that resulted in a previous failure of a component;

identifying a component problem related to an occurrence of each of the plurality of failure events;

correlating each of the plurality of failure events to a discrete sequential stage of criticality in a causal ladder representing a respective identified component problem; and

assigning at least one remedy to each discrete sequential stage of criticality of the causal ladder representing the respective identified component problem;

for each of the plurality of sequential stages of criticality in each of the respective causal ladders, associating at least one remedy for a component problem;

identifying a potential component failure from one of the plurality of causal ladders representing a component problem relating to the potential component failure, wherein identifying a potential component failure from one of the plurality of causal ladders comprises;

receiving a runtime system event;

correlating the received runtime system event to the at least one of the plurality of failure events correlated to a discrete sequential stage of criticality in one of the plurality of causal ladders representing the identified component problem; and

determining a criticality of the identified component problem based on the correlated discrete sequential stage of criticality in the causal ladder; and

determining a problem solution for the component problem based on the one of the plurality of causal ladders representing the component problem, wherein determining a problem solution for the component problem comprises;

for each remedy assigned to the correlated discrete sequential stage of criticality of the causal ladder;

identify all remedies having an estimated repair time that is greater than an estimated time until failure; and

excluding the identified remedies from ranking;

accessing policy associated with the correlated discrete sequential stage of criticality in the causal ladder;

ranking the at least one remedy for the correlated discrete sequential stage of criticality in the causal ladder representing the identified component problem based on policy associated with the correlated discrete sequential stage; and

selecting a highest ranking solution for the identified component problem.

10. The method recited in claim 9, further comprising:

ranking the at least one remedy for the correlated discrete sequential stage of criticality in the causal ladder representing the identified component problem based on a previous performance in remedying the component problem.

11. The method recited in claim 9, wherein correlating the received runtime system event to the at least one of the plurality of failure events correlated to a discrete sequential stage of criticality in one of the plurality of causal ladders representing the identified component problem, further comprises:

determining an estimated time until failure for each of the plurality of sequential discrete stages of criticality in each of the plurality of causal ladders from a previous component problem; and

associating the respective estimated time until failure with each of the plurality of sequential discrete stages of criticality in each of the plurality of causal ladders.

12. The method recited in claim 9, wherein assigning at least one remedy to each discrete sequential stage of criticality of the causal ladder representing the respective identified component problem, further comprises:

associating an estimated repair time to each of the at least one remedy for each discrete sequential stage of criticality of the causal ladder.

13. The method recited in claim 9, further comprising:

receiving a second runtime system event;

correlating the received second runtime system event to at least a second of the plurality of failure events assigned to second discrete sequential stage of criticality in one of the plurality of causal ladders representing the component problem, wherein at least one other runtime system event has correlated to the one of the plurality of causal ladders representing the component problem;

determining a second criticality of the component problem based on the correlated second discrete sequential stage of criticality from the correlated causal ladder; and

elevating criticality of the component problem based on the correlated another discrete sequential stage of criticality from the correlated causal ladder.

14. The method recited in claim 9, further comprising:

accessing policy associated with the correlated second discrete sequential stage of criticality from the correlated causal ladder;

retrieving an estimated time until failure for the correlated second discrete sequential stage of criticality from the correlated causal ladder;

retrieving a plurality of remedies associated with the second correlated discrete sequential stage of criticality of the causal ladder representing the identified component problem;

retrieving an estimated repair time associated with each of the plurality of remedies associated with the second correlated discrete sequential stage of criticality;

identifying any remedies of the plurality of remedies with an associated estimated repair time that is greater than the estimated time until failure for the correlated second discrete sequential stage of criticality from the correlated causal ladder;

excluding the identified any remedies from the plurality of remedies;

ranking the plurality of remedies based on policy associated with the correlated second discrete sequential stage of criticality; and

selecting a highest ranking of the plurality of remedies for the identified component problem.

15. A method for proactive problem determination and component failure avoidance comprising:

assembling a plurality of causal ladders, each of the plurality of causal ladders representing a plurality of sequential stages of criticality of a component problem that resulted in a system failure, wherein assembling a plurality of causal ladders comprises;

receiving a plurality of failure events that resulted in a previous failure of a component;

identifying a component problem related to an occurrence of each of the plurality of failure events;

correlating each of the plurality of failure events to a discrete sequential stage of criticality in a causal ladder representing a respective identified component problem; and

assigning at least one remedy to each discrete sequential stage of criticality of the causal ladder representing the respective identified component problem;

for each of the plurality of sequential stages of criticality in each of the respective causal ladders, associating at least one remedy for a component problem;

identifying a potential component failure from one of the plurality of causal ladders representing a component problem relating to the potential component failure, wherein identifying a potential component failure from one of the plurality of causal ladders comprises;

receiving a runtime system event;

correlating the received runtime system event to the at least one of the plurality of failure events correlated to a discrete sequential stage of criticality in one of the plurality of causal ladders representing the identified component problem, wherein correlating the received runtime system event to the at least one of the plurality of failure events correlated to a discrete sequential stage of criticality in one of the plurality of causal ladders representing the identified component problem, further comprises;

determining an estimated time until failure for each of the plurality of sequential discrete stages of criticality in each of the plurality of causal ladders from a previous component problem; and

associating the respective estimated time until failure with each of the plurality of sequential discrete stages of criticality in each of the plurality of causal ladders; and

determining a criticality of the identified component problem based on the correlated discrete sequential stage of criticality in the causal ladder; and

determining a problem solution for the component problem based on the one of the plurality of causal ladders representing the component problem, wherein determining a problem solution for the component problem comprises;

accessing policy associated with the correlated discrete sequential stage of criticality in the causal ladder;

ranking the at least one remedy for the correlated discrete sequential stage of criticality in the causal ladder representing the identified component problem based on policy associated with the correlated discrete sequential stage; and

selecting a highest ranking solution for the identified component problem.

16. The method recited in claim 15, further comprising:

ranking the at least one remedy for the correlated discrete sequential stage of criticality in the causal ladder representing the identified component problem based on a previous performance in remedying the component problem.

17. The method recited in claim 15, wherein assigning at least one remedy to each discrete sequential stage of criticality of the causal ladder representing the respective identified component problem, further comprises:

associating an estimated repair time to each of the at least one remedy for each discrete sequential stage of criticality of the causal ladder.

18. The method recited in claim 15, wherein determining a problem solution for the component problem based on the one of the plurality of causal ladders representing the component problem, further comprises:

for each remedy assigned to the correlated discrete sequential stage of criticality of the causal ladder;

identify all remedies having an estimated repair time that is greater than an estimated time until failure; and

excluding the identified remedies from ranking.

19. The method recited in claim 15, further comprising:

receiving a second runtime system event;

correlating the received second runtime system event to at least a second of the plurality of failure events assigned to second discrete sequential stage of criticality in one of the plurality of causal ladders representing the component problem, wherein at least one other runtime system event has correlated to the one of the plurality of causal ladders representing the component problem;

determining a second criticality of the component problem based on the correlated second discrete sequential stage of criticality from the correlated causal ladder; and

elevating criticality of the component problem based on the correlated another discrete sequential stage of criticality from the correlated causal ladder.

20. The method recited in claim 15, further comprising:

accessing policy associated with the correlated second discrete sequential stage of criticality from the correlated causal ladder;

retrieving an estimated time until failure for the correlated second discrete sequential stage of criticality from the correlated causal ladder;

retrieving a plurality of remedies associated with the second correlated discrete sequential stage of criticality of the causal ladder representing the identified component problem;

retrieving an estimated repair time associated with each of the plurality of remedies associated with the second correlated discrete sequential stage of criticality;

identifying any remedies of the plurality of remedies with an associated estimated repair time that is greater than the estimated time until failure for the correlated second discrete sequential stage of criticality from the correlated causal ladder;

excluding the identified any remedies from the plurality of remedies;

ranking the plurality of remedies based on policy associated with the correlated second discrete sequential stage of criticality; and

selecting a highest ranking of the plurality of remedies for the identified component problem.