Hybrid event prediction and system control

• US 7,895,323 B2
• Filed: 11/10/2008
• Issued: 02/22/2011
• Est. Priority Date: 11/24/2003
• Status: Expired due to Fees

• Current Assignee: GlobalFoundries, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Oliner, Adam J., Sahoo, Ramendra K., Moreira, Jose E., Gupta, Manish
• Primary Examiner(s): Follansbee; John
• Assistant Examiner(s): Mejia; Anthony
• Application Number:
  US12/267,762
• Publication Number:
  US 20090070628A1
• Time in Patent Office: 834 Days
• Field of Search: 709/223, 709/224, 709/226, 714/26, 706/21, 706/25, 706/908, 706/916
• US Class Current: 709/224
• CPC Class Codes: G06F 11/008 Reliability or availability analysis
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3058 Monitoring arrangements for monitoring environmental properties or parameters of the computing system or of the computing system component, e.g. monitoring of power, currents, temperature, humidity, position, vibrations thermal management in cooling arrangements of a computing system G06F1/206
• CPC Class Codes: G06F 11/3447 Performance evaluation by modeling
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: Y10S 706/908 Electronic or computer, internal or network, circuit
• CPC Class Codes: Y10S 706/916 Electronic or computer, internal or network, circuit

Abstract

A system for predicting an occurrence of a critical even in a computer cluster includes: a control system that includes an event log, a system parameter log, a memory for storing information related to occurrences of critical events, and a processor. The processor implements a hybrid prediction system; loads the information from the event log and the system performance log into a Bayesian network model; uses the Bayesian network model to predict a future critical event; makes future scheduling and current data migration selections; and adapts the Bayesian network model by feeding the scheduling and data migration selections.

Claims

1. An information processing system for predicting an occurrence of a critical event in a computer cluster, the information processing system comprising:

the computer cluster comprising a series of nodes;

a control system comprising;

A) an event log comprising information concerning critical events that occur in the computer cluster, wherein said critical events adversely affect performance of the cluster or one of its nodes;

B) a system parameter log comprising information concerning system performance parameters for each node in the cluster;

C) a processor configured for;

i) implementing a hybrid prediction system comprising rule based prediction algorithms, time-dependent variable prediction algorithms, and a warning window;

ii) maintaining the event log for only those nodes in which an error has occurred;

iii) maintaining the system parameter log for each node in the cluster;

iv) for only those nodes in which an error has occurred, loading the information from the event log and the system performance information pertaining to said error-prone nodes from the system parameter log into a Bayesian network model representing a correspondence between the system performance parameters and occurrence of the critical events;

v) using the Bayesian network model to predict a future critical event within a specified time-limit based upon the hybrid prediction system;

vi) making future scheduling and current data migration selections based upon the hybrid prediction system; and

vii) adapting the Bayesian Network Model by feeding the scheduling and data migration selections into said Bayesian Network Model; and

D) a memory for storing information related to said occurrence of said critical events and said system parameters of said nodes.

2. The information processing system of claim 1 further comprising E) a filter for filtering the event log and the system parameter log such that some critical event information and some system parameter information is eliminated in order to reduce storage requirements of the cluster.

3. The information processing system of claim 1 wherein said Bayesian Network comprises a time-series modeler for predicting future values of said system parameters.

4. The information processing system of claim 1 wherein said Bayesian Network comprises a rule based classification system for associating said system parameters with said occurrences of said critical events.

5. The information processing system of claim 1 further comprising:

F) a dynamic probe generator for determining when to collect additional information concerning said system parameters or said critical event occurrence.

6. The information processing system of claim 1 wherein the processor is further configured for maintaining the event log, comprising:

aligning the information concerning the critical events; and

categorizing the information concerning the critical events according to time-dependency.

7. The information processing system of claim 1 wherein the processor is further configured for maintaining the system parameter log, comprising:

recording a temperature of the nodes in the cluster and a corresponding time value; and

recording a utilization parameter of a central processing unit of a node in the cluster and a corresponding time value.

8. The information processing system of claim 1 wherein the rule-based prediction algorithms use associative rules based upon the critical event information and the system parameter information for predicting a probable occurrence of the critical events within a specified time window and the variables that are likely to indicate a potential occurrence of such an event.

9. The information processing system of claim 1 wherein the time-dependent variable prediction algorithms generate time-series mathematical models that predict future values of the system performance parameters.

10. The information processing system of claim 9 wherein the warning window is formed for only those nodes in the cluster in which at least one error has occurred in order to reduce system requirements, wherein said warning window comprises a predicted performance parameter or critical event occurrence for the node for a predetermined future period of time.

11. The information processing system of claim 1 further comprising an input output subsystem.

12. A non-transitory computer readable storage medium comprising program instructions that predicts an occurrence of a critical event in a computer cluster by:

implementing a hybrid prediction system comprising rule based prediction algorithms, time-dependent variable prediction algorithms, and a warning window for a computer cluster comprising a series of nodes, said hybrid prediction system carrying out instructions for;

maintaining an event log for only those nodes in which an error has occurred, wherein the error adversely affects performance of the cluster or one of its nodes;

maintaining a system parameter log for each node in the computer cluster;

for only those nodes in which an error has occurred, loading the information from the event log into a Bayesian network model representing a correspondence between the system performance parameters and occurrence of the critical events;

for only those nodes in which an error has occurred, loading system performance information pertaining to said error-prone nodes from a system parameter log comprising information concerning system performance parameters for each node in the cluster into the Bayesian network model;

using the Bayesian network model to predict a future critical event within a specified time-limit based upon the hybrid prediction system;

storing information related to said occurrence of said critical events and said system parameters of said nodes;

making future scheduling and current data migration selections based upon the hybrid prediction system; and

adapting the Bayesian Network Model by feeding the scheduling and data migration selections into said Bayesian Network Model.

13. The non-transitory computer readable storage medium of claim 12 wherein the instructions further comprise:

filtering the event log and the system parameter log such that some critical event information and some system parameter information is eliminated in order to reduce storage requirements of the cluster.

14. The non-transitory computer readable storage medium of claim 12 wherein the instructions further comprise using a dynamic probe generator for determining when to collect additional information concerning said system parameters or said critical event occurrence.

15. The non-transitory computer readable storage medium of claim 12 wherein the instructions further maintain the event log, comprising steps of:

aligning the information concerning the critical events; and

categorizing the information concerning the critical events according to time-dependency.

16. The non-transitory computer readable storage medium of claim 12 wherein the instructions further maintain the system parameter log, comprising steps of:

recording a temperature of the nodes in the cluster and a corresponding time value; and

recording a utilization parameter of a central processing unit of a node in the cluster and a corresponding time value.

17. The non-transitory computer readable storage medium of claim 12 wherein the warning window is formed for only those nodes in the cluster in which at least one error has occurred in order to reduce system requirements, wherein said warning window comprises a predicted performance parameter or critical event occurrence for the node for a predetermined future period of time.

18. The non-transitory computer readable storage medium of claim 12 further comprising an input output subsystem.