Detecting and correcting a failure sequence in a computer system before a failure occurs

• US 7,181,651 B2
• Filed: 02/11/2004
• Issued: 02/20/2007
• Est. Priority Date: 02/11/2004
• Status: Active Grant

• Current Assignee: Oracle America, Inc. (Oracle Corporation)
• Original Assignee: Sun Microsystems Incorporated (Oracle Corporation)
• Inventors: Gross, Kenny C., Votta, Lawrence G. Jr., Porter, Adam
• Primary Examiner(s): Baderman; Scott
• Assistant Examiner(s): BONURA, TIMOTHY M
• Application Number:
  US10/777,532
• Publication Number:
  US 20050188263A1
• Time in Patent Office: 1,105 Days
• Field of Search: 714/37, 714/38, 714/39, 714/47
• US Class Current: 714/37
• CPC Class Codes: G06F 11/0706 the processing taking place on a specific hardware platform or in a specific software environment
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 11/366 using diagnostics G06F11/0703 takes precedence

Abstract

One embodiment of the present invention provides a system that detects a failure sequence that leads to undesirable computer system behavior and that subsequently takes a corresponding remedial action. During operation, the system receives instrumentation signals from the computer system while the computer system is operating. The system then uses these instrumentation signals to determine if the computer system is in a failure sequence that is likely to lead to undesirable system behavior, such as a system crash, wherein the determination involves considering predetermined multivariate correlations between multiple instrumentation signals and a failure sequence that is likely to lead to undesirable system behavior. Next, if the computer system is in a failure sequence that is likely to lead to undesirable system behavior, the system takes a remedial action.

Claims

1. A method for detecting a failure sequence or other undesirable system behavior in a computer system and subsequently taking a corresponding remedial action, comprising:

receiving instrumentation signals from the computer system while the computer system is operating;

determining from the instrumentation signals if the computer system is in a failure sequence that is likely to lead to undesirable system behavior, such as a system crash, wherein determining if the computer system is in a failure sequence involves;

determining correlations between instrumentation signals in the computer system, wherein determining the correlations involves using a non-linear, non-parametric regression technique to determine the correlations, whereby the correlations can subsequently be used to generate estimated signals,

deriving estimated signals for a number of instrumentation signals, wherein each estimated signal is derived from correlations with other instrumentation signals, and

comparing an actual signal with an estimated signal for a number of instrumentation signal to determine whether the computer system is in a failure sequence;

wherein the determination involves considering predetermined multivariate correlations between multiple instrumentation signals and a failure sequence that is likely to lead to undesirable system behavior; and

if the computer system is in a failure sequence that is likely to lead to undesirable system behavior, taking a remedial action.

2. The method of claim 1, wherein taking the remedial action involves generating an alarm.

3. The method of claim 2, wherein generating the alarm involves communicating the alarm to a system administrator so that the system administrator can take the remedial action.

4. The method of claim 3, wherein communicating the alarm to the system administrator involves communicating information specifying the nature of the failure sequence to the system administrator.

5. The method of claim 1, wherein taking the remedial action can involve:

killing processes, blocking creation of new processes, or throwing away work, until the system is no longer in a failure sequence that is likely to lead to undesirable system behavior.

6. The method of claim 1, wherein comparing an actual signal with an estimated signal involves using sequential detection methods to detect changes in a relationship between the actual signal and the estimated signal.

7. The method of claim 6, wherein the sequential detection methods include the Sequential Probability Ratio Test (SPRT).

8. The method of claim 1, wherein determining the correlations involves:

deliberately overloading the computer system during a test mode to produce undesirable system behavior, such as a system crash; and

identifying multivariate correlations between multiple instrumentation signals and the system crash.

9. The method of claim 1, wherein the non-linear, non-parametric regression technique can include a multivariate state estimation technique.

10. The method of claim 1, wherein the instrumentation signals can include:

signals associated with internal performance parameters maintained by software within the computer system;

signals associated with physical performance parameters measured through sensors the computer system; and

signals associated with canary performance parameters for synthetic user transactions, which are periodically generated for performance measuring purposes.

11. A computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for detecting a failure sequence or other undesirable system behavior in a computer system and subsequently taking a corresponding remedial action, wherein the computer-readable storage medium includes magnetic and optical storage devices, disk drives, magnetic tape, CDs (compact discs), and DVDs (digital versatile discs or digital video discs), the method comprising:

the method comprising;

receiving instrumentation signals from the computer system while the computer system is operating;

determining from the instrumentation signals if the computer system is in a failure sequence that is likely to lead to undesirable system behavior, such as a system crash;

wherein determining if the computer system is in a failure sequence involves;

determining correlations between instrumentation signals in the computer system, wherein determining the correlations involves using a non-linear, non-parametric regression technique to determine the correlations, whereby the correlations can subsequently be used to generate estimated signals,

deriving estimated signals for a number of instrumentation signals, wherein each estimated signal is derived from correlations with other instrumentation signals, and

comparing an actual signal with an estimated signal for a number of instrumentation signal to determine whether the computer system is in a failure sequence;

wherein the determination involves considering predetermined multivariate correlations between multiple instrumentation signals and a failure sequence that is likely to lead to undesirable system behavior; and

if the computer system is in a failure sequence that is likely to lead to undesirable system behavior, taking a remedial action.

12. The computer-readable storage medium of claim 11, wherein taking the remedial action involves generating an alarm.

13. The computer-readable storage medium of claim 12, wherein generating the alarm involves communicating the alarm to a system administrator so that the system administrator can take the remedial action.

14. The computer-readable storage medium of claim 13, wherein communicating the alarm to the system administrator involves communicating information specifying the nature of the failure sequence to the system administrator.

15. The computer-readable storage medium of claim 11, wherein taking the remedial action can involve:

killing processes, blocking creation of new processes, or throwing away work, until the system is no longer in a failure sequence that is likely to lead to undesirable system behavior.

16. The computer-readable storage medium of claim 11, wherein comparing an actual signal with an estimated signal involves using sequential detection methods to detect changes in a relationship between the actual signal and the estimated signal.

17. The computer-readable storage medium of claim 16, wherein the sequential detection methods include the Sequential Probability Ratio Test (SPRT).

18. The computer-readable storage medium of claim 11, wherein determining the correlations involves:

deliberately overloading the computer system during a test mode to produce undesirable system behavior, such as a system crash; and

identifying multivariate correlations between multiple instrumentation signals and the system crash.

19. The computer-readable storage medium of claim 11, wherein the non-linear, non-parametric regression technique can include a multivariate state estimation technique.

20. The computer-readable storage medium of claim 11, wherein the instrumentation signals can include:

signals associated with internal performance parameters maintained by software within the computer system;

signals associated with physical performance parameters measured through sensors the computer system; and

signals associated with canary performance parameters for synthetic user transactions, which are periodically generated for performance measuring purposes.

21. An apparatus that detects a failure sequence or other undesirable system behavior in a computer system and subsequently takes a corresponding remedial action, comprising:

a monitoring mechanism configured to monitor instrumentation signals from the computer system while the computer system is operating;

a determination mechanism configured to determine from the instrumentation signals if the computer system is in a failure sequence that is likely to lead to undesirable system behavior, such as a system crash, wherein determining if the computer system is in a failure sequence involves;

determining correlations between instrumentation signals in the computer system, wherein determining the correlations involves using a non-linear, non-parametric regression technique to determine the correlations, whereby the correlations can subsequently be used to generate estimated signals,

deriving estimated signals for a number of instrumentation signals, wherein each estimated signal is derived from correlations with other instrumentation signals, and

comparing an actual signal with an estimated signal for a number of instrumentation signal to determine whether the computer system is in a failure sequence;

wherein the determination mechanism is based on multivariate correlations between multiple instrumentation signals and a failure sequence that is likely to lead to undesirable system behavior; and

a remediation mechanism that is configured to take a remedial action if the computer system is in a failure sequence that is likely to lead to undesirable system behavior.