Fault handling process for enabling recovery, diagnosis, and self-testing of computer systems

• US 6,691,250 B1
• Filed: 06/29/2000
• Issued: 02/10/2004
• Est. Priority Date: 06/29/2000
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Roeck, Guenter E., Chandiramani, Anil K.
• Primary Examiner(s): Beausoliel, Robert
• Assistant Examiner(s): PUENTE, EMERSON C
• Application Number:
  US09/607,108
• Time in Patent Office: 1,321 Days
• Field of Search: 714/25, 714/26, 714/47, 714/48, 714/15, 714/37
• US Class Current: 714/25
• CPC Class Codes: G06F 11/0706 the processing taking place on a specific hardware platform or in a specific software environment
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40

Abstract

Methods, apparatus, and computer program products are disclosed for analyzing and recovering from severe to catastrophic faults in a computer system. When a fault that cannot be handled by the computer system'"'"'s normal fault handling processes, a shadow mode created by a fault handling virtual machine is invoked. The fault handling virtual machine executes only when the normally nonrecoverable fault is encountered and executes as a triangulated or shadow mode on the system. Once shadow mode is invoked, fault context data is collected on the system and used to analyze and recover from the fault. More specifically, one or more post-fault stable states are constructed by the fault handling virtual machine. These stable states are used to bring the computer system back to a normal operating state in which the component or action causing the initial nonrecoverable fault is avoided. Persistent faults may be encountered while the virtual machine is attempting to recover from the initial fault.

Claims

1. A method of handling a fault in a computer system comprising:

recognizing that an initial fault has occurred by operating in a first sequence in the computer system;

invoking an alternative mode of operation for the computer system upon recognizing the initial fault;

using the alternative mode to track performance of the system after the initial fault thereby gathering post-fault state information for fault diagnosis and system recovery; and

preventing a subsequent fault from reoccurring as a result of recovery from the initial fault by using a dynamic state of the computer system to cause the computer system to operate in a second sequence such that the initial fault and the subsequent fault are potentially avoided.

2. A method as recited in claim 1 further comprising installing a fault handling virtual machine residing on the computer system when the initial fault cannot be handled by an existing fault handling process.

3. A method as recited in claim 2 further comprising developing expert knowledge based on observation of a dynamic response of the computer system to fault conditions, wherein the expert knowledge is used by the fault handling virtual machine so that the computer system can recover from a fault.

4. A method as recited in claim 2 further comprising continuing operation of the existing fault handler process on the computer system while the alternative mode is in operation.

5. A method as recited in claim 2 further comprising determining whether the initial fault is a fault created by the fault handling virtual machine in the course of deriving and testing fault recovery algorithms for a plurality of targeted faults.

6. A method as recited in claim 1 wherein using the alternative mode further comprises saving a fault context of the computer system from the initial fault.

7. A method as recited in claim 1 further comprising constructing a post-fault stable state of the computer system using the dynamic state data and a predefined set of methods, thereby allowing the computer system to recover from the initial fault and resume operation.

8. A method as recited in claim 7 wherein constructing a post-fault stable state further comprises utilizing a computer system state at the time of the initial fault and not requiring a previously stored system data check point.

11. A method as recited in claim 1 wherein, when the computer system is operating normally, the alternative mode of operation is not invoked, thereby not utilizing computer system resources.

12. A method as recited in claim 1 further comprising collecting the system state data on the computer system in a manner that does not interfere with normal operation of the computer system.

13. A method as recited in claim 1 further comprising using the alternative mode to supervise normal operation of the computer system, wherein normal operation includes user mode and supervisor mode.

14. A method as recited in claim 1 further comprising entering the alternative mode without effecting processes executing in the system.

15. A method as recited in claim 1 further comprising handling one or more persistent faults in alternative mode after detecting the initial fault.

16. A method as recited in claim 1 wherein the initial fault would cause the computer to crash if the alternative mode of operation had not been invoked.

17. A method as recited in claim 1 further comprising utilizing a fault instance handler to fix predetermined faults.

18. A method as recited in claim 1 further comprising creating a detailed fault context containing information that would normally not be gathered by an existing fault handling mechanism.

19. A method as recited in claim 1 further comprising developing a training mode for destructive and non-destructive fault simulation to test the computer system.

9. A method as recited in 7 further comprising mutating the post-fault stable state by performing one of progressing the post-fault stable state and regressing the post-fault stable state, and observing a response of the computer system to the mutated post-fault stable state.

10. A method as recited in claim 9 further comprising iteratively modifying the mutated post-fault stable state to achieve desired fault recovery characteristics for the computer system.

20. A fault handling virtual machine installed on a computer system upon detection of an unrecoverable fault, the fault handling virtual machine comprising:

a post fault stable state constructor for constructing a normal operating state for the computer system after a fault occurs;

a fault data collector for collecting specific information on the state of the computer system at the time of the fault; and

a fault data examination component for examining the specific information on the state of the computer system after a fault occurs.

21. A fault handling virtual machine as recited in claim 20 further including a persistent fault handler.

22. A fault handling virtual machine as recited in claim 20 that extends the capabilities of the computer system to fault diagnosis and recovery by applying expert knowledge of the computer system.

23. A fault handling virtual machine as recited in claim 20 that emulates operation of the computer system.

24. A fault handling virtual machine as recited in claim 20 further including a fault type component for determining whether the fault is a training mode fault.

25. A fault handling virtual machine as recited in claim 20 further including a fault severity measuring component for determining the severity of a fault by looking at expert knowledge of the computer system and a current fault state.

26. A fault handling virtual machine as recited in claim 20 wherein the post-fault stable state constructor performs iteratively.

27. A fault handling virtual machine as recited in claim 20 further including a context check component for checking the normal state of the computer.

28. A fault handling virtual machine as recited in claim 20 further including a fault context validator for saving and validating state information at a specific time.

29. A fault handling component in a computer system for handling a severe fault comprising:

a means for recognizing that an initial fault has occurred by operating in a first sequence in the computer system;

a means for invoking an alternative mode of operation for the computer system upon recognizing the initial fault;

a means for tracking performance of the system after the initial fault using the alternative mode thereby gathering post-fault state information for fault diagnosis and system recovery; and

a means for preventing a subsequent fault from reoccurring as a result of recovery from the initial fault by using a dynamic state of the computer system to cause the computer system to operate in a second sequence such that the initial fault and the subsequent fault are potentially avoided.

30. A computer-readable medium containing programmed instructions arranged to handle a fault in a computer system, the computer-readable medium including programmed instructions for:

recognizing that an initial fault has occurred by operating in a first sequence in the computer system;

invoking an alternative mode of operation for the computer system upon recognizing the initial fault;

using the alternative mode to track performance of the system after the initial fault thereby gathering post-fault state information for fault diagnosis and system recovery; and

preventing a subsequent fault from reoccurring as a result of recovery from the initial fault by using a dynamic state of the computer system to cause the computer system to operate in a second sequence such that the initial fault and the subsequent fault are potentially avoided.

31. A component in a computer system for handling a fault in a computer system, the component comprising:

a memory; and

a processor coupled to the memory, wherein the processor is programmed to perform the steps of;

recognizing that an initial fault has occurred by operating in a first sequence in the computer system;

invoking an alternative mode of operation for the computer system upon recognizing the initial fault;

using the alternative mode to track performance of the system after the initial fault thereby gathering post-fault state information for fault diagnosis and system recovery; and

preventing a subsequent fault from reoccurring as a result of recovery from the initial fault by using a dynamic state of the computer system to cause the computer system to operate in a second sequence such that the initial fault and the subsequent fault are potentially avoided.