System and method for fault detection and recovery

• US 7,386,764 B2
• Filed: 11/29/2005
• Issued: 06/10/2008
• Est. Priority Date: 11/28/2001
• Status: Expired due to Fees

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Kolluru, Nagendra V., Lash, John K., LeDuc, Douglas E.
• Primary Examiner(s): MASKULINSKI, MICHAEL C
• Application Number:
  US11/290,777
• Publication Number:
  US 20060143492A1
• Time in Patent Office: 924 Days
• Field of Search: 714/2, 714/48
• US Class Current: 714/48
• CPC Class Codes: G06F 11/0715 in a system implementing multitasking multitasking per se G06F9/46
• CPC Class Codes: G06F 11/076 by exceeding a count or rate limit, e.g. word- or bit count limit
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40

Abstract

An apparatus and method for automatically detecting and recovering from a fault in a microprocessor-based system. The apparatus and method utilizes a leaky bucket routine and an event handler procedure. The method may further use Object Oriented techniques that abstracts differences between hardware and software faults to allow for the development of a common framework.

Claims

1. A method for automatically detecting and recovering from a fault in a microprocessor-based system, comprising:

capturing the fault, by the microprocessor-based system, as an event;

processing the event, by the microprocessor-based system, including thresholding the event and co-relating the event to a cause;

determining, by the microprocessor-based system, an elapsed time the microprocessor-based system has been running;

determining a recovery action, by the microprocessor-based system, as a function of the thresholding, the co-relating, and the elapsed time the microprocessor-based system has been running, and the recovery action is used to perform at least one of a restart and a cleanup of memory and

performing, by the microprocessor-based system, the recovery action.

2. The method as recited in claim 1, wherein the fault is indicative of at least one of a heartbeat failure, a hardware error, a hardware malfunction, and a software process that is failing to respond.

3. The method as recited in claim 1, wherein the microprocessor-based system is defined as a hierarchical set of objects to model the software and hardware components of the microprocessor-based system.

4. The method as recited in claim 3, wherein the thresholding comprises using a leaky bucket algorithm and error counting.

5. The method as recited in claim 4, further comprising using relationships imposed by the hierarchical set of objects when determining a recovery action.

6. The method as recited in claim 5, where the objects represent at least one of hardware, software, and external entities the system communicates with.

7. The method as recited in claim 6, where event histories and system state information in combination with the objects in the hierarchy that span the hardware software and external entity domains are used to determine appropriate recovery actions for events in the system.

8. The method as recited in claim 7, where root cause correlation of events occurs across all domains and hierarchies of the system to determine recovery actions that provide for specific recoveries and system escalations.

9. The method as recited in claim 8, wherein sources of the events include at least one of monitor processes, software, driver/system call errors, hardware interrupts, and event detectors that proactively detect faults in unused components of the system.

10. The method as recited in claim 1, wherein the recovery actions is additionally used to perform at least one of a reset to a hardware module, a reset to a sub-assembly, and a releasing of resources that are marked unavailable.

11. A method for automatically detecting and recovering from a fault in a microprocessor-based system defined as a hierarchical set of objects to model software, hardware, and external entities the system communicates with, the method comprising:

capturing, by the microprocessor-based system, the fault as an event;

processing, by the microprocessor-based system, the event including thresholding the event and co-relating the event to a cause, the thresholding using a leaky bucket algorithm and error counting;

determining, by the microprocessor-based system, an elapsed time the microprocessor-based system has been running;

determining, by the microprocessor-based system, a recovery action as a function of the thresholding, the co-relating, and the elapsed time the microprocessor-based system has been running, relationships imposed by the hierarchical set of objects, event histories, and system state information, wherein root cause correlation of events occurs across all domains and hierarchies of the system to determine recovery actions that provide for specific recoveries and system escalations, and the recovery action is used to perform at least one of a restart and a cleanup of memory; and

performing, by the microprocessor-based system, the recovery action.

12. The method as recited in claim 11, wherein the recovery action is additionally used to perform at least one of a reset to a hardware module, a reset to a sub-assembly, and a releasing of resources that are marked unavailable.

13. A method of automatically detecting and recovering from a fault in a microprocessor-based system, comprising:

capturing, by the microprocessor-based system, the fault as an event

processing, by the microprocessor-based system, the event including thresholding the event and co-relating the event to a cause selected from a list of potential causes, the list of potential causes including potential hardware causes and potential software causes;

determining, by the microprocessor-based system, an elapsed time the microprocessor-based system has been running;

determining, by the microprocessor-based system, a recovery action as a function of the thresholding and the co-relating and the elapsed time the microprocessor-based system has been running, and the recovery action is used to perform at least one of a restart and a cleanup of memory; and

performing, by the microprocessor-based system, the recovery action.

14. The method as recited in claim 13, wherein the microprocessor-based system is defined as a hierarchical set of objects to model software and hardware components of the microprocessor-based system.

15. The method as recited in claim 14, further comprising using relationships imposed by the hierarchical set of objects when determining the recovery action.

16. The method as recited in claim 13, wherein the recovery action is additionally used to perform at least one of a reset to a hardware module, a reset to a sub-assembly, and a releasing of resources that are marked unavailable.

17. A method of automatically detecting and recovering from a fault in a microprocessor-based system, comprising:

capturing, by the microprocessor-based system, the fault as an event;

processing, by the microprocessor-based system, the event including thresholding the event and co-relating the event to a cause selected from a list of potential causes, the list of potential causes including potential causes internal to the microprocessor-based system and potential causes external to the microprocessor-based system;

determining, by the microprocessor-based system, an elapsed time the microprocessor-based system has system has been running;

determining, by the microprocessor-based system, a recovery action as a function of the thresholding and the co-relating and the elapsed time the microprocessor-based system has been running, and the recovery action is used to perform at least one of a restart and a cleanup of memory; and

performing, by the microprocessor-based system, the recovery action.

18. The method as recited in claim 17, wherein the microprocessor-based system is defined as a hierarchical set of objects to model software and hardware components of the microprocessor-based system.

19. The method as recited in claim 18, further comprising using relationships imposed by the hierarchical set of objects when determining the recovery action.

20. The method as recited in claim 17, wherein the recovery action is additionally used to perform at least one of a reset to a hardware module, a reset to a sub-assembly, and a releasing of resources that are marked unavailable.

21. A method for recovering from a fault in a microprocessor-based system, the method comprising:

capturing, by the microprocessor-based system, the fault as an event;

determining, by the microprocessor-based system, an elapsed time the microprocessor-based system has been running;

determining, by the microprocessor-based system, a recovery action as a function of the elapsed time the microprocessor-based system has been running, and the recovery action is used to perform at least one of a restart and a cleanup of memory; and

performing, by the microprocessor-based system, the recovery action.

22. The method as recited in claim 21, wherein the microprocessor-based system is defined as a hierarchical set of objects to model software and hardware components of the microprocessor-based system and relationships imposed by the hierarchical set of objects are used to determine the recovery action.