System and method for fault detection and recovery

• US 20060143492A1
• Filed: 11/29/2005
• Published: 06/29/2006
• Est. Priority Date: 11/28/2001
• Status: Active Grant

• Current Assignee: Douglas E. Leduc, John K. Lash, Nagendra V. Kolluru
• Original Assignee: Douglas E. Leduc, John K. Lash, Nagendra V. Kolluru
• Inventors: Kolluru, Nagendra V., Lash, John K., LeDuc, Douglas E.
• Granted Patent: US 7,386,764 B2
• Time in Patent Office: Days
• US Class Current: 714/2
• CPC Class Codes: G06F 11/0715 in a system implementing multitasking multitasking per se G06F9/46
• CPC Class Codes: G06F 11/076 by exceeding a count or rate limit, e.g. word- or bit count limit
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40

Abstract

An apparatus and method for automatically detecting and recovering from a fault in a microprocessor-based system. The apparatus and method utilizes a leaky bucket routine and an event handler procedure. The method may further use Object Oriented techniques that abstracts differences between hardware and software faults to allow for the development of a common framework.

Claims

1. A method for automatically detecting and recovering from a fault in a microprocessor-based system, comprising:

reporting the fault as an event;

processing the event including thresholding the event and co-relating the event to a cause;

determining a recovery action as a function of the thresholding, the co-relating, and an elapsed time the system has been running; and

performing the recovery action.

2. The method as recited in claim 1, wherein the fault is indicative of at least one of a heartbeat failure, a hardware error, a hardware malfunction, and a software process that is failing to respond.

3. The method as recited in claim 1, wherein the microprocessor-based system is defined as a hierarchical set of objects to model the software and hardware components of the microprocessor-based system.

4. The method as recited in claim 3, wherein the thresholding comprises using a leaky bucket algorithm and error counting.

5. The method as recited in claim 4, further comprising using relationships imposed by the hierarchical set of objects when determining a recovery action.

6. The method as recited in claim 5, where the objects represent at least one of hardware, software, and external entities the system communicates with.

7. The method as recited in claim 6, where event histories and system state information in combination with the objects in the hierarchy that span the hardware software and external entity domains are used to determine appropriate recovery actions for events in the system.

8. The method as recited in claim 7, where root cause correlation of events occurs across all domains and hierarchies of the system to determine recovery actions that provide for specific recoveries and system escalations.

9. The method as recited in claim 8, wherein sources of the events include at least one of monitor processes, software, driver/system call errors, hardware interrupts, and event detectors that proactively detect faults in unused components of the system.

10. The method as recited in claim 1, wherein the recovery actions are used to perform at least one of a restart of the system, a cleanup of memory, a reset to a hardware module, a reset to a sub-assembly, and a releasing of resources that are marked unavailable.

11. A method for automatically detecting and recovering from a fault in a microprocessor-based system defined as a hierarchical set of objects to model software, hardware, and external entities the system communicates with, the method comprising:

reporting the fault as an event;

processing the event including thresholding the event and co-relating the event to a cause, the thresholding using a leaky bucket algorithm and error counting;

determining a recovery action as a function of the thresholding, the co-relating, an elapsed time the system has been running, relationships imposed by the hierarchical set of objects, event histories, and system state information, wherein root cause correlation of events occurs across all domains and hierarchies of the system to determine recovery actions that provide for specific recoveries and system escalations; and

performing the recovery action.

12. The method as recited in claim 11, wherein the recovery action is used to perform at least one of a restart of the microprocessor-based system, a cleanup of memory, a reset to a hardware module, a reset to a sub-assembly, and a releasing of resources that are marked unavailable.

13. A method of automatically detecting and recovering from a fault in a microprocessor-based system, comprising:

reporting the fault as an event

processing the event including thresholding the event and co-relating the event to a cause selected from a list of potential causes, the list of potential causes including potential hardware causes and potential software causes;

determining a recovery action as a function of the thresholding and the co-relating; and

performing the recovery action.

14. The method as recited in claim 13, wherein determining the recovery action is further a function of an elapsed time the microprocessor-based system has been running.

15. The method as recited in claim 13, wherein the microprocessor-based system is defined as a hierarchical set of objects to model software and hardware components of the microprocessor-based system.

16. The method as recited in claim 15, further comprising using relationships imposed by the hierarchical set of objects when determining the recovery action.

17. The method as recited in claim 13, wherein the recovery action is used to perform at least one of a restart of the microprocessor-based system, a cleanup of memory, a reset to a hardware module, a reset to a sub-assembly, and a releasing of resources that are marked unavailable.

18. A method of automatically detecting and recovering from a fault in a microprocessor-based system, comprising:

reporting the fault as an event

processing the event including thresholding the event and co-relating the event to a cause selected from a list of potential causes, the list of potential causes including potential causes internal to the microprocessor-based system and potential causes external to the microprocessor-based system; and

determining a recovery action as a function of the thresholding and the co-relating.

19. The method as recited in claim 18, wherein determining the recovery action is further a function of an elapsed time the microprocessor-based system has been running

20. The method as recited in claim 18, wherein the microprocessor-based system is defined as a hierarchical set of objects to model software and hardware components of the microprocessor-based system.

21. The method as recited in claim 20, further comprising using relationships imposed by the hierarchical set of objects when determining the recovery action.

22. The method as recited in claim 18, wherein the recovery action is used to perform at least one of a restart of the microprocessor-based system, a cleanup of memory, a reset to a hardware module, a reset to a sub-assembly, and a releasing of resources that are marked unavailable.

23. A method of automatically detecting and recovering from a fault in a microprocessor-based system, comprising:

creating a list of object-cause pairs relevant to a fault event;

incrementing a count in a leaky bucket associated with each object-cause pair; and

performing an action associated with a threshold if the count of the leaky bucket exceeds the threshold, the action being selected from a list of potential actions, the list of potential actions including hardware actions and software actions.

24. The method as recited in claim 23, further comprising determining the action as a function of an elapsed time the microprocessor-based system has been running

25. The method as recited in claim 23, wherein the microprocessor-based system is defined as a hierarchical set of objects to model software and hardware components of the microprocessor-based system and relationships imposed by the hierarchical set of objects are used to determine the action.

26. The method as recited in claim 23, wherein the action is used to perform at least one of a restart of the microprocessor-based system, a cleanup of memory, a reset to a hardware module, a reset to a sub-assembly, and a releasing of resources that are marked unavailable.

27. A method of automatically detecting and recovering from a fault in a microprocessor-based system, comprising:

creating a list of object-cause pairs relevant to a fault event;

incrementing a count in a leaky bucket associated with each object-cause pair; and

performing an action associated with a threshold if the count of the leaky bucket exceeds the threshold, the action being selected from a list of potential actions, the list of potential actions including actions internal to the microprocessor-based system and actions external to the microprocessor-based system.

28. The method as recited in claim 27, further comprising determining the action as a function of an elapsed time the microprocessor-based system has been running

29. The method as recited in claim 27, wherein the microprocessor-based system is defined as a hierarchical set of objects to model software and hardware components of the microprocessor-based system and relationships imposed by the hierarchical set of objects are used to determine the action.

30. The method as recited in claim 27, wherein the action is used to perform at least one of a restart of the microprocessor-based system, a cleanup of memory, a reset to a hardware module, a reset to a sub-assembly, and a releasing of resources that are marked unavailable.

31. A method for recovering from a fault in a microprocessor-based system, the method comprising:

reporting the fault as an event;

determining an elapsed time the microprocessor-based system has been running;

determining a recovery action as a function of the elapsed time the microprocessor-based system has been running; and

performing the recovery action.

32. The method as recited in claim 31, wherein the microprocessor-based system is defined as a hierarchical set of objects to model software and hardware components of the microprocessor-based system and relationships imposed by the hierarchical set of objects are used to determine the recovery action.

33. The method as recited in claim 31, wherein the recovery action is used to perform at least one of a restart and a cleanup of memory.