Hierarchy of fault isolation timers

• US 20030196141A1
• Filed: 04/21/2003
• Published: 10/16/2003
• Est. Priority Date: 04/20/2000
• Status: Active Grant

• Current Assignee: Valtrus Innovations Limited (f/k/a Dolya Holdco 9 Limited) (Key Patent Innovations Limited)
• Original Assignee: Mark D. Shaw
• Inventors: Shaw, Mark
• Granted Patent: US 6,857,086 B2
• Time in Patent Office: Days
• US Class Current: 714/27
• CPC Class Codes: G06F 11/0724 in a multiprocessor or a multi-core unit multiprocessors per se G06F15/80
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40

Abstract

In the present invention, a coordinated hierarchy of timing mechanisms preferably cooperate to report errors at different operational levels of a complex of computing devices. Preferably, each timer is able to identify a failure condition at its own level of operation and transmit a time-out condition to a higher level device which may also be a timer. Upon generation of a time-out condition, a system component experiencing a fault condition preferably continues to operate in a degraded mode, informs devices attempting to communicate with the faulty component of a status of the fault condition, and preferably proceeds to identify and correct a failure which caused the time out condition. The timers may be implemented in hardware or software.

Claims

1. A method for isolating a fault condition in a complex of computing devices, the method comprising the steps of:

establishing a hierarchy of transactions executed within said complex;

monitoring at least one transaction of said transactions with a transaction-specific timer, wherein each said transaction-specific timer has an expiration time, thereby establishing a hierarchy of timers and a hierarchy of expiration times corresponding to said hierarchy of transactions;

setting said expiration times of said transaction-specific timers to progressively higher values at progressively higher levels of said hierarchy of timers; and

upon expiration of a transaction-specific timer, transmitting a time-out condition to a next higher level timer in said hierarchy of timers, thereby indicating a fault condition within a component of said complex including said expired timer, wherein said component is a faulty component.

2. The method of claim 1 comprising the further step of:

halting said next higher level timer prior to an expiration of said next higher level timer.

3. The method of claim 1 comprising the further step of:

operating said faulty component in a degraded mode until said fault condition is corrected.

4. The method of claim 1 comprising the further step of:

indicating a fault condition at said faulty component to at least one other component in said complex.

5. The method of claim 4 comprising the further step of:

causing said at least one other component in said complex to avoid communication with said faulty component until said fault condition is corrected, thereby operating to isolate said fault condition to said faulty component and enabling a remainder of said complex to continue operating.

6. The method of claim 1 further comprising the steps of:

logging error information describing said fault condition; and

executing recovery code within said faulty component upon identification of a fault condition employing said logged error information.

7. The method of claim 1 wherein said faulty component is a central processing unit.

8. The method of claim 1 wherein said faulty component is a system chip.

9. The method of claim 1 wherein said faulty component is a subset of a system chip.

10. The method of claim 1 further comprising the step of:

monitoring a highest level transaction of said hierarchy of transactions with a central processing unit.

11. A system for preserving an operation of a distributed computing complex during a fault condition, the system comprising:

a plurality of devices operating within a first functional component of said complex;

a sequence of transactions executed in said first functional component linking a plurality of said plurality of devices, wherein the sequence has a first transaction and a last transaction;

a plurality of timers in communication with at least one of said plurality of devices for timing at least one of said sequence of transactions;

means for setting expiration values of said plurality of timers to progressively lower levels for timers of said plurality of timers timing transactions in increasing proximity to said last transaction, thereby establishing a hierarchy of timers and a hierarchy of expiration values;

means for transmitting a time-out condition to a next higher level timer of said plurality of timers upon expiration of a timer of said plurality of timers;

means for logging an error at said higher level timer, thereby identifying a fault condition within said first functional component of said complex;

means for informing other functional components of said complex of said fault condition in said first functional component of said complex, thereby avoiding propagation of said fault condition to said other functional components of said complex and preserving the operation of said complex during said fault condition.

12. The system of claim 11 further comprising:

means for operating said first functional component in a degraded mode until said fault condition is corrected.

13. The system of claim 11 further comprising:

means for executing recovery code in said first functional component in response to a fault condition to identify and correct a cause of said fault condition.

14. The system of claim 11 further comprising:

at least one timer of said plurality of timers implemented in software.

15. The system of claim 11, further comprising:

at least one timer of said plurality of timers implemented in hardware.

16. A computer program product having a computer readable medium having computer program logic recorded thereon for isolating a fault condition in a complex of computing devices, the computer program product comprising:

code for establishing a hierarchy of transactions executed within said complex;

code for monitoring at least one transaction of said transactions with a transaction-specific timer, wherein each said transaction-specific timer has an expiration time, thereby establishing a hierarchy of timers and a hierarchy of expiration times corresponding to said hierarchy of transactions;

code for setting said expiration times of said transaction-specific timers to progressively higher values at progressively higher levels of said hierarchy of timers; and

code for transmitting a time-out condition to a next higher level timer in said hierarchy of timers upon expiration of a transaction-specific timer, thereby indicating a fault condition within a component of said complex including said expired timer, wherein said component is a faulty component.

17. The computer program product of claim 16 further comprising:

code for halting said next higher level timer prior to an expiration of said next higher level timer.

18. The computer program product of claim 16 further comprising:

code for operating said faulty component in a degraded mode until said fault condition is corrected.

19. The computer program product of claim 16 further comprising:

code for logging error information describing said fault condition; and

code for executing recovery code within said faulty component upon identification of a fault condition employing said logged error information.

20. The computer program product of claim 19 further comprising:

code for causing said at least one other component in said complex to avoid communication with said faulty component until said fault condition is corrected, thereby operating to isolate said fault condition to said faulty component and enabling a remainder of said complex to continue operating.