Fault recovery for real-time, multi-tasking computer system

• US 20060195751A1
• Filed: 02/16/2005
• Published: 08/31/2006
• Est. Priority Date: 02/16/2005
• Status: Active Grant

• Current Assignee: Honeywell International Inc.
• Original Assignee: Honeywell International Inc.
• Inventors: Hess, Richard, Rogers, Randy, Stange, Kent A., Kelley, Gerald B.
• Granted Patent: US 7,971,095 B2
• Time in Patent Office: Days
• US Class Current: 714/746
• CPC Class Codes: G06F 11/1438 Restarting or rejuvenating
• CPC Class Codes: G06F 11/1645 and the comparison itself uses redundant hardware
• CPC Class Codes: G06F 11/1658 Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit
• CPC Class Codes: G06F 11/1679 at clock signal level

Abstract

System and methods for providing a recoverable real time multi-tasking computer system are disclosed. In one embodiment, a system comprises a real time computing environment, wherein the real time computing environment is adapted to execute one or more applications and wherein each application is time and space partitioned. The system further comprises a fault detection system adapted to detect one or more faults affecting the real time computing environment and a fault recovery system, wherein upon the detection of a fault the fault recovery system is adapted to restore a backup set of state variables.

Claims

1. A recoverable real time multi-tasking computer system comprising:

a real time computing platform, wherein the real time computing platform is adapted to execute one or more applications, wherein each application is time and space partitioned;

a fault detection system adapted to detect one or more faults affecting the real time computing platform; and

a fault recovery system, wherein upon the detection of a fault by the fault detection system, the fault recovery system is adapted to restore a duplicate set of state variables.

2. The system of claim 1, wherein each application of the one or more applications is executed by the real time computing platform during one or more computational frames, wherein the fault detection system is further adapted to detect the one or more faults within the computational frame in which the fault occurred.

3. The system of claim 2, wherein the fault recovery system is further adapted to restore the duplicate set of state variables from a computational frame occurring before the computational frame in which the fault occurred.

4. The system of claim 3, wherein the fault recovery system is further adapted to restore a duplicate set of state variables from a computational frame occurring one computational frame before the computational frame in which the fault occurred.

5. The system of claim 1, the fault recovery system further comprising:

a first duplicate memory;

an even frame memory, wherein the fault recovery system is adapted to duplicate state variables computed by the real time computing platform during even computational frames into the even frame memory; and

an odd frame memory, wherein the fault recovery system is adapted to duplicate state variables computed by the real time computing platform during odd computational frames into the odd frame memory;

wherein, the even frame memory and odd frame memory toggle back and forth duplicating state variables into the first duplicate memory for computational frames in which no fault was detected by the fault detection system.

6. The system of claim 5, wherein the even frame memory and odd frame memory are further adapted to not duplicate into the first duplicate memory state variables for computational frames in which a fault was detected by the fault detection system.

7. The system of claim 5, wherein the first duplicate memory, the even frame memory and the odd frame memory are further adapted to duplicate state variables computed by the real time computing platform during initialization of the real time computing platform.

8. The system of claim 5, further comprising:

a second duplicate memory, wherein the fault recovery system stores duplicate sets of state variables for a plurality of computational frames.

9. The system of claim 5, wherein the first duplicate memory is protected from corruption due to environmental factors by one or more of shielding from a metal enclosure, signal buffers, isolated power supplies and hardened memory.

10. The system of claim 1, the fault recovery system further comprising:

a variable identity array, adapted to identify a predefined subset of state variables, wherein the fault recovery system duplicates only the subset of state variables.

11. A recoverable real time multi-tasking computer system, the system comprising:

one or more applications;

one or more processors, the one or more processors executing the one or more applications, wherein each application is time and space partitioned;

one or more scratchpad memories, wherein the one or more processors store state variables for the one or more applications in the one or more scratchpad memories;

one or more fault monitors, the one or more fault monitors adapted to detect one or more system faults occurring during the execution of a first application of the one or more applications; and

a fault recovery system adapted to duplicate state variables stored in the one or more scratchpad memories;

wherein upon the detection of a fault, the fault recovery system is further adapted to restore a duplicate set of state variables into the one or more scratchpad memories, wherein the one or more processors are adapted to resume processing of the first application using the duplicate set of state variables.

12. The system of claim 11, wherein upon the detection of the fault, the one or more fault monitors are further adapted to notify the fault recovery system.

13. The system of claim 11, wherein upon the detection of the fault, the one or more fault monitors are further adapted to notify a first application affected by the fault, wherein the first application is adapted to notify the fault recovery system.

14. The system of claim 11, wherein each application of the one or more applications is executed by the one or more processors during one or more computational frames, wherein the one or more fault monitors are further adapted to detect one or more system faults within the computational frame in which the fault occurred.

15. The system of claim 14, the fault recovery system further comprising:

a first duplicate memory;

an even frame memory, wherein the fault recovery system is adapted to duplicate state variables stored in the one or more scratchpad memories during even computational frames into the even frame memory; and

an odd frame memory, wherein the fault recovery system is adapted to duplicate state variables stored in the one or more scratchpad memories during odd computational frames into the odd frame memory;

wherein the even frame memory and odd frame memory toggle back and forth duplicating state variables into the first duplicate memory for computational frames in which no fault was detected by the one or more fault monitors.

16. The system of claim 15, further comprising:

a second duplicate memory, wherein the fault recovery system stores duplicate sets of state variables for a plurality of computational frames.

17. A method for fault recovery for applications executing on real time multi-tasking computer systems wherein each application is time and space partitioned, the method comprising:

duplicating state variables for one or more computational frames;

detecting a fault from an upset event within the computational frame in which the upset event occurred; and

recovering state variable data duplicated during a computational frame prior to the upset event.

18. The method of claim 17 further comprising:

halting the execution of an application affected by the upset event; and

resuming processing the application affected by the upset event after recovering state variable data.

19. The method of claim 17, wherein duplicating state variables for one or more computational frames further comprises:

duplicating state variables from an even computational frame into a first memory;

duplicating state variables from an odd computational frame into a second memory; and

alternately duplicating state variables from the first memory and the second memory into a third memory.

20. The method of claim 19, wherein recovering state variable data from the computational frame duplicated prior to the upset event further comprises:

duplicating state variables from the third memory into one or more scratchpad memories.

21. A computer-readable medium having program instructions for a method for fault recovery for applications executing on real time multi-tasking computer systems wherein each application is time and space partitioned, the method comprising:

duplicating state variables for one or more computational frames;

detecting a fault from an upset event within the computational frame in which the upset event occurred; and

recovering state variable data duplicated during a computational frame prior to the upset event.

22. The method of claim 21 further comprising:

halting the execution of an application affected by the upset event; and

resuming processing the application affected by the upset event after recovering state variable data.

23. The method of claim 21, wherein duplicating state variables for one or more computational frames further comprises:

duplicating state variables from an even computational frame into a first memory;

duplicating state variables from an odd computational frame into a second memory; and

alternately duplicating state variables from the first memory and the second memory into a third memory.

24. The method of claim 23, wherein recovering state variable data from the computational frame duplicated prior to the upset event further comprises:

duplicating state variables from the third memory into one or more scratchpad memories.

25. A rapid recovery mechanism for a self-checking lock-step computing lane including two or more processors, two or more scratchpad memories and two or more fault monitors, the self-checking lock-step computing lane adapted to execute two or more applications, wherein each application is time and space partitioned, wherein each application of the one or more applications is executed by the two or more processors during one or more computational frames, wherein the two or more fault monitors are further adapted to detect one or more system faults within the computational frame in which the fault occurred, the rapid recovery mechanism comprising:

a first duplicate memory adapted to store state variables duplicated from the one or more scratchpad memories; and

a recovery control logic module adapted to receive fault detection signals from the two or more fault monitors;

wherein upon the detection of a fault, the recovery control logic module is adapted to restore a duplicate set of state variables into the two or more scratchpad memories.

26. The rapid recovery mechanism of claim 25, wherein the recovery control logic module is further adapted to restore a duplicate set of state variables from a computational frame occurring before the computational frame in which the fault occurred.

27. The rapid recovery mechanism of claim 26, further comprising:

an even frame memory adapted to duplicate state variables stored in the one or more scratchpad memories during even computational frames into the even frame memory; and

an odd frame memory adapted to duplicate state variables stored in the one or more scratchpad memories during odd computational frames into the odd frame memory;

wherein the even frame memory and odd frame memory toggle back and forth duplicating state variables into the first duplicate memory for computational frames in which no fault was detected by the two or more fault monitors.

28. The rapid recovery mechanism of claim 27, wherein the even frame memory and odd frame memory are further adapted to discard state variables for computational frames in which a fault was detected by the one or more fault monitors.

29. A recoverable real time multi-tasking computer system comprising:

means for executing two or more time and space partitioned software applications;

means for detecting one or more faults affecting at least one of the two or more time and space partitioned software applications; and

means for restoring a duplicate set of state variables upon the detection of a fault affecting the at least one of the two or more time and space partitioned software applications.

30. The system of claim 29, wherein the means for detecting one or more faults further comprises:

means for detecting the one or more faults within a computational frame in which the one or more fault occurred.

31. The system of claim 30, wherein the means for restoring a duplicate set of state variables further comprises:

a first means for storing state variables;

a second means for storing state variables adapted to duplicate state variables computed during even computational frames; and

a third means for storing state variables adapted to duplicate state variables computed during odd computational frames;

wherein the second means for storing state variables and the third means for storing state variables toggle back and forth duplicating state variables into the first means for storing state variables for computational frames in which no fault was detected;

wherein the means for restoring a duplicate set of state variables is further adapted to restore the state variables from the first means for storing state variables.