Fault recovery for real-time, multi-tasking computer system

• US 7,971,095 B2
• Filed: 02/16/2005
• Issued: 06/28/2011
• Est. Priority Date: 02/16/2005
• Status: Active Grant

• Current Assignee: Honeywell International Inc.
• Original Assignee: Honeywell International Inc.
• Inventors: Hess, Richard, Rogers, Randy, Kelly, Gerald B., Stange, Kent A.
• Primary Examiner(s): Baderman; Scott T
• Assistant Examiner(s): PATEL, JIGAR P
• Application Number:
  US11/058,764
• Publication Number:
  US 20060195751A1
• Time in Patent Office: 2,323 Days
• Field of Search: 714 1- 52
• US Class Current: 714/15
• CPC Class Codes: G06F 11/1438 Restarting or rejuvenating
• CPC Class Codes: G06F 11/1645 and the comparison itself uses redundant hardware
• CPC Class Codes: G06F 11/1658 Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit
• CPC Class Codes: G06F 11/1679 at clock signal level

Abstract

System and methods for providing a recoverable real time multi-tasking computer system are disclosed. In one embodiment, a system comprises a real time computing environment, wherein the real time computing environment is adapted to execute one or more applications and wherein each application is time and space partitioned. The system further comprises a fault detection system adapted to detect one or more faults affecting the real time computing environment and a fault recovery system, wherein upon the detection of a fault the fault recovery system is adapted to restore a backup set of state variables.

Claims

1. A recoverable real time multi-tasking computer system comprising:

a real time avionics computing platform adapted to execute two or more avionics applications simultaneously, wherein each avionics application is time and space partitioned;

a fault detection system adapted to detect one or more faults affecting the real time avionics computing platform; and

a fault recovery system, wherein upon the detection of a fault by the fault detection system, the fault recovery system is adapted to restore a duplicate set of state variables, wherein the fault recovery system is further adapted to;

store, duplicate, and recover only selected state variables from one or more frame times; and

recover state variables pertaining to any one or more of the avionics applications simultaneously;

wherein the fault recovery system operates without any involvement from the avionics applications, and

wherein when a recovery of the one or more avionics applications occurs, the other avionics applications continue to operate without disturbance.

2. The system of claim 1, wherein each application of the two or more avionics applications is executed by the real time avionics computing platform during one or more computational frames, wherein the fault detection system is further adapted to detect the one or more faults.

3. The system of claim 2, wherein the fault recovery system is further adapted to restore the duplicate set of state variables from a computational frame occurring more than one frame before the computational frame in which the fault occurred.

4. The system of claim 3, wherein the fault recovery system is further adapted to restore a duplicate set of state variables from a computational frame occurring one computational frame before the computational frame in which the fault occurred.

5. The system of claim 1, wherein the fault recovery system further comprises:

a first duplicate memory;

an even frame memory, wherein the fault recovery system is adapted to duplicate state variables computed by the real time avionics computing platform during even computational frames into the even frame memory; and

an odd frame memory, wherein the fault recovery system is adapted to duplicate state variables computed by the real time avionics computing platform during odd computational frames into the odd frame memory;

wherein, the even frame memory and odd frame memory toggle back and forth duplicating state variables into the first duplicate memory for computational frames in which no fault was detected by the fault detection system.

6. The system of claim 5, wherein the even frame memory and odd frame memory are further adapted to not duplicate into the first duplicate memory state variables for computational frames in which a fault was detected by the fault detection system.

7. The system of claim 5, wherein the first duplicate memory, the even frame memory and the odd frame memory are further adapted to duplicate state variables computed by the real time avionics computing platform during initialization of the real time avionics computing platform.

8. The system of claim 5, further comprising:

a second duplicate memory, wherein the fault recovery system stores duplicate sets of state variables for a plurality of computational frames.

9. The system of claim 5, wherein the first duplicate memory is protected from corruption due to environmental factors by one or more of shielding from a metal enclosure, signal buffers, isolated power supplies and hardened memory.

10. The system of claim 1, the fault recovery system further comprising:

a variable identity array, adapted to identify a predefined subset of state variables, wherein the fault recovery system duplicates only the subset of state variables.

31. The system of claim 5, wherein the first duplicate memory, the even frame memory, and the odd frame memory are adapted to duplicate state variables computed by a real time operating system of the real time avionics computing platform.

11. A recoverable real time multi-tasking computer system comprising:

two or more avionics applications;

an avionics computing platform comprising one or more processors, the one or more processors executing the two or more avionics applications simultaneously, wherein each application is time and space partitioned;

one or more scratchpad memories, wherein the one or more processors store state variables for the two or more avionics applications in the one or more scratchpad memories;

one or more fault monitors, the one or more fault monitors adapted to detect one or more system faults occurring during the execution of a first application of the two or more avionics applications; and

a fault recovery system adapted to duplicate state variables stored in the one or more scratchpad memories, wherein the fault recovery system is further adapted to;

store, duplicate, and recover only selected state variables from one or more frame times; and

recover state variables pertaining to any one or more of the avionics applications simultaneously;

wherein the fault recovery system operates without any involvement from the avionics applications,

wherein upon the detection of a fault, the fault recovery system is further adapted to restore a duplicate set of state variables into the one or more scratchpad memories,

wherein the one or more processors are adapted to resume processing of the first application using the duplicate set of state variables, and

wherein when a recovery of the one or more avionics applications occurs, the other avionics applications continue to operate without disturbance.

12. The system of claim 11, wherein upon the detection of the fault, the one or more fault monitors are further adapted to notify the fault recovery system.

13. The system of claim 11, wherein upon the detection of the fault, the one or more fault monitors are further adapted to notify a first application affected by the fault, wherein the first application is adapted to notify the fault recovery system.

14. The system of claim 11, wherein each application of the two or more avionics applications is executed by the one or more processors during one or more computational frames, wherein the one or more fault monitors are further adapted to detect one or more system faults within the computational frame in which the fault occurred.

15. The system of claim 14, the fault recovery system further comprising:

a first duplicate memory;

an even frame memory, wherein the fault recovery system is adapted to duplicate state variables stored in the one or more scratchpad memories during even computational frames into the even frame memory; and

an odd frame memory, wherein the fault recovery system is adapted to duplicate state variables stored in the one or more scratchpad memories during odd computational frames into the odd frame memory;

wherein the even frame memory and odd frame memory toggle back and forth duplicating state variables into the first duplicate memory for computational frames in which no fault was detected by the one or more fault monitors.

16. The system of claim 15, further comprising:

a second duplicate memory, wherein the fault recovery system stores duplicate sets of state variables for a plurality of computational frames.

17. A method for fault recovery, the method comprising:

executing a plurality of avionics applications simultaneously on a real time multi-tasking avionics computer system wherein each avionics application is time and space partitioned;

duplicating state variables for one or more computational frames;

detecting a fault from an upset event within the computational frame of one of the applications in which the upset event occurred;

recovering state variable data duplicated during a computational frame prior to the upset event; and

restoring the duplicated state variable data to a computational frame of the one of the applications that occurs immediately after the computational frame in which the upset event occurred, wherein the duplicated state variable data is restored without any involvement from the avionics applications, and wherein during recovery of the one of the applications, the other applications continue to operate without disturbance.

18. The method of claim 17, further comprising:

halting the execution of an application affected by the upset event; and

resuming processing the application affected by the upset event after recovering state variable data.

19. The method of claim 17, wherein duplicating state variables for one or more computational frames further comprises:

duplicating state variables from an even computational frame into a first memory;

duplicating state variables from an odd computational frame into a second memory; and

alternately duplicating state variables from the first memory and the second memory into a third memory.

20. The method of claim 19, wherein recovering state variable data from the computational frame duplicated prior to the upset event further comprises:

duplicating state variables from the third memory into one or more scratchpad memories.

21. A computer-readable medium having program instructions for a method for fault recovery, the method comprising:

executing a plurality of avionics applications simultaneously on a real time multi-tasking avionics computer system wherein each avionics application is time and space partitioned;

duplicating state variables for one or more computational frames;

detecting a fault from an upset event within the computational frame of one of the applications in which the upset event occurred;

recovering state variable data duplicated during a computational frame prior to the upset event; and

restoring the duplicated state variable data to a computational frame of the one of the applications that occurs immediately after the computational frame in which the upset event occurred, wherein the duplicated state variable data is restored without any involvement from the avionics applications, and wherein during recovery of the one of the applications, the other applications continue to operate without disturbance.

22. The computer-readable medium of claim 21, the method further comprising:

halting the execution of an application affected by the upset event; and

resuming processing the application affected by the upset event after recovering state variable data.

23. The computer-readable medium of claim 21, wherein duplicating state variables for one or more computational frames further comprises:

duplicating state variables from an even computational frame into a first memory;

duplicating state variables from an odd computational frame into a second memory; and

alternately duplicating state variables from the first memory and the second memory into a third memory.

24. The computer-readable medium of claim 23, wherein recovering state variable data from the computational frame duplicated prior to the upset event further comprises:

duplicating state variables from the third memory into one or more scratchpad memories.

25. A system comprising:

a self-checking lock-step avionics lane including two or more processors;

two or more scratchpad memories and two or more fault monitors, the self-checking lock-step avionics lane adapted to execute two or more avionics applications simultaneously, wherein each application is time and space partitioned, wherein each application of the two or more avionics applications is executed by the two or more processors during one or more computational frames, wherein the two or more fault monitors are further adapted to detect one or more system faults within the computational frame in which the fault occurred;

a rapid recovery mechanism comprising;

a first duplicate memory adapted to store state variables duplicated from the two or more scratchpad memories; and

a recovery control logic module adapted to receive fault detection signals from the two or more fault monitors;

wherein the rapid recovery mechanism is further adapted to;

store, duplicate, and recover only selected state variables from one or more frame times; and

recover state variables pertaining to any one or more of the avionics applications simultaneously;

wherein the rapid recovery mechanism is further adapted to;

store, duplicate, and recover only selected state variables from one or more frame times; and

recover state variables pertaining to any one or more of the avionics applications simultaneously;

wherein the rapid recovery mechanism operates without any involvement from the avionics applications,

wherein upon the detection of a fault, the recovery control logic module is adapted to restore a duplicate set of state variables into the two or more scratchpad memories, and

wherein when a recovery of the one or more avionics applications occurs, the other avionics applications continue to operate without disturbance.

26. The system of claim 25, wherein the recovery control logic module is further adapted to restore a duplicate set of state variables from a computational frame occurring more than one frame before the computational frame in which the fault occurred.

27. The system of claim 26, the rapid recovery mechanism further comprising:

an even frame memory adapted to duplicate state variables stored in the two or more scratchpad memories during even computational frames into the even frame memory; and

an odd frame memory adapted to duplicate state variables stored in the two or more scratchpad memories during odd computational frames into the odd frame memory;

wherein the even frame memory and odd frame memory toggle back and forth duplicating state variables into the first duplicate memory for computational frames in which no fault was detected by the two or more fault monitors.

28. The system of claim 27, wherein the even frame memory and odd frame memory are further adapted to discard state variables for computational frames in which a fault was detected by the one or more fault monitors.

29. A recoverable real time multi-tasking computer system comprising:

means for executing two or more time and space partitioned avionics applications simultaneously;

means for detecting one or more faults affecting at least one of the two or more time and space partitioned avionics applications; and

means for restoring a duplicate set of selected state variables upon the detection of a fault affecting the at least one of the two or more time and space partitioned avionics applications;

wherein the means for restoring operates without any involvement from the avionics applications, and wherein when a recovery of the one or more avionics applications occurs, the other avionics applications continue to operate without disturbance.

30. The system of claim 29, wherein the means for restoring a duplicate set of state variables further comprises:

a first means for storing state variables;

a second means for storing state variables adapted to duplicate state variables computed during even computational frames; and

a third means for storing state variables adapted to duplicate state variables computed during odd computational frames;

wherein the second means for storing state variables and the third means for storing state variables toggle back and forth duplicating state variables into the first means for storing state variables for computational frames in which no fault was detected;

wherein the means for restoring a duplicate set of state variables is further adapted to restore the state variables from the first means for storing state variables.