Method and system for automatic fault detection and recovery in a data processing system

• US 5,502,812 A
• Filed: 04/06/1995
• Issued: 03/26/1996
• Est. Priority Date: 10/04/1991
• Status: Expired due to Fees

• Current Assignee: Aerospatiale Societe Nationale Industrielle
• Original Assignee: Aerospatiale Societe Nationale Industrielle
• Inventors: Leyre, Xavier, Senaux, Michel
• Primary Examiner(s): Beausoliel, Jr., Robert W.
• Assistant Examiner(s): Palys, Joseph E.
• Application Number:
  US08/418,264
• Time in Patent Office: 355 Days
• Field of Search: 395/182.08, 395/182.09, 395/182.11, 395/184.01, 395/185.08
• US Class Current: 714/10
• CPC Class Codes: G05D 1/0077 using redundant signals or controls
• CPC Class Codes: G06F 11/0739 in a data processing system embedded in automotive or aircraft systems
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/1666 where the redundant component is memory or memory area
• CPC Class Codes: G06F 11/2007 using redundant communication media
• CPC Class Codes: G06F 11/2025 using centralised failover control functionality
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/2038 with a single idle spare processing component
• CPC Class Codes: G06F 11/22 Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing
• CPC Class Codes: G06F 2201/88 Monitoring involving counting

Abstract

In the event of failure of a processor module (PMI) which is part of a data processing system the processor module (III.1) is turned off and then on; preferably, complete reconfiguration (III.2) of the module is commanded only if at least one other fault, the number of which can be chosen at will, is detected in a given time (T_max) from the first fault replacing it with another available cold redundant processor module.

Claims

1. A method of automatic fault detection and recovery in a data processing system, said data processing system comprising a central control unit, a plurality of system units, a bus, and a plurality of associated coupler units connecting said plurality of system units and said central control unit to said bus, wherein said central control unit comprises a watchdog circuit and a primary processor unit coupled to a reconfiguration module, and each of said plurality of coupler units further includes a corresponding, redundant backup coupler, said method comprising the steps of:

providing a second processor in parallel with said primary processor unit, said second processor not being in service while said primary processor unit is in service;

providing a backup reconfiguration module for said reconfiguration module, said backup reconfiguration module not being in service while said reconfiguration module is in service;

providing a backup bus for said bus;

providing at least one backup system unit for said plurality of system units;

coupling said reconfiguration module to said plurality of system units independently of said bus;

monitoring the operation of said central control unit and said plurality of system units;

upon detection of a fault, said watchdog circuit generating an alarm signal indicative of which component has malfunctioned within said data processing system;

classifying a fault type and severity based on said generated alarm signal; and

selecting one of a plurality of fault recovery options based on the type and severity of said detected fault.

2. The method of claim 1, wherein the step of classifying fault type and severity comprises the steps of:

transmitting a first alarm signal to a first clock circuit, said first alarm signal being indicative that said primary processor may have failed;

initiating said first clock circuit for a first predetermined period of time; and

selecting a fault recovery option of replacing said primary processor unit with said second processor to control said data processing system if a second alarm signal indicating said primary processor unit has failed is produced during said first predetermined period of time.

3. The method of claim 2, wherein said first predetermined period of time is between about 5 and about 60 seconds.

4. The method of claim 3, wherein said first predetermined period of time is approximately 10 seconds.

5. The method of claim 1, further comprising the step of providing a self-test unit in said reconfiguration module for determining whether said generated alarm signal is produced by alarm conditions or by test signals.

6. The method of claim 5, wherein said step of determining whether said generated alarm signal is produced by alarm conditions or by test signals comprises evaluating a voltage and pulse of said generated alarm signal.

7. The method of claim 2, further comprising the step of providing a self-test unit in said reconfiguration module for determining whether said first alarm signal and said second alarm signals are produced by alarm conditions or by test signals.

8. The method of claim 7, wherein said step of determining whether said first alarm signal and said second alarm signal are produced by alarm conditions or by test signals comprises evaluating a voltage and pulse of said first and second alarm signals.

9. The method of claim 2, further comprising the steps of:

monitoring the operation of said plurality of coupler units based upon signals received by said primary processor unit;

initiating a second clock circuit for a second predetermined period of time at least equal to a time needed to turn off and on said primary processor unit when a signal is received in said primary processor unit indicating that one of said plurality of coupler units is malfunctioning;

disabling said second clock circuit if said primary processor unit is turned off and on as a selected recovery option before the expiration of said second predetermined period of time; and

selecting a recovery option of replacing said malfunctioning coupler unit with said backup coupler after the expiration of said second predetermined period of time.

10. The method of claim 9, wherein said second predetermined period of time is between about 0.5 and about 50 seconds.

11. The method of claim 9, further comprising the step of utilizing a driver external to said primary processor unit to replace said malfunctioning coupler.

12. The method of claim 9, further comprising the steps of:

monitoring the operation of said plurality of system units based upon signals received by said primary processor unit to determine the existence of a malfunctioning system unit;

initiating a third clock circuit for a third predetermined period of time at least equal to an amount of time required to complete replacing of a malfunctioning coupler unit with one backup coupler unit of said at least one backup coupler unit; and

selecting a recovery option of replacing said malfunctioning system unit with one backup system unit of said at least one backup system unit when said system unit still appears malfunctioning after the expiration of said third predetermined period of time.

13. The method of claim 12, wherein said third predetermined period of time is between about 1 and 10 seconds.

14. The method of claim 13, wherein said second predetermined period of time is approximately 0.5 seconds and said third predetermined period of time is approximately 1 second.

15. The method of claim 12, wherein said data processing system is located in a spacecraft, said method further comprising the steps of:

monitoring an attitude of said spacecraft and after expiration of a fourth predetermined period of time, selecting a recovery option of replacing said primary processor unit with said second processor;

replacing said reconfiguration module with said backup reconfiguration module;

replacing each of said plurality of coupler units with said corresponding redundant backup couplers; and

replacing said bus with said backup bus.

16. The method of the claim 15, wherein said fourth predetermined period of time is equal to or greater than 1 minute.

17. The method of claim 16, wherein said fourth predetermined period of time is between about 1 minute and about 10 minutes.

18. The method of claim 1, wherein the step of classifying fault type and severity comprises the steps of:

monitoring the operation of said plurality of coupler units based upon signals received by said primary processor unit;

initiating a second clock circuit for a second predetermined period of time at least equal to a time needed to turn off and on said primary processor unit when a signal is received in said primary processor unit indicating that one of said plurality of coupler units is malfunctioning;

disabling said second clock circuit if said primary processor unit is turned off and on as a selected fault recovery option before the expiration of said second predetermined period of time; and

selecting a fault recovery option of replacing said malfunctioning coupler unit with said backup coupler after the expiration of said second predetermined period of time.

19. The method of claim 18, wherein said second predetermined period of time is between about 0.5 and about 50 seconds.

20. The method of claim 18, wherein said second predetermined period of time is between about 0.5 and about 5 seconds.

21. The method of claim 18, further comprising the step of utilizing a driver external to said primary processor unit to replace said malfunctioning coupler.

22. The method of claim 18, further comprising the steps of:

monitoring the operation of said plurality of system units based upon signals received by said primary processor unit to determine the existence of a malfunctioning system unit;

initiating a third clock circuit for a third predetermined period of time at least equal to an amount of time required to complete replacing of a malfunctioning coupler unit with one said at least one backup coupler unit; and

selecting a recovery option of replacing said malfunctioning unit with one said at least one backup system unit when said system unit still appears malfunctioning after the expiration of said third predetermined period of time.

23. The method of claim 22, wherein said third predetermined period of time is between about 1 and 10 seconds.

24. The method of claim 23, wherein said second predetermined period of time is approximately 0.5 seconds and said third predetermined period of time is approximately 1 second.

25. The method of claim 22, wherein said data processing system is located in a spacecraft, said method further comprising the steps of:

monitoring an attitude of said spacecraft and after expiration of a fourth predetermined period of time, selecting a recovery option of replacing said primary processor with said second processor;

replacing said reconfiguration module with said backup reconfiguration module;

replacing each of said plurality of coupler units with said corresponding redundant backup couplers; and

replacing said bus with said backup bus.

26. The method of claim 25, wherein said fourth predetermined period of time is equal to or greater than 1 minute.

27. The method of claim 26, wherein said fourth predetermined period of time is between about 1 minute and about 10 minutes.

28. A centralized data processing system located in a hostile environment comprising:

a plurality of processor control modules;

a plurality of buses;

a plurality of system units including at least one backup system unit;

a plurality of couplers including a plurality of backup couplers which connect said plurality of system units to said plurality of processor control modules via one of said plurality of buses; and

a plurality of reconfiguration modules for replacing said plurality of couplers with said plurality of backup couplers independent of said plurality of processor control modules, each reconfiguration module of said plurality of reconfiguration modules comprising;

a data transfer bus;

a self-test unit connected to said data transfer bus;

a processor module reconfiguration unit connected to said self-test unit;

a first test verification unit connected to said processor module reconfiguration unit;

a coupler reconfiguration unit connected to said data transfer bus;

a second test verification unit connected to said coupler reconfiguration unit;

an interface module connected to said data transfer bus; and

an output means connected to each reconfiguration module of said plurality of reconfiguration modules and each coupler of said plurality of couplers.

29. The centralized data processing system in accordance with claim 28, further comprising a backup memory provided in each reconfiguration module of said plurality of said reconfiguration modules.