Fault tolerant computer system

• US 5,903,717 A
• Filed: 04/02/1997
• Issued: 05/11/1999
• Est. Priority Date: 04/02/1997
• Status: Expired due to Term

• Current Assignee: General Dynamics Mission Systems Inc. (General Dynamics Corporation)
• Original Assignee: General Dynamics Information Systems, Inc. (General Dynamics Corporation)
• Inventors: Wardrop, Andrew J.
• Primary Examiner(s): Palys, Joseph E.
• Application Number:
  US08/832,479
• Time in Patent Office: 769 Days
• Field of Search: 395/182.09, 395/182.1, 395/182.08, 395/182.11, 395/182.13, 395/181, 395/182.02, 395/185.01, 395/185.03, 371/36
• US Class Current: 714/12
• CPC Class Codes: G05B 9/03 with multiple-channel loop, i.e. redundant control systems
• CPC Class Codes: G06F 11/1658 Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit
• CPC Class Codes: G06F 11/1679 at clock signal level
• CPC Class Codes: G06F 11/181 Eliminating the failing redundant component
• CPC Class Codes: G06F 11/184 where the redundant components implement processing functionality
• CPC Class Codes: G06F 11/187 Voting techniques
• CPC Class Codes: G06F 11/2038 with a single idle spare processing component
• CPC Class Codes: G06F 11/2043 where the redundant components share a common memory address space
• CPC Class Codes: G06F 11/22 Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing

Abstract

A fault tolerant computer system is disclosed which uses redundant voting at the hardware clock level to detect and to correct single event upsets (SEU) and other random failures. In one preferred embodiment, the computer (30) includes four or more commercial processing units (CPUs) (32) operating in strict "lock-step" and whose outputs (33, 37) to system memory (46) and system bus (12) are voted by a gate array (50) which may be implemented in a custom integrated circuit (34). A custom memory controller (18) interfaces to the system memory (46) and system bus (12). The data and address (35, 37) at each write to and read from memory (46) within the computer (30) are voted at each CPU clock cycle. A vote status and control circuit (38) "reads" the status of the vote and controls the state of the CPUs using hardware and software. The majority voted signals (35) are used by the agreeing CPUs 32 to continue processing operations without interruption. The system logic selects the best chance of recovering from a detected fault by resynchronizing all CPUs (32), powering down a faulty CPU or switching to a spare computer (30), resetting and re-booting the substituted CPUs (32).

Claims

1. A fault tolerant computer system (10) comprising:

a plurality of central processing units (CPUs) (22) operating synchronously, each operating step of each of said plurality of CPUs (22) being accomplished in parallel and substantially simultaneously with each of the other of said plurality of CPUs (22) each clock cycle of said CPUs (22);

each of said plurality of CPUs (22) having a plurality of CPU inputs (14) and a plurality of CPU outputs (15);

a voter (16) coupled to each said plurality of CPU inputs (14) and each said plurality of CPU outputs (15);

said voter (16) using redundant voting of said plurality of CPU outputs (15) to detect errors and failures in any one of said plurality of CPUs (22) whose plurality of CPU outputs (15) disagrees with said plurality of CPU outputs (15) of a majority of said plurality of CPUs (22);

each said plurality of CPU outputs (15) being compared one with another by said voter (16) each said clock cycle;

a system bus (12);

a first computer (11a);

said first computer (11a) including said plurality of CPUs (22) and said voter (16) and coupled to said system bus (12);

said first computer (11a) further including

a system memory (20);

a memory controller (18);

said memory controller (18) coupled to said voter (16), said memory (20) and said system bus (12);

said system memory (20) providing memory output signals (19a) to said memory controller (18);

said memory output signals (19a) being distributed equally as said plurality of CPU inputs (14) to each of said CPUs (22) through said memory controller (18) and said voter (16);

agreement in said voter (16) of a majority of said plurality of CPU outputs (15) resulting in a voted output signal (17a);

said voted output signal (17a) having a value of said majority of said plurality of CPU outputs (15);

disagreement of any one of said plurality of CPU outputs (15) with a majority of said plurality of CPU outputs (15) being detected by said voter (16), an error signal produced by said disagreement causing resynchronizing of said plurality of CPUs (22) in said first computer (11a);

a gate array(50);

said gate array(50) including

a first AND gate array (53);

each gate (58) of said first AND gate array (53) having as a first input (52a-d), CPU signals (52) derived from said plurality of CPU outputs (15), and having as a second input (54a-d), enable signals (54) derived from a vote status and control signal (39a);

each said gate (58) having an output (60);

a second AND gate array (57);

each gate (59) of said second AND gate array (53) having as a first input, an output (60) from one gate (58) of said first AND gate array (53) and having as a second input, an output (60) from another gate (58) of said first AND gate array (53);

each said gate (59) having an output (61);

a first OR gate (63) having as input said output (61) of each said gate (59) of said second AND gate array (57), and having an output (66);

an array of exclusive OR gates (65) coupled to an output (66) of said first OR gate (63) and coupled respectively to each of said CPU signals (52);

each gate (64) of said array of exclusive OR gates (65) having an output (68);

each of said CPU signals (52) input to said first AND gate array (53) being output by application of each of said enable signals (54) to a corresponding gate of said first AND gate array (53);

said output (60) of said first AND gate array (53) being compared, one to another in said second AND gate array (57);

each of said outputs (61) from said second AND gate array (57) having a value of the majority of said CPU signals (52) which are in agreement;

said signals (52) which emerge as outputs (61) from said second AND gate array (57) being output (66) from said first OR gate (63);

said first OR gate output (66) having a value of said CPU signals (52) which agree;

said output (66) from said first OR gate (63) being applied as a first input to each gate (64) of said exclusive OR gate array (65) and said CPU signals (52) being applied, respectively as a second input to each gate (64) of said exclusive OR gate array (65); and

an error signal (68) being output from each gate (64) of said exclusive OR gate array (65) where said applied CPU signals (52) and said applied output (66) from said first OR gate (63) do not agree.

2. The fault tolerant computer system (10) as claimed in claim 1 in which said gate array (50) further comprises:

a bypass AND gate array (55);

each gate (51) of said bypass AND gate array (55) having as a first input said CPU signals (52) and having as a second input a bypass signal (56);

each gate (51) of said bypass AND gate array (55) having an output (65) when enabled by said bypass signal (56);

a bypass OR gate (62) having as input said output (69) of said bypass AND gate array (55), and having an output (61a) which is fed to said first OR gate (63);

each of said CPU signals (52) input to a gate (51) of said bypass AND gate array (55) being output to said bypass OR gate (62) by application of each of said bypass signals (56) to a corresponding gate (51) of said bypass AND gate array (55); and

said bypass signals (56) derived from a vote status and control signal (39a) upon a remote command to test or bypass said voter action.

3. The fault tolerant computer system (10) as claimed in claim 1 in which said CPU signals (52) are data signals.

4. The fault tolerant computer system (10) as claimed in claim 1 in which said CPU signals (52) are address signals.

5. The fault tolerant computer system (10) as claimed in claim 3 in which each one of said plurality of CPUs (22) is in one of the following stable states:

enabled and voting;

commanded disabled;

non-agreement and disabled;

bypassed; and

powered down.

6. A fault tolerant computer system (10) comprising:

a plurality of reduced instruction set (RIS) central processing units (CPUs) (32) operating in synchronism, each operating step of each of said plurality of central processing units (32) being accomplished in parallel and substantially simultaneously with each of the other of said plurality of RIS CPUs (32) each CPU clock cycle;

each of said plurality of RIS CPUs (32) having a plurality of CPU outputs (33b) and a plurality of CPU inputs (33a);

a system bus (12);

a CPU bus (35a);

a voter (34);

said voter having a voted output (35) carried on said CPU bus (35a);

said voter (34) being coupled to each one of said plurality of RIS CPUs (32);

each said CPU output (33b) being compared one with another by said voter (34) each said clock cycle;

a memory (46);

a memory controller (36) having a vote status and control circuit (38),

a memory interface (40) and a bus interface (42);

said memory controller (36) coupled to said voter (34) through said CPU bus (35a), coupled to said system bus (22) through said bus interface (42) and coupled to said memory (46) through said memory interface (40);

said memory (46) providing memory output (37a) to said memory controller (36);

said memory output (37a) being distributed equally as said plurality of CPU inputs (33a) to each of said plurality of RIS CPUs (32) through said memory controller (36) and said voter (34);

said voted output (35) being provided by said memory controller (36) as input to said system memory (46), said vote status and control circuit (38) and to said system bus (12);

agreement of a majority of said plurality of CPU outputs (33b) supplied to said voter (34) resulting in said voted output (35) having a value of said majority of said plurality of CPU outputs (33b);

any one of said plurality of CPU outputs (33b) which does not agree with a majority of said plurality of CPU outputs (33b) being detected by said voter (34) and an error signal produced thereby being sent to said vote status and control circuit (38);

said vote status and control circuit (38) reacting to said error signal indicating disagreement of said plurality of CPU outputs (33b) and causing re-synchronizing of said plurality of RIS CPUs (32).

7. The fault tolerant computer system (10) as claimed in claim 6 in which said plurality of RIS CPUs (32) number at least four.

8. The fault tolerant computer system (10) as claimed in claim 6 in which said plurality of RIS CPUs (32) are commercial, non-radiation-hardened chips.

9. The fault tolerant computer system (10) as claimed in claim 6, further comprising:

a first computer (30);

a second computer (30);

said first computer (30) including said plurality of CPUs (32), said voter (34) said memory (46), said memory controller (36) and said CPU bus (35a);

said second computer (30) including said plurality of CPUs (32), said voter (34) said memory (46), said memory controller (36) and said CPU bus (35a);

said first computer (30) being coupled to said system bus (12);

said second computer (11b) being coupled to said system bus (12) and maintained as an error-free replacement for said first computer (11a);

said error signal causing said vote status and control circuit (36) to react and substitute said second computer (30) for said first computer (30) when less than three of said plurality of RIS CPUs (32) are operable and less than two of said plurality of said RIS CPUs (32) agree.

10. The fault tolerant computer system (10) as claimed in claim 9 in which resynchronization of said RIS CPUs (32) and substitution of said computer (30) are controlled by software run by said RIS CPUs (32).

11. The fault tolerant computer system (10) as claimed in claim 9 in which each one of said plurality of RIS CPUs (32) is in one of the following stable states:

enabled and voting;

commanded disabled;

disagreeing and disabled;

bypassed; and

powered down.

12. A method of fault management for computer systems comprising the steps of:

providing a first computer (30) having a plurality of reduced instruction set (RIS) central processing units (CPUs) (32);

operating said plurality of RIS CPUs (32) in synchronism;

accomplishing each operating step of each of said plurality of RIS CPUs (32) in parallel and substantially simultaneously with each of the other of said plurality of RIS CPUs (32);

providing a voter (34) within said computer (30);

coupling said voter (34) to a plurality of CPU outputs (33b) of each one of said plurality of RIS CPUs (32);

coupling each one of said plurality of RIS CPUs (32) to a plurality of outputs (33a) of said voter (34);

providing a memory (46);

providing a system bus (12);

providing a memory controller (36) having a vote status and control circuit (38), a memory interface (40) and a bus interface (42);

coupling said memory controller (36) to said voter (34), to said system bus (12) through said bus interface (42), and to said memory (46) through said memory interface (40);

providing memory output signals (37a) from said memory (46) to said memory controller (36);

distributing said memory output signals (37a) to each of said RIS CPUs (32) through said memory controller (36), equally through said plurality of outputs (33a) of said voter (34);

providing said plurality of CPU outputs (33b) from said plurality of RIS CPUs (32) as inputs to said voter (34) for comparison;

comparing said plurality of CPU outputs (33b) in said voter (34);

agreement of a majority of said plurality of CPU outputs (33b) supplied to said voter (34) resulting in a voted output signal (35);

said voted output signal (35) having a value of said majority of said plurality of CPU outputs (33b);

providing said voted output signal (35) from said voter (34) as input signals (37b) to said system memory (46), as input to said vote status and control circuit (38) and as input (41) to said system bus (12);

detecting disagreement of any one of said plurality of CPU outputs (33b) with a majority of said plurality of CPU outputs (33b) with said voter (34) and producing thereby error signals (39b);

applying said error signals (39b) to said vote status and control circuit (38); and

re-synchronizing said plurality of RIS CPUs (32) by action of said vote status and control circuit (36).

13. The method as claimed in claim 12 further including the step of substituting an error-free second computer (30) for a faulty said first computer (30) by action of said vote status and control circuit (36) when less than three of said plurality of RIS CPUs 32 are operable and less than two of said plurality of CPUs 32 agree.

14. The method as claimed in claim 12 further including the step of substituting an error-free computer (30) when less than two said RIS CPUs (32) are operable.