Fault-tolerant computer system with online recovery and reintegration of redundant components

• US 6,073,251 A
• Filed: 06/09/1997
• Issued: 06/06/2000
• Est. Priority Date: 12/22/1989
• Status: Expired due to Fees

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Compaq Computer Corporation (HP Inc.)
• Inventors: Westbrook, deceased, Donald C., DeBacker, Kenneth C., Jewett, Douglas E., Banton, Randall G., Cutts, Richard W. Jr., Fey, Krayn W. Jr., Bereiter, Tom, Mehta, Nikhil A., Vetter, Bryan, Posdro, John
• Primary Examiner(s): Baker, Stephen M.
• Application Number:
  US08/871,077
• Time in Patent Office: 1,093 Days
• Field of Search: 395/182.04, 395/182.05, 395/183.13, 214/6, 214/7, 214/8, 214/11
• US Class Current: 714/6.32
• CPC Class Codes: G06F 1/12 Synchronisation of different clock signals provided by a plurality of clock generators
• CPC Class Codes: G06F 1/30 Means for acting in the event of power-supply failure or interruption, e.g. power-supply fluctuations for resetting only G06F1/24
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0724 in a multiprocessor or a multi-core unit multiprocessors per se G06F15/80
• CPC Class Codes: G06F 11/073 in a memory management context, e.g. virtual memory or cache management memory management G06F12/00; testing of static memory units G11C29/00
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1441 Resetting or repowering
• CPC Class Codes: G06F 11/1658 Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit
• CPC Class Codes: G06F 11/1666 where the redundant component is memory or memory area
• CPC Class Codes: G06F 11/1683 at instruction level
• CPC Class Codes: G06F 11/1687 at event level, e.g. by interrupt or result of polling
• CPC Class Codes: G06F 11/1691 using a quantum
• CPC Class Codes: G06F 11/181 Eliminating the failing redundant component
• CPC Class Codes: G06F 11/185 and the voting is itself performed redundantly
• CPC Class Codes: G06F 11/20 using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
• CPC Class Codes: G06F 11/2005 using redundant communication controllers
• CPC Class Codes: G06F 11/2007 using redundant communication media
• CPC Class Codes: G06F 11/2015 Redundant power supplies power supply failure G06F1/30
• CPC Class Codes: G06F 11/22 Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing
• CPC Class Codes: G06F 11/2736 using a dedicated service processor for test

Abstract

A computer system in a fault-tolerant configuration employs multiple identical CPUs executing the same instruction stream, with multiple, identical memory modules in the address space of the CPUs storing duplicates of the same data. The system detects faults in the CPUs and memory modules, and places a faulty unit offline while continuing to operate using the good units. The faulty unit can be replaced and reintegrated into the system without shutdown. The multiple CPUs are loosely synchronized, as by detecting events such as memory references and stalling any CPU ahead of others until all execute the function simultaneously; interrupts can be synchronized by ensuring that all CPUs implement the interrupt at the same point in their instruction stream. Memory references via the separate CPU-to-memory busses are voted at the three separate ports of each of the memory modules. I/O functions are implemented using two identical I/O busses, each of which is separately coupled to only one of the memory modules. A number of I/O processors are coupled to both I/O busses. I/O devices are accessed through a pair of identical (redundant) processors, but only one is designated to actively control a given device; in case of failure of one I/O processor, however, an I/O device can be accessed by the other one without system shutdown.

Claims

1. In a computer system having a plurality of Central Processor Units (CPUs) and a plurality of components coupled to the CPUs, a method comprising:

detecting an interrupt indicating a fault in a system component always designated as a primary system component, the primary system component having at least one matching redundant component always designated as a backup component of the primary system component, where a redundant component receives, during normal operation, the same input data as the primary system component, but, during normal operation, does not output the data output by the primary system component;

isolating the fault in the primary system component;

taking the faulty primary system component off-line while maintaining system operations and, without affecting systems operation, using the redundant system component matching the faulty primary system component, where the redundant system component receives and outputs data that the faulty primary system component would have received and output, respectively;

upon repair or replacement of the faulty primary system component, reinitializing the repaired or replacement primary system component;

initiating a test procedure in the repaired or replacement primary system component;

reintegrating the repaired or replacement primary system component if the primary system component passes the test procedure; and

placing the repaired or replacement primary system component online if the reintegration step is successfully completed.

2. The method of claim 1, wherein said faulty primary system component is a primary memory module and wherein said step of reintegrating comprises mapping the data stored in an operational redundant memory module to the repaired or replacement primary memory module.

3. The method of claim 2, wherein said mapping step comprises reading data from a memory location In said operational memory module and writing the data to both the operational and repaired or replacement primary memory modules.

4. The method of claim 3, wherein said mapping step further comprises repeating said reading and writing steps until all of the memory locations in said operational memory module have been rewritten into both memory modules.

5. A fault-tolerant computing system, comprising:

a plurality of Central Processor Units (CPUs);

a first system component always designated as a primary component and coupled to the plurality of CPUs for performing a function in the computing system and for inputting and outputting data;

a second system component always designated as a backup component of the primary component and coupled to the plurality of CPUs, that receives, during normal operation, the same input data that the first system component receives, but that does not, during normal operation, output data when the first system component outputs data, the second system component therefore functioning as a redundant component;

means for detecting an interrupt indicating a fault in the first system component;

means for isolating the fault in the first system component; and

means for taking the first system component off-line while maintaining system operations and without affecting systems operation and, while continuing to use the primary component, using the second system component to perform the function of the first system component, where the second system component receives and outputs data that the first system component would have received and output, respectively, in response to data requests.

6. The computing system of claim 5, further comprising:

means for, upon repair or replacement of the first system component, reinitializing the first system component;

means for initiating a test procedure in the repaired or replacement first system component;

means for reintegrating the repaired or replacement first system component if the repaired or replacement first system component passes the test procedure; and

means for placing the repaired or replacement system component online if the reintegration step is successfully completed.

7. The computing system of claim 5, wherein the first and second system components are memory modules.

8. A fault-tolerant computing system, comprising:

a plurality of Central Processor Units (CPUs);

a first memory module always designated as a primary memory module and coupled to the plurality of CPUs for performing a function in the computing system and for inputting and outputting data;

a second memory module always designated as a backup memory module of the primary memory module and coupled to the plurality of CPUs, that receives the same input data that the first memory module receives, but that does not, during normal operation, output data when the first memory module outputs data, the second memory module therefore functioning as a redundant component;

means for detecting an interrupt indicating a fault in the first memory module;

means for isolating the fault in the first memory module; and

means for taking the first memory module off-line while maintaining system operations and without affecting systems operation and, while continuing to use the primary component, using the second memory module to perform the function of the first memory module, where the second memory module receives and outputs data that the first memory module would have received and output, respectively, in response to data requests.

9. The computing system of claim 8, further comprising:

means for, upon repair or replacement of the first memory module, reinitializing the first memory module;

means for initiating a test procedure in the repaired or replacement first memory module;

means for reintegrating the repaired or replacement first memory module if the repaired or replacement first memory module passes the test procedure; and

means for placing the repaired or replacement first memory module online if the reintegration step is successfully completed.

10. A computer program product, comprising:

a computer usable medium having computer readable code embodied therein for performing online recovery and reintegration of system components in a computer system having a plurality of Central Processor Units (CPUs) and a plurality of components coupled to the CPUs, the computer program product comprising;

computer readable code configured to cause a computer to effect detecting an interrupt indicating a fault in a system component always designated as a primary system component, the primary system component having at least one matching redundant component always designated as a backup component of the primary system component, where a redundant component receives, during normal operation, the same input data as the primary system component, but, during normal operation, does not output the data output by the primary system component;

computer readable code configured to cause a computer to effect isolating the fault in the primary system component;

computer readable code configured to cause a computer to effect taking the faulty primary system component off-line while maintaining system operations and without affecting systems operation and, while continuing to use the primary component, using the redundant system component matching the faulty primary system component, where the redundant system component receives and outputs data that the faulty primary system component would have received and output, respectively;

computer readable code configured to cause a computer to effect, upon repair or replacement of the faulty primary system component, reinitializing the repaired or replacement primary system component;

computer readable code configured to cause a computer to effect initiating a test procedure in the repaired or replacement primary system component;

computer readable code configured to cause a computer to effect reintegrating the repaired or replacement primary system component if the primary system component passes the test procedure; and

computer readable code configured to cause a computer to effect placing the repaired or replacement primary system component online if the reintegration step is successfully completed.

11. The computer program product of claim 10, wherein said faulty primary system component is a primary memory module and wherein the computer readable code configured to cause a computer to effect reintegrating comprises computer readable code configured to cause a computer to effect mapping the data stored in an operational redundant memory module to the repaired or replacement primary memory module.

12. The computer program product of claim 11, wherein said computer readable code further comprises computer readable code configured to cause a computer to effect reading data from a memory location in said operational memory module and writing the data to both the operational and repaired or replacement primary memory modules.

13. The computer program product of claim 12, wherein said computer readable code further comprises computer readable code configured to cause a computer to effect repeating said reading and writing steps until all the memory locations in said operational memory module have been rewritten into both memory modules.

14. In a computer system having a plurality of Central Processor Units (CPUs) and a plurality of components coupled to the CPUs, a method comprising:

detecting an interrupt indicating a fault in one of a primary system component and a redundant system component, where the redundant component receives, during normal operation, the same input data as the primary system component, but, during normal operation, does not output the data output by the primary system component;

reading, by each of the plurality of CPUs, a respective interrupt cause register;

voting their interrupt cause registers, by each of the plurality of CPUs;

in accordance with the voting step, taking the faulty system component off-line without affecting system operations, the remaining, non-faulty system component handling both the read and write operation;

upon repair or replacement of the faulty system component, reinitializing the repaired or replacement system component;

initiating a test procedure in the repaired or replacement component;

reintegrating the repaired or replacement system component if the system component passes the test procedure; and

placing the repaired or replacement system component online if the reintegration step is successfully completed.