Interrupts between asynchronously operating CPUs in fault tolerant computer system

• US 5,890,003 A
• Filed: 09/07/1993
• Issued: 03/30/1999
• Est. Priority Date: 12/09/1988
• Status: Expired due to Term

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: Tandem Computers Incorporated (HP Inc.)
• Inventors: Southworth, Richard A., Allison, John David, Horst, Robert W., Jewett, Douglas E., Cutts, Richard W. Jr., Mehta, Nikhil A., Debacker, Kenneth C.
• Primary Examiner(s): Ellis, Richard L.
• Application Number:
  US08/116,950
• Time in Patent Office: 2,030 Days
• Field of Search: 395/550, 395/800, 395/591, 395/736, 395/800.28, 395/800.42, 395/182.09, 395/182.19, 371/11.3, 371/36, 371/68.3
• US Class Current: 710/263
• CPC Class Codes: G06F 11/0724 in a multiprocessor or a multi-core unit multiprocessors per se G06F15/80
• CPC Class Codes: G06F 11/1405 at machine instruction level
• CPC Class Codes: G06F 11/1645 and the comparison itself uses redundant hardware
• CPC Class Codes: G06F 11/1658 Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit
• CPC Class Codes: G06F 11/1666 where the redundant component is memory or memory area
• CPC Class Codes: G06F 11/1679 at clock signal level
• CPC Class Codes: G06F 11/1683 at instruction level
• CPC Class Codes: G06F 11/1687 at event level, e.g. by interrupt or result of polling
• CPC Class Codes: G06F 11/1691 using a quantum
• CPC Class Codes: G06F 11/181 Eliminating the failing redundant component
• CPC Class Codes: G06F 11/182 based on mutual exchange of the output between redundant processing components
• CPC Class Codes: G06F 11/185 and the voting is itself performed redundantly
• CPC Class Codes: G06F 11/20 using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
• CPC Class Codes: G06F 11/2007 using redundant communication media
• CPC Class Codes: G06F 11/2015 Redundant power supplies power supply failure G06F1/30
• CPC Class Codes: G06F 11/2017 where memory access, memory control or I/O control functionality is redundant redundant communication control functionality G06F11/2005; redundant storage control functionality G06F11/2089
• CPC Class Codes: G06F 9/52 Program synchronisation; Mutual exclusion, e.g. by means of semaphores

Abstract

A computer system in a fault-tolerant configuration employs three identical CPUs executing the same instruction stream, with two identical, self-checking memory modules storing duplicates of the same data. Memory references by the three CPUs are made by three separate busses connected to three separate ports of each of the two memory modules. The three CPUs are loosely synchronized, as by detecting events such as memory references and stalling any CPU ahead of others until all execute the function simultaneously; interrupts can be synchronized by ensuring that all three CPUs implement the interrupt at the same point in their instruction stream. Memory references via the separate CPU-to-memory busses are voted at the three separate ports of each of the memory modules. I/O functions are implemented using two identical I/O busses, each of which is separately coupled to only one of the memory modules. A number of I/O processors are coupled to both I/O busses.

Claims

1. A multiple CPU system, comprising:

a) a plurality of CPUs concurrently executing a same instruction stream, the CPUs being the only CPUs in the multiple CPU system and wherein all of the CPUs in the multiple CPU system are clocked independently of one another to provide separate machine clock cycles for each CPU, so that said instruction stream is executed asynchronously over plural instructions and any of said CPUs may be leading other of said CPUs, said machine clock cycles including execution cycles where an instruction of said instruction stream is executed and stall cycles where an instruction of said instruction stream is not executed, each CPU having a memory request input/output port;

b) a common memory coupled to the input/output ports of said CPUs, the common memory implementing a memory request only after receiving identical requests from all of said CPUs, the memory sending an acknowledge signal to the CPUs when implementing a memory request, each of the CPUs executing stall cycles while awaiting implementation of a memory request by the common memory as signaled by said acknowledge signal;

c) each of the CPUs having a counter to count machine clock cycles corresponding to execution cycles but which is inhibited from counting machine clock cycles corresponding to stall cycles; and

d) all of said CPUs in the system having an interrupt circuit responsive to an external interrupt request occurring at any time unsynchronized with said execution of said instruction stream, said interrupt circuit being coupled to said counters in said CPUs and responsive to a selected count in each of said counters for separately interrupting each CPU at an identical instruction execution cycle, any one of the CPUs which may be leading being interrupted first while other of said CPUs continue to execute instructions so that if said CPUs are executing different instructions in said stream then said CPUs may be interrupted at different instants in real time.

2. A system according to claim 1 wherein each of said CPUs has a local memory not accesable by the other CPUs.

3. A system according to claim 1 wherein there are three of said CPUs, and wherein said requests to said common memory via said port are voted by said common memory.

4. A system according to claim 1 wherein said interrupt circuit interrupts said CPUs only on a selected value registered by said counters.

5. A system according to claim 1 wherein said interrupt circuit includes for each CPU means responsive to an indication of receipt of an interrupt request when a selected value is registered in said counter of each of the other CPUs.

6. A multiple CPU system with synchronization of external interrupts, comprising:

a) a plurality of CPUs independently and concurrently executing the same instruction stream, the CPUs being the only CPUs in the multiple CPU system and wherein all of the CPUs in the multiple CPU system are each clocked independently of one another by respective independent machine clock cycles so that said instruction stream is executed asynchronously over plural instructions and any one of said CPUs may lead or lag other of said CPUs, to provide execution cycles during which instructions of said same instruction stream are executed and to provide stall cycles during which instructions are not executed;

b) each of the CPUs having a counter counting only machine clock cycles corresponding to said execution cycles and which is inhibited from counting machine clock cycles corresponding to stall cycles; and

c) and an interrupt circuit connected to all of said CPUs in the multiple CPU system and responsive to an external interrupt request occurring at any time uncorrelated with which CPU is leading or lagging in said executing of said instruction stream, said interrupt circuit being responsive to a selected count in each of said counters for interrupting each CPU in the multiple CPU system separately at a same execution cycle in said instruction stream without stalling a CPU which may be leading another of the CPUs.

7. A system according to claim 6 wherein there are three said CPUs, and wherein all three CPUs access a common memory module, requests to said common memory from said CPUs being voted by said memory module.

8. A system according to claim 6 wherein each of said CPUs has a local memory not accessible by other CPUs.

9. A multiple CPU system, comprising:

a) a plurality of CPUs concurrently executing a same instruction stream, the CPUs being the only CPUs in the multiple CPU system and wherein all of the CPUs in the multiple CPU system are clocked independently of one another by respective independent machine clock cycles to provide execution cycles during which instructions are executed and stall cycles during which instructions are not executed, so that said instruction stream is executed asynchronously over plural instructions and any one of the CPUs may be leading or lagging other of said CPUs,

b) each of the CPUs in the multiple CPU system having counter means for counting machine clock cycles but inhibiting counting during clock cycles corresponding to stalls to provide a count corresponding to execution cycles but not stall cycles; and

c) and an interrupt circuit connected to each of said CPUs in the multiple CPU system and responsive to an external interrupt request separate from and not correlated with which CPU may be leading in executing said instruction stream, said interrupt circuit being responsive to a selected count in each of said counter means for separately interrupting each CPU in the multiple CPU system at a same execution cycle in said instruction stream, a leading one of said CPUs continuing to execute instructions without being stalled by said interrupt circuit even if others of the CPUs are executing earlier instructions from said instruction stream.

10. A system according to claim 9 wherein there are three said CPUs, and wherein all three CPUs access a common memory module, requests to said common memory from said CPUs being voted by said memory module.

11. A system according to claim 9 wherein each of said CPUs has a local memory not accessible by other CPUs.

12. A system according to claim 9 wherein there are three of said CPUs, and wherein each of said CPUs makes access requests to a common memory, and said access requests are voted by said common memory.

13. A system according to claim 9 wherein said interrupt circuit includes for each CPU means responsive to receipt of an interrupt request when a selected value is registered in said counter of each of the other CPUs.

14. A multiple CPU system, comprising:

a) a plurality of CPUs, each of the CPUs independently and concurrently executing a same instruction stream, the CPUs being the only CPUs in the multiple CPU system and wherein all of the CPUs in the multiple CPU system are clocked independently of one another in accordance with respective independent machine clock cycles to provide execution cycles where instructions are executed and stall cycles where instructions are not executed so that said instruction stream is executed asynchronously over plural instructions and any one of said CPUs in the multiple CPU system may be leading or lagging other of said CPUs, the CPUs each having an input/output port, at least one shared input/output device being coupled to said input/output ports of the plurality of CPUs;

b) each of the CPUs in the multiple CPU system having a modulo N counter to count machine clock cycles corresponding to execution cycles but which is inhibited from counting during stall cycles; and

c) an interrupt circuit coupled to each of said CPUs in the multiple CPU system and responsive to an external interrupt request occurring uncorrelated with which CPU may lead in executing said instruction stream, said interrupt circuit being correlated with said counters for applying an interrupt separately to each CPU in the multiple CPU system at a preselected count value of each of the counters, so that each CPU in the multiple CPU system is interrupted at a same execution cycle in said instruction stream but not necessarily at a same point in real time as the other CPUs.

15. A system according to claim 14 wherein there are three said CPUs, and wherein all three CPUs access a common memory module, requests to said common memory from said CPUs being voted by said memory module.

16. A system according to claim 14 wherein said shared input/output device is a common memory.

17. A system according to claim 14 wherein said interrupt circuit interrupts each of said CPUs when a selected value is registered by said counter for each CPU without stalling execution of instructions by faster ones of said CPUs.

18. A system according to claim 14 wherein said interrupt circuit includes for each CPU means responsive to receipt of an interrupt request when a selected value is registered in said counter of each of the other CPUs.

19. A system according to claim 14 wherein each of said CPUs has a local memory not accessible by other CPUs.

20. A method of operating a multiple CPU computer system having a plurality of separately-clocked CPUs where any of the CPUs in the multiple CPU system may lead or lag other of the CPUs by plural instructions, comprising the steps of:

a) executing a same instruction stream concurrently on all of said CPUs in the multiple CPU system in accordance with execution clock cycles;

b) performing stall clock cycles when instructions are not being executed;

c) counting execution clock cycles for each of said CPUs in the multiple CPU system;

d) inhibiting said counting during stall clock cycles for each of said CPUs in the multiple CPU system; and

e) detecting an external interrupt request and applying an interrupt signal separately to each one of the CPUs in the multiple CPU system at a selected execution clock cycle count, said external interrupt request being uncorrelated with which CPU may be leading or lagging other CPUs in executing said instruction stream, said interrupt signal being applied to each CPU not necessarily at a same point in real time as an interrupt signal is applied to other of the CPUs.

21. A method according to claim 20 including the step of making memory access requests by said CPUs to a common memory and voting each of said requests by said common memory.

22. A method according to claim 21 including the step of accessing a local memory separately by each of said CPUs, the local memory for each CPU not being accessible by other CPUs.

23. A method according to claim 20 wherein said step of detecting an external interrupt request is performed only at a selected value of said count of instructions.