Fail-fast, fail-functional, fault-tolerant multiprocessor system

• US 5,751,932 A
• Filed: 06/07/1995
• Issued: 05/12/1998
• Est. Priority Date: 12/17/1992
• Status: Expired due to Term

• Current Assignee: HTC Corporation
• Original Assignee: Tandem Computers Incorporated (HP Inc.)
• Inventors: Brown, John Michael, Banton, Randall G., Elrod, Harry Frank, Jewett, Douglas Eugene, Drexler, Barry Lee, Hintikka, Paul N., Garcia, David J., Sonnier, David P., Bruckert, William F., Horst, Robert W., Meyers, Steven C., Whiteside, Patricia L., Baker, William Edward, Krause, John C., Cutts, Richard W. Jr., Williams, Frank A., Low, Stephen G., Klecka, James Stevens, Coddington, John Deane, Watson, William Joel, Fowler, Daniel L., Bunton, William Patterson, Jones, Curtis Willard Jr., Zalzala, Linda Ellen, Meredith, Susan Stone, Iswandhi, Geoffrey I., Campbell, Gary F.
• Primary Examiner(s): Beausoliel, Jr., Robert W.
• Assistant Examiner(s): Elmore, Stephen C.
• Application Number:
  US08/485,217
• Time in Patent Office: 1,070 Days
• Field of Search: 395/182.1, 395/182.01, 395/182.08, 395/185.02, 395/182.02, 395/200.01, 395/200.19, 395/250, 395/401, 395/500, 395/821, 395/733, 341/50, 364/268.9, 364/944.61
• US Class Current: 714/12
• CPC Class Codes: G01R 31/31727 Clock circuits aspects, e.g. test clock circuit details, timing aspects for signal generation, circuits for testing clocks G01R31/31725 takes precedence; concerning scan test G01R31/318552, for tester hardware G01R31/31922
• CPC Class Codes: G01R 31/318533 using scanning techniques, e.g. LSSD, Boundary Scan, JTAG
• CPC Class Codes: G06F 1/12 Synchronisation of different clock signals provided by a plurality of clock generators
• CPC Class Codes: G06F 11/004 Error avoidance G06F11/07 and subgroups take precedence
• CPC Class Codes: G06F 11/10 Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
• CPC Class Codes: G06F 11/1044 with specific ECC/EDC distribution
• CPC Class Codes: G06F 11/1604 where the fault affects the clock signals of a processing unit and the redundancy is at or within the level of clock signal generation hardware
• CPC Class Codes: G06F 11/1633 using mutual exchange of the output between the redundant processing components
• CPC Class Codes: G06F 11/1645 and the comparison itself uses redundant hardware
• CPC Class Codes: G06F 11/165 with continued operation after detection of the error
• CPC Class Codes: G06F 11/1658 Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit
• CPC Class Codes: G06F 11/167 Error detection by comparing the memory output
• CPC Class Codes: G06F 11/1679 at clock signal level
• CPC Class Codes: G06F 11/1683 at instruction level
• CPC Class Codes: G06F 11/1687 at event level, e.g. by interrupt or result of polling
• CPC Class Codes: G06F 11/1695 which are operating with time diversity
• CPC Class Codes: G06F 11/20 using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
• CPC Class Codes: G06F 11/2043 where the redundant components share a common memory address space
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 11/2736 using a dedicated service processor for test
• CPC Class Codes: G06F 12/0815 : Cache consistency protocols
• CPC Class Codes: G06F 12/1483 : using an access-table, e.g. matrix or list
• CPC Class Codes: G06F 2201/845 : Systems in which the redundancy can be transformed in increased performance
• CPC Class Codes: G06F 9/30189 : according to execution mode, e.g. mode flag
• CPC Class Codes: G06F 9/3836 : Instruction issuing, e.g. dynamic instruction scheduling or out of order instruction execution
• CPC Class Codes: H04L 12/56 : Packet switching systems
• CPC Class Codes: H04L 47/10 : Flow control; Congestion control
• CPC Class Codes: H04L 49/90 : Buffering arrangements
• CPC Class Codes: H04L 49/901 : using storage descriptor, e.g. read or write pointers
• CPC Class Codes: H04L 49/9057 : Arrangements for supporting packet reassembly or resequencing
• CPC Class Codes: H04L 69/40 : for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

A multiprocessor system includes a number of sub-processor systems, each substantially identically constructed, and each comprising a central processing unit (CPU), and at least one I/O device, interconnected by routing apparatus that also interconnects the sub-processor systems. A CPU of any one of the sub-processor systems may communicate, through the routing elements, with any I/O device of the system, or with any CPU of the system. The CPUs are structured to operate in one of two modes: a simplex mode in which the two CPUs operate independently of each other, and a duplex mode in which the CPUs operate in lock-step synchronism to execute each instruction of identical instruction streams at substantially the same time. Communications between I/O devices and CPUs is by packetized messages. Interrupts from I/O devices are communicated from the I/O devices to the CPUs (or from one CPU to another CPU) as message packets. CPUs and I/O devices may write to, or read from, memory of a CPU of the system. Memory protection is provided by an access validation method maintained by each CPU in which CPUs and/or I/O devices are provided with a validation to read/write memory of that CPU, without which memory access is denied.

Claims

1. A data processing system, comprising:

at least two sub-processing systems, each sub-processing system including;

a central processing unit, the central processing unit including at least a pair of processor elements operating in lock-step, self-checking configuration, and executing substantially identical instructions at substantially the same time,

a peripheral unit, and

a routing element coupled to the central processing unit of each of the sub-processing systems and to the peripheral unit for communicating data between the central processing units and peripheral unit of the sub-processing systems.

2. The processing system of claim 1, wherein the central processing units are operable in a first mode independent of one another to execute instructions of different instruction streams, and a second mode in which the central processing units operate in lock-step synchronism to execute the same instructions at substantially the same time.

3. The processing system of claim 2, wherein the routing element of each of the sub-processing systems includes means to communicate data from the peripheral unit to both central processing units substantially at the same time when the central processing units are operating in the second mode.

4. The processing system of claim 3, each of the central processor units further including data synchronization means coupled to receive the data from the routing elements of each sub-processing system for synchronizing the received data for presentation to each of the central processor units at substantially the same time when the central processing units are operating in the second mode, the data synchronization means including temporary storage means having a plurality of storage locations for storing data received from the routing elements, first addressing means operated by the routing elements to store data from the routing elements in the temporary storage means, and second addressing means operated by the central processing unit to remove data from the temporary storage means.

5. The processing system of claim 4, wherein the temporary storage means is in the form of a first-in-first-out storage means.

6. The processing system of claim 1, wherein the data is communicated between the central processing units and the peripheral units of each sub-processing system in the form of one or more message packets each containing multiple ones of multi-bit data words.

7. The processing system of claim 6, wherein the message packets each contain information identifying the source and destination of the message packets.

8. The processing system of claim 7, wherein each of the central processing units includes memory means and access means permitting data contained in predetermined ones of the message packets to be written to the memory means of the central processing unit.

9. The processing system of claim 8, wherein each of the central processing units includes means responsive to other ones of the message packets requesting data from the memory means to form and send the predetermined ones of the message packets containing requested data from the memory means.

10. The processing system of claim 7, each of the central processing units including a memory means and access validation means operating in response to receipt of the message packets to inspect the source and destination information and determine therefrom whether data will be allowed to be read from or written to the memory means.

11. The processing system of claim 1, wherein the routing element of each sub-processing systems is coupled to the central processing unit of each other of the sub-processing systems by each of a pair of bi-directional data communicating links, respectively.

12. The processing system of claim 1, wherein a second routing element is coupled to the central processing unit of each of the two sub-processing systems by each of a pair bi-directional data communicating links, respectively.

13. The processing system of claim 1, wherein the data communicated between the central processing units is in the form of an information packet having a destination field identifying a destination of the information packet, and a source field identifying a sender of the information packet.

14. A processing system, comprising:

at least a pair of central processing means for performing data processing functions;

a plurality of peripheral elements; and

routing means having a plurality of ports coupled to the pair of central processing means and to the plurality of peripheral elements for communicating data therebetween, each of the plurality of ports being configured to provide simultaneous bi-directional communication;

whereby the routing means is operable to provide a first communication path between the pair of central processing means and a plurality of second communication paths between either one of the pair of central processing means and any one of the plurality of peripheral elements.

15. The processing system of claim 14, wherein the pair of central processing means are operated in lock-step synchronism to each execute each instruction of identical instruction streams at substantially the same time, the routing means being operable to receive data from a one of the peripheral elements, replicating the data, and transmitting the data to the pair of central processing means substantially simultaneously.

16. The processing system of claim 14, wherein the data communicated between the pair of central processing means and the plurality of peripheral elements is in the form of message packets, each message packet including a plurality of data words.

17. The processing system of claim 16, each message packet further including a destination field, and wherein the routing means operates to route each message packet according to the destination field.

18. The processing system of claim 14, wherein the pair of central processing means are operable in a first mode to execute different instruction streams, and in a second mode in which the pair of central processing means execute substantially identical instruction streams in lock-step synchronism.

19. A processing system, comprising:

a pair of central processing units each including at least a pair of processor elements configured to operate in lock-step, self-checking configuration to execute a stream of instructions, the pair of central processor units being configured to operate in a first mode whereby a one of the pair of central processing units executes instructions that are different from instructions executed by the other of the pair of central processing units, and a second mode in which the pair of central processor units operate in a lock-step mode to executed substantially the same instructions at substantially the same time;

first and second router elements each coupled to communicate data with the pair of central processing units; and

at least first and second peripheral units respectively coupled to the first and second router elements;

wherein the first and second router elements are operable, when the pair of central processor units are in the second mode, to communicate data from the first or the second peripheral units to both the pair of central processor units substantially simultaneously.

20. The processing system of claim 19, wherein the data is in the form of a message packet including a destination field identifying the destination of the message packet.

21. The processing system of claim 20, wherein the first and second router elements each include a plurality of input-output ports whereat the data is received and/or transmitted, the pair of central processing elements each being coupled to a corresponding one of the input-output ports.

22. The processing system of claim 21, wherein the first and second router elements each operate to receive the message packet with the destination identifying a one of the pair of central processing units to communicate the message packet to the pair of central processing units at substantially the same time when the central processing units are operating in the second mode.

23. In a computing system having a pair of central processing units structured to operate in a mode to execute each instruction of substantially the same instruction stream at substantially the same time using a local clock signal, the pair of central processing units being coupled to a router element for communicating data to and from at least one peripheral unit that is coupled to the router element, a method of communicating data from the peripheral unit to the pair of central processor units, comprising:

the router element transmitting the data to the pair of central processor units in synchronized fashion together with a transmitting clock signal;

each of the pair of central processor units storing the data received from the router element at locations of a storage medium synchronous with the transmitting clock signal;

each of the pair of central processor units sequentially accessing the received data from the locations of the storage medium with a receiving clock signal that is synchronized to the local clock signal.

24. A data processing system, comprising:

at least a pair of central processing units, each of the pair of central processing units having first and second data ports for receiving and transmitting data;

at least first and second peripheral control units for receiving and sending data; and

a first routing element having a number of ports whereat data is received and transmitted, each of a pair of the number of ports being coupled to the first data ports of the pair of central processing units, another of the number of ports being coupled to at least the first peripheral control unit; and

a second routing element having a number of ports whereat data is received and transmitted, each of a pair of the number of ports being coupled to the first data ports of the pair of central processing units, another of the number of ports being coupled to at least the second peripheral control unit.

25. The data processing system of claim 24, wherein the first and second peripheral control units each has a pair of ports each respectively coupled to a predetermined one of the number of ports of the first and second routing elements.

26. The data processing system of claim 24, wherein data is communicated between the pair of central processing units and the first and second peripheral control units in the form of message packets, each message packet including information identifying a sender of the message packet and a receiver of the message packet.

27. The data processing system of claim 26, wherein predetermined ones of message packets include data.

28. The data processing system of claim 27, wherein certain ones of the predetermined ones of message packets from the first or the second peripheral control units includes data to be written to a one of the pair of central processing units.

29. A data processing system, comprising:

at least two sub-processing systems, each sub-processing system including;

a central processing unit, including at least a pair of processor elements operating in lock-step, self-checking configuration, and executing substantially identical instructions at substantially the same time, the central processing unit of each of the two sub-processing systems being operable in a first mode independent of one another to execute instructions of different instruction streams, and a second mode in which the central processing unit of each of the two sub-processing systems operate in lock-step synchronism to execute the same instructions at substantially the same time,

a peripheral unit, and

a routing element coupled to the central processing unit of each of the sub-processing systems and to the peripheral unit for communicating data between the central processing units and peripheral unit of the sub-processing systems the routing element including means to communicate data from the peripheral unit to both central processing units substantially at the same time when the central processing units are operating in the second mode;

the central processing unit including data synchronization means coupled to receive the data from the routing elements of each sub-processing system for synchronizing the received data for presentation to each of the central processor units at substantially the same time when the central processing units are operating in the second mode, the data synchronization means including temporary storage means having a plurality of storage locations for storing data received from the routing elements, first addressing means operated by the routing elements to store data from the routing elements in the temporary storage means, and second addressing means operated by the central processing unit to remove data from the temporary storage means.

30. A data processing system, comprising:

at least two sub-processing systems, each sub-processing system including;

(a) a central processing unit, each central processing unit including at least a pair of processor elements operating in lock-step, self-checking configuration, and executing substantially identical instructions at substantially the same time, a peripheral unit, and

(b) a routing element coupled to the central processing unit of each of the sub-processing systems and to the peripheral unit for communicating data between the central processing units and peripheral unit of the sub-processing systems, the data being communicated between the central processing units and the peripheral units of each sub-processing system in the form of one or more message packets each containing multiple ones of multi-bit data words, and wherein the message packets each contain information identifying the source and destination of the message packets; and

(c) the central processing unit including a memory element having an access validation table that is used in response to receipt of the message packets to inspect the source and destination information and determine therefrom whether data will be allowed to be read from or written to the memory means.