Multiprocessor system

• US 4,228,496 A
• Filed: 09/07/1976
• Issued: 10/14/1980
• Est. Priority Date: 09/07/1976
• Status: Expired due to Term

• Current Assignee: TCI-Delaware Incorporated
• Original Assignee: Tandem Computers Incorporated (HP Inc.)
• Inventors: Green, Michael D., Wierenga, Steven W., Bartlett, Joel F., Mackie, David R., McEvoy, Dennis L., Katzman, James A., Treybig, James G., Hayashi, Steven J., Bixler, Richard M., Despotakis, John A., Davidow, William H., Greig, David A., Graziano, Peter J.
• Primary Examiner(s): Nusbaum, Mark E.
• Application Number:
  US05/721,043
• Time in Patent Office: 1,498 Days
• Field of Search: 364/200 MS File, 364/900 MS File
• US Class Current: 710/100
• CPC Class Codes: G06F 11/10 Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
• CPC Class Codes: G06F 11/1044 with specific ECC/EDC distribution
• CPC Class Codes: G06F 11/2007 using redundant communication media
• CPC Class Codes: G06F 11/2015 Redundant power supplies power supply failure G06F1/30
• CPC Class Codes: G06F 11/2023 Failover techniques
• CPC Class Codes: G06F 11/2035 without idle spare hardware
• CPC Class Codes: G06F 11/2043 where the redundant components share a common memory address space
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 12/08 in hierarchically structured memory systems, e.g. virtual memory systems
• CPC Class Codes: G06F 12/14 Protection against unauthorised use of memory or access to memory
• CPC Class Codes: G06F 12/1458 by checking the subject access rights
• CPC Class Codes: G06F 13/122 where hardware performs an I/O function other than control of data transfer
• CPC Class Codes: G06F 13/20 for access to input/output bus
• CPC Class Codes: G06F 13/287 Multiplexed DMA G06F13/30 takes precedence
• CPC Class Codes: G06F 13/366 using a centralised polling arbiter
• CPC Class Codes: G06F 13/38 Information transfer, e.g. on bus G06F13/14 takes precedence
• CPC Class Codes: G06F 15/16 Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs coordinating program control therefor G06F9/52; in regulating and control system G05B
• CPC Class Codes: G06F 15/167 using a common memory, e.g. mailbox
• CPC Class Codes: G06F 15/173 using an interconnection network, e.g. matrix, shuffle, pyramid, star, snowflake
• CPC Class Codes: G06F 5/06 for changing the speed of data flow, i.e. speed regularising or timing, e.g. delay lines, FIFO buffers; over- or underrun control therefor G06F7/78 takes precedence
• CPC Class Codes: G06F 7/78 : for changing the order of data flow, e.g. matrix transposition or LIFO buffers; Overflow or underflow handling therefor

Abstract

A multiprocessor system the kind in which two or more separate processor modules are interconnected for parallel processing includes two redundant interprocessor buses dedicated exclusively to interprocessor communication. Any processor module may send information to any other processor module by either bus. The buses are shared in use by the processor modules on a time-sharing basis. Use of each bus is controlled by a special bus controller.

The multiprocessor system includes an input/output system having multi-port device controllers and input/output buses connecting each device controller for access by the input/output channels of at least two different processor modules. Each device controller includes logic which insures that only one port is selected for access at a time.

The multiprocessor system includes a distributed power supply system which insures nonstop operation of the remainder of the multiprocessor system in the event of a failure of a power supply for a part of the system. The distributed power supply system includes a separate power supply for each processor module and two separate power supplies for each device controller. Either one of the two power supplies provides the entire power for the device controller in the event the other power supply fails. The distributed power supply system permits any processor module or device controller to be powered down so that on-line maintenance can be performed in a power-off condition while the rest of the multiprocessor system is on-line and functional.

The multiprocessor system includes a memory system in which the memory of each processor module is divided into four logical address areas--user data, system data, user code and system code. The memory system includes a map which translates logical addresses to physical addresses and which coacts with the multiprocessor system to bring pages from secondary memory into primary main memory as required to implement a virtual memory system. The map also provides a protection function. It provides inherent protection among users in a multiprogramming environment, isolates programs from data and protects system programs from the actions of user programs. The map also provides a reference history information for each logical page as an aid to efficient memory management by the operating system.

The multiprocessor system includes in the memory of each processor module an error detection and correction system which detects all single bit and double bit errors and which corrects all single bit errors in semiconductor memory storage.

Claims

1. A multiprocessor system of the kind in which separate processor modules operate concurrently and cooperatively for system control and application processing, said multiprocessor system comprising,

a plurality of separate processor modules,

each processor module comprising a central processing unit within the processor module, a separate main read-write word addressable memory within the processor module having sufficient capacity for the storage of system control and application programs and data, and a dedicated memory bus within the processor module connecting the separate main memory in the processor module exclusively to its associated central processing unit in the processor module for access to said programs and data without contention with central processing units in other processor modules,

interprocessor bus means separate and distinct from an input/output system and from a memory bus and interconnecting the processor modules for direct processor module to processor module signaling and data transfer,

said interprocessor bus means including,

a common shared interprocessor bus,

interprocessor control means in each processor module for connection that processor module to the interprocessor bus, and

centralized bus controller means operatively associated with the interprocessor bus and each interprocessor control means for determining the priority of data transfers over the interprocessor bus and for controlling transmissions over the interprocessor bus,

said bus controller means including,

arbitration means for centrally arbitrating the priority of said data transfers and

bus clock generator means for controlling the time sequence of data transfer by a processor module over the interprocessor bus, said processor module including:

reading means for reading data from the associated main read/write memory of that processor module for transmission to the interprocessor bus, and

writing means for writing data received from the interprocessor bus into a specied location in the association main read/write memory of that processor module, wherein said reading and writing means are effective to transfer data from the memory of a sender processor module to the memory of a receiver processor module, and wherein each processor module includes a buffer for each other processor module and also includes location pointing means for directing incoming data from an interprocessor bus to a specified location in a related buffer in the memory of a receiver processor module.

2. The invention defined in claim 1 wherein the dedicated memory bus in each processor module connects the memory in a given processor module directly to the central processor unit in that processor module and is effective to exclude any direct connection to a central processing unit in any other processor module so that the multiprocessor system is not subject to contention by central processor units for a shared memory and is not subject to failure because of failure of a shared memory.

3. The invention defined in claim 1 wherein the bus controller means include processor select logic means and individual select lines, each select line extending from the processor select logic means to the interprocessor control means of one associated processor module, and wherein the processor select logic means generate select signals on the select lines to guarantee that one and only one processor module can send data on the interprocessor bus at a time.

4. The invention efined in claim 3 wherein the processor select logic means generate select signals over the individual select lines to enable transmissions from a processor module to the interprocessor bus solely under the control of the bus controller means.

5. The invention defined in claim 3 wherein the bus controller means also include bus control state logic means for identifying a sender-receiver pair of processor modules and for defining a time frame for a transfer of data over the bus between the sender-receiver pair.

6. The invention defined in claim 5 wherein the processor select logic means and interprocessor control means are operatively associated by the select signals.

7. The invention defined in claim 5 including qualifying means for enabling the receiver processor module to receive data from another processor module without execution of a receive data instruction by the receiver processor module.

8. The invention defined in claim 1 wherein the bus controller means include bus control state logic means wherein the qualifying means include the combination of first means in the processor select logic means for generating a select signal on the select line to the receiver processor module, second means in the bus control state logic means for producing a receive command signal to the receiver processor module for interrogating readiness of that receiver processor module to receive, and third means in the interprocessor control means of the receiver processor module for generating an affirmative or negative receive acknowledge signal to the bus control state logic means for indicating said readiness to receive.

9. The invention defined in claim 8 wherein the processor select logic means generate only one select signal at a time to prevent other processor modules from transmitting on the bus simultaneously with the established sender processor module.

10. The invention defined in claim 3 wherein each connection of the interprocessor bus to a processor module comprises a plurality of common data path lines which are shared by all modules of the multiprocessor system, a plurality of common bus protocol lines, one bus clock line and one select line.

11. The invention defined in claim 10 wherein the bus controller means and the interprocessor control means of a sender and a receiver processor module are operatively associated and include means to generate on the interprocessor bus in sequence a

send request signal, from any processor module which is ready to send, to cause the bus controller to leave an idle condition and to begin a polling operation,

select signal, from the bus controller to poll in sequence each processor module until a processor module which is ready to send is identified,

send acknowledge signal, with the receiver processor module number on the data lines, from a processor module to indicate readiness to send in response to the select signal and also to identify the receiver processor module to the bus controller,

select signal, with the receive command signal, and with the identified sender processor module number on the data lines, from the bus controller to interrogate the ready to receive condition of the identified receiver processor module and to identify the sender processor module to that receiver processor module,

receive acknowledge signal, from the receiver processor module to indicate the ready to receive condition and in response to the select signal and the receive command signal from the bus controller means,

select signal, with the send command signal, from the bus controller means in response to a positive receive acknowledge indication, to enable the sender processor module to sequentially transmit a plurality of data words on the data lines to the receiver processor module, after which the bus controller means returns to an idle condition.

12. The invention defined in claim 1 wherein each processor module includes a bus receive table means which contains a separate entry corresponding to each processor module and specifies both the buffer address where incoming data from a particular processor module is to be stored and the number of data words expected.

13. The invention defined in claim 1 wherein the memory address space of each processor is divided into four separate logical address areas comprising a user data area, to system data area, a user code area and a system code area to separate code from data so that code can be made non-modifiable and also to separate operating system programs and data from user programs and data so that user programs cannot destroy the operating system.

14. A multiprocesor system of the kind in which separate processor modules operate concurrently and cooperatively for system control and application processing, said multiprocessor system comprising,

a plurality of separate processor modules,

each processor module comprising a central processing unit within the processor module, a separate main read-write word addressable memory within the processor module having sufficient capacity for the storage of system control and application programs and data, and a dedicated memory bus within the processor module connecting the separate main memory in the processor module exclusively to its associated central processing unit in the processor module for access to said programs and data without contention with central processing units in other processor modules,

interprocessor bus means separate and distinct from an input/output system and from a memory bus and interconnecting the processor modules for direct processor module to processor module signaling and data transfer,

said interprocessor bus means including,

a common shared interprocessor bus,

interprocessor control means in each processor module for connecting that processor module to the interprocessor bus, and

centralized bus controller means operatively associated with the interprocessor bus and each interprocessor control means for determining the priority of data transfers over the interprocessor bus and for controlling transmissions over the interprocessor bus,

said bus controller means including,

arbitration means for centrally arbitrating the priority of data transfers and

bus clock generator means for controlling the time sequence of data transfer by a processor module over the interprocessor bus,

said interprocessor control means in each processor module including,

inqueue means for receiving a sequence of data words from the interprocessor bus in a time interval defined by the bus controller means and

outqueue means for transmitting a sequence of data words to the interprocessor bus in the time interval defined by the bus controller means

said processor module including:

reading means for reading data from the associated main read/write memory of that processor module for transmission to the interprocessor bus, and

writing means for writing data received from the interprocessor bus into a specied location in the associated main read/write memory of that processor module, wherein said reading and writing means are effective to transfer data from the memory of a sender processor module to the memory of a receiver processor module, and wherein each processor module includes a buffer for each other processor module and also includes location pointing means for directing incoming data from an interprocessor bus to a specified location in a related buffer in the memory of a receiver processor module.

15. The invention defined in claim 14 wherein each processor module includes a central processing unit, the bus clock generator means include a bus clock and each central processing unit includes its own separate processor clock and including interlock means operatively associated with the inqueue means and the outqueue means for operating a part of the inqueue means and a part of the outqueue means in synchronism with the interprocessor bus at the bus clock rate and for operating another part of the inqueue means and another part of the outqueue means in synchronism with the central processing unit of each processor module at its own separate processor clock rate without loss of data.

16. The invention defined in claim 15 wherein the inqueue means and the outqueue means are constructed to provide transfer of information over the interprocessor bus at a rate independent of and substantially faster than the rate permitted by direct memory accesses into and out of the memory sections of the processor modules to insure adequate interprocessor bus availability when more than two processor modules are connected to the interprocessor bus.

17. The invention defined in claim 15 wherein the inqueue means include bus fill stage logic and the outqueue means include bus empty state logic which operate in synchronism with the bus clock and wherein the inqueue means include processor empty state logic and the outqueue means include procesor fill state logic which operate in synchronism with the central processing unit clock.

18. The invention defined in claim 17 wherein the interlock means include logic interlocks between the processor fill state logic and the bus empty state logic of the outqueue means and between the processor empty state logic and the bus fill stage logic of the inqueue means and wherein the logic interlocks qualify certain transitions of the logic from one state to another state and make correct operation independent of the relative clock rates.

19. The invention defined in claim 15 including a second interprocessor bus and a second bus controller means and wherein the interprocessor control means of each processor module include inqueue means for each interprocessor bus and outqueue means and wherein the interprocessor control means fill the outqueue means and empty and inqueue means in synchronism with the central processing unit clock and empty the outqueue means and fill the inqueue means in synchronism with the associated interprocessor bus clock.

20. The invention defined in claim 19 wherein each outqueue means include bus empty state logic, processor fill state logic, and outqueue counter, an outqueue buffer, a receive register and an outqueue pointer for connecting the outqueue buffer to one of the two interprocessor buses.

21. The invention defined in claim 15 wherein the inqueue means includes logic means for performing interprocessor bus control and data transfer concurrently with computation by the receiver processor module.

22. The invention defined in claim 14 wherein the bus controller means include processor select logic means for generating a signal for selecting a particular processor module as a receiver processor module and for generating a second signal for selecting another particular processor module as a sender processor module, bus control state logic means for responding to a signal from the receiver processor module by generating a send command signal for enabling the sender processor module to transmit information on the bus, and packet counter means for counting the bus clock cycles comprising a packet of information transmitted on the bus.

23. The invention defined in claim 14 wherein each inqueue means include bus fill state logic means for providing a ready to receive signal to the bus controller and for controlling the storing of words received from the interprocessor bus into inqueue buffer means, processor empty state logic means for providing an inqueue full signal to the central processor unit and for controlling the reading of words stored in inqueue buffer means, inqueue buffer means for storing words received from the interprocessor bus, and inqueue counter means for counting words stored into and read from the inqueue buffer means.

24. The invention defined in claim 14 wherein each central processing unit includes a microprocessor and a control memory and wherein the control memory controls the microprocessor to cause the microprocessor to interact with the interprocessor control means to effect the inqueue emptying and the outqueue filling and wherein the bus controller means control the actual transmission of the data on the interprocessor bus including outqueue emptying and inqueue filling.

25. A multiprocessor system of the kind in which separate processor modules operate concurrently and cooperatively for system control and application processing, said multiprocessor system comprising,

a plurality of separate processor modules, each processor module comprising a central processing unit within the processor module, a separate main read-write word addressable memory within the processor module having sufficient capacity for the storage of system control and application programs and data, and a dedicated memory bus within the processor module connecting the separate main memory in the processor module exclusively to its associated central processing unit in the processor module for access to said programs and data without contention with central processing units in other processor modules,

a plurality of interprocessor bus means each separate and distinct from an input/output system and from a memory bus and each interconnecting the processor modules for direct processor module to processor module signaling and data transfer and wherein each interprocessor bus means is separate and distinct from the other interprocessor bus means so that failure of a particular interprocessor bus means does not prevent signaling and data transfer over the other interprocessor bus means,

each said interprocessor bus means including,

a common shared interprocessor bus,

interprocessor control means in each processor module for connecting that processor module to each interprocessor bus, and

a separate bus controller means operatively associated with a particular interprocessor bus and with each interprocessor control means for determining the priority of data transfers over the interprocessor bus and for controlling transmissions over the interprocessor bus,

each of said bus controller means including, arbitration means for centrally arbitrating the priority of data transfers and

bus clock generator means for controlling the time sequence of data transfer by a processor module over the interprocessor bus.

26. The invention defined in claim 25 wherein the interprocessor control means and bus controller means are operatively associated to permit each processor module to send data over any one of the interprocessor buses to any one of the processor modules and to permit each processor module to receive data simultaneously over a plurality of the interprocessor buses

27. The invention defined in claim 26 including operating means for operating all of the interprocessor buses and bus controller means simultaneously and independently of each other to provide multiple paths of interconnection between the separate processor modules.

28. A multiprocessor system of the kind in which separate processor modules operate concurrently and cooperatively for system control and application processing, said multiprocessor system comprising,

a plurality of separate processor modules, each processor module comprising a central processing unit within the processor module, a separate main read-write word addressable memory within the processor module having sufficient capacity for the storage of system control and application programs and data, and a dedicated memory bus within the processor module connecting the separate main memory in the processor module exclusively to its associated central processing unit in the processor module for access to said programs and data without contention with central processing units in other processor modules,

interprocessor bus means separate and distinct from an input/output system and from a memory bus and interconnecting the processor modules for direct processor module to processor module signaling and data transfer,

said interprocessor bus means including,

a common shared interprocessor bus,

interprocessor control means in each processor module for connecting that processor module to the interprocessor bus, and

centralized bus controller means operatively associated with the interprocessor bus and each interprocessor control means for determining the priority of data transfers over the interprocessor bus and for controlling transmissions over the interprocessor bus,

said bus controller means including,

arbitration means for centrally arbitrating the priority of data transfers and

bus clock generator means for controlling the time sequence of data transfer by a processor module over the interprocessor bus,

said interprocessor means in each processor module including,

inqueue means for receiving a sequence of data words from the interprocessor bus in a time interval defined by the bus controller means and

outqueue means for transmitting a sequence of data words to the interprocessor bus in a time interval defined by the bus controller means,

each central processing unit including a microprocessor and an associated control memory,

said control memory including,

send controller means for dividing a block of data into individual packets each comprising a predetermined number of data words and for interacting with the interprocessor control means to fill the outqueue means with each packet transmitted, and

bus receive controller means for interacting with the interprocessor control means to empty the inqueue means of each packet received and to reassemble received packets into the original data block, and

wherein the bus controller means includes packet counter means operatively associated with the bus clock generator means for defining a time interval for transmission of each packet over the interprocessor bus to permit the usage of the interprocessor bus to be multiplexed on a packet by packet basis.

29. The invention defined in claim 28 including a plurality of interprocessor buses operatively associated with each processor module and a separate bus controller means operatively associated with each interprocessor bus for determining the priority of each data packet transferred between any two processor modules over that interprocessor bus and wherein the interprocessor control means in each processor module include inqueue means operatively associated with each interprocessor bus connected to that processor module.

30. The invention defined in claim 29 wherein each processor module memory includes receiver buffer means for accumulating packets received from each related processor module and also includes bus receive table means for directing incoming data from an interprocessor bus to a specified location in the related receiver buffer means in the memory of the processor module.

31. The invention defined in claim 30 wherein the bus receive table means in the receiver processor module provide a bus receive table entry which contains the current address where the incoming data is to be stored and the remaining number of words expected from the sender processor module.

32. The invention defined in claim 30 wherein the bus receive controller means are operatively associated with the inqueue means and the bus receive table means for storing the received data in the memory of a receiver processor module and for updating the count and address words in the bus receive table without interrupting the program currently executing in the receiving processor module.

33. The invention defined in claim 32 wherein the bus receive controller means update the bus receive table means after the receipt of each packet by the receiver processor module and provide a program interrupt when the bus receive table means indicate an entire data block has been received.

34. The invention defined in claim 32 wherein the bus receive controller means also provide a bus receive completion interrupt to the receiver processor module in response to the detection of an error by the bus receive controller means.

35. The invention defined in claim 29 including receiver control means in each processor module for controlling the ability of that processor module to receive packets over each interprocessor bus.

36. The invention defined in claim 35 including a bus receive table in each processor module for directing incoming data to a specified location in memory and wherein the receiver control means include mask register means for providing an interlock to prevent the receipt of additional data over an interprocessor bus until after certain critical operations such as software updating of the bus receive table have been completed.

37. The invention defined in claim 36 wherein the receiver control means include a completion interrupt flag set by the bus receive controller means.

38. The invention defined in claim 37 wherein the receiver control means include an interrupt mask register in each central processing unit corresponding to each interprocessor bus and having a mask which permits microinterrupts for that bus when the mask bit is on.

39. The invention defined in claim 29 wherein the central processing unit control memory in each processor module includes a send controller which causes the microprocessor in the central processing unit to interact with the outqueue means of the interprocessor control means to read a data block from memory, to divide the data block into packets, to calculate and to append a packet checksum on the data words in each packet, and to fill the packet into the outqueue for transmission one packet at a time over the interprocessor bus.

40. The invention defined in claim 39 wherein each central processing unit control memory includes a bus receive controller means in the receiving processor module for taking a received data packet from an inqueue means, storing the data into a memory buffer, and verifying the packet checksum.

41. The invention defined in claim 40 including bus receive table means in the receiver processor module operatively associated with the bus receive controller means for reassembling the packets into blocks.

42. The invention defined in claim 41 wherein the bus receive table means and the bus receive controller means cause the microprocessor to perform interleaved assembly of data blocks from packets from a plurality of sender processor modules and error checking transparently to the process executing in the receiving processor module.

43. The invention defined in claim 29 including error check means for insuring that data on each interprocessor bus is transmitted and received correctly.

44. The invention defined in claim 43 wherein the error check means include the addition of a checkword containing a checksum of the data words as the last word in each packet for verifying correct packet transfer.

45. The invention defined in claim 44 wherein the control memory includes a sending processor module controller means for generating the checksum and a receiving processor module controller means for comparing the checksum to verify the integrity of the transmission.

46. The invention defined in claim 45 wherein the error check means include an additional check for the data block to insure that a packet is not lost.

47. The invention defined in claim 29 wherein a data block sent over an interprocessor bus in one or more packets is always preceded by a control packet and is always followed by a trailer packet.

48. The invention defined in claim 47 wherein the control packet identifies the message and indicates how much data is to be received in the message.

49. The invention defined in claim 48 wherein the trailer packet means indicate whether or not there is an error during the data transmission and also provides a means for recognizing when too little or too much data has been transmitted.

50. The invention defined in claim 29 including power failure warning signal means for supplying a power failure warning signal to the inqueue means to cause the inquene means to load, at most, one further packet of information after the receipt of a power failure warning signal.

51. The invention defined in claim 28 wherein the control memory causes the microprocessor to transfer an entire block of data in response to a single send instruction.

52. The invention defined in claim 51 wherein the bus controller means include bus control state logic means for identifying a sender-receiver pair of processor modules and for defining the time frame for a packet transfer over the interprocessor bus between the sender-receiver pair.

53. The invention defined in claim 52 wherein the inqueue means include microinterrupt means for supplying a microinterrupt to the microprocessor in the receiver processor module after receipt of each packet and wherein the microprocessor supplies a program interrupt to the central processing unit of the receiver processor module after an entire data block has been received in the receiver processor module.

54. The invention defined in claim 52 wherein the interprocessor control means of the sender and receiver processor modules include means for logically disconnecting those processor modules from the interprocessor bus at the end of each data packet transfer to permit the bus control state logic means to establish a different sender-receiver pair of processor modules and to define a time frame for making a packet transfer between the other pair of sender-receiver processor modules so that a plurality of data block transfer between different sender-receiver pairs of processor modules can be interleaved on a packet by packet basis.

55. The invention defined in claim 54 including polling means in the bus controller means for polling at least one processor module to identify a processor module which is ready to send data on the interprocessor bus.

56. The invention defined in claim 55 wherein the polling means include sender counter means for sequentially polling the processor modules in a circular sequence starting after the last identified sender processor module to thereby insure that all processor modules have an opportunity to send data over the interprocessor bus.

57. A multiprocessor system of the kind in which separate processor modules operate concurrently and cooperatively for system control and application processing, said multiprocessor system comprising,

a plurality of separate processor modules, each processor module comprising a central processing unit within the processor module, a separate main read-write word addressable memory within the processor module having sufficient capacity for the storage of system control and application programs and data, and a dedicated memory bus within the processor module connecting the separate main memory in the processor module exclusively to its associated central processing unit in the processor module for access to said programs and data without contention with central processing units in other processor modules,

interprocessor bus means separate and distinct from an input/output system and from a memory bus and interconnecting the processor modules for direct processor module to processor module signaling and data transfer,

said interprocessor bus means including,

a common shared interprocessor bus,

interprocessor control means in each processor module for connecting that processor module to the interprocessor bus, and

centralized bus controller means operatively associated with the interprocessor bus and each interprocessor control means for determining the priority of data transfers over the interprocessor bus,

said bus controller means including,

arbitration means for centrally arbitrating the priority of data transfers and

bus clock generator means for controlling the time sequence of data transfer by a processor module over the interprocessor bus,

input/output channel means in each processor module for connecting that processor module to a device controller, and

a device controller operatively associated with the input/output channel means for controlling the transfer of data between the processor module and peripheral devices,

said processor module including:

reading means for reading data from the associated main read/write memory of that processor module for transmission to the interprocessor bus, and

writing means for writing data received from the interprocessor bus into a specied location in the associated main read/write memory of that processor module, wherein said reading and writing means are effective to transfer data from the memory of a sender processor module to the memory of a receiver processor module, and wherein each processor module includes a buffer for each other processor module and also includes location pointing means for directing incoming data from an interprocessor bus to a specified location in a related buffer in the memory of a receiver processor module.

58. The invention defined in claim 57 including a plurality of peripheral devices, a plurality of ports in each device controller, and a plurality of input/output buses connecting each device controller for access by a plurality of different processor modules.

59. The invention defined in claim 58 including a second interprocessor bus and a second bus controller means operatively associated with each interprocessor control means to provide, in combination with the multiple input/output buses, multiple separate paths for connecting each device controller with each processor module so that in the event of a failure of one path any processor module can still communicate with any device controller over another path.

60. The invention defined in claim 58 including interface common logic means operatively associated with each device controller for insuring that only one port of the device controller is logically connected to the device controller at a time.

61. The invention defined in claim 58 including stress responsive buffer means in each device controller for permitting data to be transmitted over each input/output bus between a port and an input/output channel means in a series of bursts.

62. The invention defined in claim 61 including jumpering means for varying the stress at which a buffer reconnect request is made to the input/output channel means.

63. The invention defined in claim 58 including input/output control table means having an entry for every device connected to the input/output channel means and wherein each entry contains a buffer address in memory and a count word for the number of bytes to be transferred in an input/output data transfer.

64. The invention defined in claim 63 wherein the input/output control table entries are located in the memory to protect that memory from a failing device controller or input/output bus.

65. The invention defined in claim 64 including another copy of the input/output control table located in each processor module that is operatively associated with the same device controller so that the failure of one processor module does not cause the loss of the device controller to the system.

66. A multiprocessor system of the kind in which separate processor modules operate concurrently and cooperatively for system control and application processing, said multiprocessor system comprising,

a plurality of separate processor modules, each processor module comprising a central processing unit within the processor module, a separate main read-write word addressable memory within the processor module having sufficient capacity for the storage of system control and application programs and data, a dedicated memory bus within the processor module connecting the separate main memory in the processor module exclusively to its associated central processing unit in the processor module for access to said programs and data without contention with central processing units in other processor modules, and an input/output channel,

interprocessor bus means separate and distinct from an input/output system and from a memory bus and interconnecting the processor modules for direct processor module to processor module signaling and data transfer,

said interprocessor bus means including,

a common shared interprocessor bus,

interprocessor control means in each processor module for connecting that processor module to the interprocessor bus, and

centralized bus controller means operatively associated with the interprocessor bus and each interprocessor control means for determining the priority of data transfers over the interprocessor bus and for controlling transmissions over the interprocessor bus,

said bus controller means including,

arbitration means for centrally arbitrating the priority of data transfers and

bus clock generator means for controlling the time sequence of data transfer by a processor module over the interprocessor bus,

a plurality of peripheral devices,

a plurality of device controllers for controlling the transfer of data between the processor modules and the peripheral devices,

multiple ports in each device controller,

multiple associated input/output buses connecting each device controller for access by input/output channels of different processor modules,

multiple separate power supplies operatively associated with each device controller for powering each device controller, and

connecting means for powering a device controller from the remaining associated power supplies in the event of a failure or turn off of another power supply for that device controller whereby the power supply operatively associated with a processor module can be powered down while the rest of the multiprocessor system is on-line and functional.

67. The invention defined in claim 66 wherein the connecting means include diode switching means for providing logic power from the associated power supplies when the associated power supplies are operative and for providing logic power from the remaining power supplies in the event one power supply fails or is turned off and for providing a changeover in the event of failure or turn off which is smooth and without interruption or pulsation in the logic power supplied.

68. The invention defined in claim 66 including power-on circuit means in each processor module and device controller for detecting when the logic power supply to the processor module or device controller is below specifications for correct logic operation and for resetting the processor module or device controller to prevent all activity at its interfaces and to hold it in a state that is controlled when its logic power is affected by loss of line power, turn off, power supply failure, or power distribution failure.

69. The invention defined in claim 68 wherein the power-on circuit means also release the processor module or device controller and return it to operation after the power is restored and logic power is supplied at the proper specification to it.

70. The invention defined in claim 69 including transceiver means in the processor modules and device controllers which coact with the power-on circuit means to control the output of the transceiver means at the processor module and device controller interfaces as power comes up or goes down rather than leaving the output of the transceiver means undefined when power drops below the level required for correct operation.

71. The invention defined in claim 70 including an enable line connected to the transceivers at the device controller interfaces and wherein the power-on circuit coacts with the enable line to keep the transceivers in the off state until the power has risen to the level where the off state will be maintained by correct functioning of the logic elements in the device controller.

72. The invention defined in claim 66 wherein the interprocessor control means are constructed to enable additional processor modules to be added to the system without the need to modify either the existing hardware or software and wherein the input/output channel means are constructed to enable additional device controllers and peripheral devices to be added to the system whereby the system can be expanded both in processing capacity with additional processor modules and in input/output capacity with additional input/output channels, device controllers, and peripheral devices.

73. A processor module for a multiprocessor system of the kind in which separate processor modules communicate for signaling and data transfer over an interprocessor bus having a bus clock provided by an associated bus controller, said processor module comprising,

a central processing unit having a processor clock which is independent of the bus clock,

a memory containing instructions and data for that module, and

interprocessor control means for connecting the processor module to the interprocessor bus,

said interprocessor control means including buffer means for storing data transferred between the interprocessor bus and the processor memory, and buffer control means for controlling filling and emptying of the buffer means,

said buffer control means having first logic means for operating in synchronism with the bus clock and second logic means for operating in synchronism with the processor clock, and

first interlock means associated with the first logic means for receiving the state of the second logic means and for responding to enable certain state changes of said first logic means, and

second interlock means associated with the second logic means for receiving the state of the first logic means and for responding to enable certain state changes of said second logic means, so that the interprocessor bus operates in synchronism with the bus clock and the central processing unit and memory of the processor module operate in synchronism with the processor clock without loss or duplication of the data being transferred and without loss of information about the state of the transfer sequence.

74. A method of interconnecting a plurality of processor modules by a multimodule communication path for insuring that one processor module ready to send can establish a sender-receiver path with another one of the processor modules ready to receive and can transmit data on a time synchronized basis without reliance on handshake acknowledgement between the pair of processor modules for each data word transmitted, said method comprising,

connecting all of the processor modules with a multimodule communication path,

controlling the multimodule communication path by a bus controller,

operating the bus as a synchronous bus with a portion of each processor module synchronized with a bus clock provided by the bus controller,

transmitting data in the form of multiword packets,

polling the processor modules on a demand basis to identify a processor module which is ready to send a packet on the bus and to identify an intended receiver processor module,

establishing a sender-receiver path by interrogating the readiness of the receiver processor module to receive and then allocating a bus time for the sending of a packet which time corresponds exactly to the length of the packet itself, and

then releasing the bus and returning the bus controller to its idle state, or if further demanded again to its polling state.

75. The invention defined in claim 74 including specifying the intended receiver processor module and detecting the readiness of the receiver processor module to receive before sending the packet.

76. The invention defined in claim 75 wherein each processor module has an individual select line extending to that processor module from the bus controller and including producing select signals over the individual select lines to set up data transfers over the bus to establish the sender-receiver pair of processor modules and to define the time frame for the transfer of information over the bus between the sender-receiver and to exclude other processor modules from transmitting on the bus simultaneously with the selected sender processor module.

77. A multiprocessor system of the kind in which separate processor modules operate concurrently and cooperatively for system control and application processing, said multiprocessor system comprising,

a plurality of separate processor modules,

interprocessor bus means including an interprocessor bus connecting each processor module for direct processor module to processor module signaling and data transfer and wherein the interprocessor bus means are separate and distinct from an input/output system and from a memory system bus,

each processor module having a central processing unit, a memory for that module, and interprocessor control means for connecting the processor module to the interprocessor bus for signaling and data transfer with another processor module,

bus controller means operatively associated with the interprocessor bus and each interprocessor control means for determining the priority of data transfers between any two processor modules over the interprocessor bus and for controlling transmissions over the interprocessor bus,

each processor module including input/output channel means for transferring data between the processor module and one or more device controllers,

a plurality of peripheral devices,

a plurality of device controllers for controlling the transfer of data between the processor modules and the peripheral devices,

a plurality of ports in each device controller,

a plurality of input/output buses connecting each device controller for access by a plurality of different processor modules and

wherein the input/output channel means, input/output buses and device controllers are operatively connected to provide, directly or in combination with the interprocessor bus means and a processor module, a path for data transfer between each processor module and any device controller.

78. The invention defined in claim 77 including a second interprocessor bus interconnecting each processor module and providing an alternate path for direct processor module to processor module signaling and data transfer.

79. The invention defined in claim 78 including distributed power supply means for insuring continuous operation of the remainder of the multiprocessor system in the event of a failure of a single power supply to a part of the multiprocessor system, said distributed power supply means including a separate power supply for each processor module and connecting means for connecting a plurality of the separate power supplies to each device controller and effective to supply power to each device controller from the remaining separate power supplies in the event of the failure of one of the power supplies connected to any device controller whereby the power supply operatively associated with a processor module can be powered down so that on-line maintenance can be performed in a power-off condition on that processor module while the rest of the multiprocessor system is on-line and functional.

80. A multiprocessor system constructed for continued system operation in the event of a failure of any single component in the system and comprising,

a plurality of separate processor modules, each module including a central processing unit and a local memory,

interprocessor bus means including a plurality of interprocessor buses for signaling and data transfer between separate processor modules,

at least two interprocessor buses being connected to each processor module,

each interprocessor bus being separate and distinct from an input/output system and from a memory bus,

a plurality of device controllers each adapted for connection to at least one peripheral device, and

a plurality of input/output buses each connected between a related processor module and one or more device controllers,

each device controller being accessible to at least two processor modules by means of associated input/output buses

each of said processor modules including:

reading means for reading data from the associated local memory of that processor module for transmission to an interprocessor bus, and

writing means for writing data received from an interprocessor bus into a specified location in the associated local memory of that processor module, wherein said reading and writing means are effective to transfer data from the memory of a sender processor module to the memory of a receiver processor module, and wherein each processor module includes buffer for each other processor module and also includes location pointing means for directing incoming data from an interprocessor bus to a specified location in a related buffer in the memory of a receiver processor module.