Multipurpose scalable server communication link

• US 20060129709A1
• Filed: 12/09/2004
• Published: 06/15/2006
• Est. Priority Date: 12/09/2004
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Bandholz, Justin P., Borkenhagen, John M., Heinzmann, Andrew S., Lyon, Terry L.
• Application Number:
  US11/008,811
• Publication Number:
  US 20060129709A1
• Time in Patent Office: Days
• US Class Current: 710/30
• CPC Class Codes: G06F 9/52 Program synchronisation; Mutual exclusion, e.g. by means of semaphores

Abstract

Methods and apparatus that may be utilized to improve the scalability of multi-processor systems are provided. Data packets constructed in accordance with a defined coherence protocol may be encapsulated in standard I/O packets. As a result, the same interconnect fabric may be used to route coherent data traffic and I/O data traffic.

Claims

1. A method of maintaining memory coherency in a multi-node system, with each node comprising one or more processors with access to a shared memory pool, comprising:

encapsulating coherency control information in an input/output (I/O) packet in accordance with an I/O protocol, the data having been received from a processor at a first node; and

transmitting the I/O packet to a second node via a switch mechanism compatible with the I/O protocol.

2. The method of claim 1, wherein the I/O protocol comprises at least one of:

Infiniband, Gigabit Ethernet, FibreChannel, and PCI-Express protocols.

3. The method of claim 1, wherein encapsulating the coherency control information in the I/O packet comprises generating header information for the I/O packet indicating one or more nodes that are to receive the I/O packet.

4. The method of claim 1, wherein transmitting the I/O packet to a second node comprises:

selecting, from a plurality of I/O links, an I/O link having the least amount of traffic; and

transmitting the I/O packet to the second node via the selected link.

5. The method of claim 1, wherein transmitting the I/O packet to a second node comprises generating a control signal having a first state to select, as an input to a transmit link, the I/O packet with the encapsulated coherency control information.

6. The method of claim 5, further comprising generating a control signal having a second state to select, as an input to the transmit link, an I/O packet to be transmitted to one or more I/O boards.

7. The method of claim 1, further comprising:

receiving an I/O packet via the switch mechanism;

determining whether the received I/O packet contains coherency control information; and

if so, extracting the coherency control information and forwarding the coherency data on to one or more processors at the first node.

8. The method of claim 1, wherein the first and second nodes are contained in separate clusters of nodes coupled to a network.

9. The method of claim 8, wherein the switching mechanism comprises a network adapter.

10. A method of maintaining memory coherency in a multi-node system, with each node comprising one or more processors with access to a shared memory pool, comprising:

receiving, by a first one of the nodes, an input/output (I/O) packet from a second one of the nodes, the I/O packet in accordance with an I/O protocol and containing coherency control information encapsulated therein;

extracting the coherency control information from the I/O packet; and

forwarding the coherency control information on to one or more processors on the first node.

11. The method of claim 10, further comprising, determining whether the I/O packet contains coherency control information by examining header information contained in the I/O packet.

12. The method of claim 10, wherein the first and second nodes are contained in separate clusters of nodes coupled to a network.

13. The method of claim 12, wherein the switching mechanism comprises a network adapter.

14. A communications controller, comprising:

at least a first input/output (I/O) link comprising a transmitter circuit and a receiver circuit;

at least a first coherency protocol engine configured to encapsulate coherency control information in an I/O packet and transmit the I/O packet to a second node via the transmitter circuit, wherein the coherency control information is received from a processor on a first node; and

at least a first packet router configured to receive an I/O packet via the receiver circuit, extract coherency control information encapsulated in the received I/O packet, and forward the extracted coherency control information to the coherency protocol engine.

15. The controller of claim 14, further comprising:

an I/O protocol engine configured to transmit I/O packets without coherency control information to one or more I/O nodes via the transmitter circuit; and

a transmit controller configured to select, as input to the transmitter circuit, I/O packets from the I/O protocol engine or I/O packets with encapsulated coherency control information from the coherency protocol engine.

16. The controller of claim 14, further comprising:

at least a second input/output (I/O) link comprising a transmitter circuit and a receiver circuit;

at least a second coherency protocol engine configured to encapsulate coherency control information from a processor on a first node in an I/O packet and transmit the I/O packet to a second node via the transmitter circuit of the second I/O link; and

at least a second packet router configured to receive an I/O packet via the receiver circuit of the second I/O link, extract coherency control information encapsulated in the received I/O packet, and forward the extracted coherency control information to the first or second coherency protocol engine.

17. The controller of claim 16, wherein at least two coherency protocol engines are coupled with a common transmitter circuit.

18. The controller of claim 14, further comprising:

at least one coherency link for transmitting coherency control information to at least the second node; and

a switching mechanism for routing coherency control information from the coherency protocol engine to either the coherency link or to a packetizer configured to encapsulate the coherency control information in an I/O message, depending on the state of one or more control signals.

19. A server system, comprising:

one or more input/output (I/O) boards, each comprising an I/O controller and one or more I/O devices;

a plurality of processor boards, each comprising one or more processors;

an I/O switching mechanism for exchanging I/O packets, in accordance with a defined protocol, between the processor boards and the I/O boards; and

for each processor board, a communications controller configured to exchange I/O packets with I/O boards and other processor boards via the switching mechanism, wherein the controller is configured to encapsulate coherency control information in I/O messages to be transmitted to other processor boards.

20. The system of claim 19, wherein:

the communications controller is configured to generate header information in I/O messages encapsulating coherency control information; and

the I/O switching mechanism is configured to examine the header information and, in response, route the I/O messages encapsulating the coherency control information to one or more processor boards.

21. The system of claim 19, wherein the plurality of processor boards comprises processor boards contained in at least a first and second cluster separated by a network connection.

22. The system of claim 21, wherein each cluster has an I/O switching mechanism allowing the exchange of I/O messages encapsulating coherency control information via the network connection.

23. The system of claim 19, wherein the I/O switching mechanism is integrated into a backplane coupled to the I/O and processor boards.

24. The system of claim 19, wherein the communications controller of one or more of the processor boards is capable of being configured to exchange coherency control information via a dedicated communications link rather than via I/O messages encapsulating the coherency control information.

25. The system of claim 19, wherein the communications controller, for at least one of the processor boards, is integrated on the processor board.