Communication between and within multi-processor clusters of multi-cluster computer systems

• US 7,395,347 B2
• Filed: 08/05/2003
• Issued: 07/01/2008
• Est. Priority Date: 08/05/2003
• Status: Expired due to Fees

• Current Assignee: Sanmina-SCI Corporation
• Original Assignee: Newisys, Inc. (Sanmina-SCI Corporation)
• Inventors: Glasco, David B., Zeitler, Carl, Prasadh, Guru, Kota, Rajesh, Nemawarkar, Shashank
• Primary Examiner(s): Nguyen, Phuoc H
• Application Number:
  US10/635,744
• Publication Number:
  US 20050034033A1
• Time in Patent Office: 1,792 Days
• Field of Search: 709/232, 709/234, 709/236, 709/238, 709/249
• US Class Current: 709/232
• CPC Class Codes: H04L 1/0083 Formatting with frames or packets; Protocol or part of protocol for error control
• CPC Class Codes: H04L 1/1664 the supervisory signal being transmitted together with payload signals; piggybacking

Abstract

Improved techniques are provided for detecting and correcting errors and skew in inter-cluster communications within computer systems having a plurality of multi-processor clusters. The local nodes of each cluster include a plurality of processors and an interconnection controller. Intra-cluster links are formed between the local nodes, including the interconnection controller, within a cluster. Inter-cluster links are formed between interconnection controllers of different clusters. Intra-cluster packets may be serialized and encapsulated as inter-cluster packets for transmission on inter-cluster links, preferably with link-layer encapsulation. Each inter-cluster packet may include a sequence identifier and error information computed for that packet. Clock data may be embedded in symbols sent on each bit lane of the inter-cluster links. Copies of transmitted inter-cluster packets may be stored until an acknowledgement is received. The use of inter-cluster packets on an inter-cluster link is preferably transparent to other links and to the protocol layer.

Claims

1. An interconnection controller, comprising:

an intra-cluster interface configured for coupling with intra-cluster links to a plurality of local processors arranged in a point-to-point architecture in a local cluster;

an inter-cluster interface configured for coupling with an inter-cluster link to a non-local interconnection controller in a non-local cluster;

a transceiver configured to receive an intra-cluster packet from the local processors via the intra-cluster links of the coherent interface and encapsulate the intra-cluster packet as a high-speed link packet by adding a sequence identifier and a packet type identifier to the intra-cluster packet; and

a serializer/deserializer configured to serialize the high-speed link packet and forward the serialized high-speed link packet to the inter-cluster interface for transmission to the non-local interconnection controller via an inter-cluster link;

wherein the interconnection controller is configured to compute a cyclic redundancy code value based only on the high-speed link packet; and

wherein the serializer/deserializer is further configured to encode the cyclic redundancy code value in the high-speed link packet.

2. The interconnection controller of claim 1, wherein the inter-cluster interface is further configured to receive high-speed link packets from the non-local interconnection controller, wherein the serializer/deserializer is further configured to deserialize the encoded, serialized packets and wherein the transceiver is further configured to perform a cyclic redundancy code check on the deserialized packets.

3. The interconnection controller of claim 1, wherein the sequence identifier and the packet type identifier are encoded in a header of the high-speed link packet.

4. The interconnection controller of claim 1, wherein the sequence identifier and the packet type identifier are encoded in a portion of the high-speed link packet reserved for link-layer encoding.

5. The interconnection controller of claim 1, wherein the transceiver is further configured to encode a packet length field in a header of the high-speed link packet.

6. The interconnection controller of claim 1, wherein the transceiver is further configured to form high-speed link packets that are transparent to a protocol layer.

7. An integrated circuit comprising the interconnection controller of claim 1.

8. The integrated circuit of claim 7, wherein the integrated circuit comprises an application-specific integrated circuit.

9. A set of semiconductor processing masks representative of at least a portion of the interconnection controller of claim 1.

10. At least one computer-readable medium having data structures stored therein representative of the interconnection controller of claim 1.

11. The at least one computer-readable medium of claim 10, wherein the data structures comprise a simulatable representation of the interconnection controller.

12. The at least one computer-readable medium of claim 11, wherein the simulatable representation comprises a netlist.

13. The at least one computer-readable medium of claim 10, wherein the data structures comprise a code description of the interconnection controller.

14. The at least one computer-readable medium of claim 13, wherein the code description corresponds to a hardware description language.

15. An interconnection controller, comprising:

an intra-cluster interface configured for coupling with intra-cluster links to a plurality of local processors arranged in a point-to-point architecture in a local cluster;

an inter-cluster interface configured for coupling with an inter-cluster link to a non-local interconnection controller in a non-local cluster;

a transceiver configured to receive an intra-cluster packet from the local processors via the intra-cluster links of the coherent interface and encapsulate the intra-cluster packet as a high-speed link packet by adding a sequence identifier and a packet type identifier to the intra-cluster packet; and

a serializer/deserializer configured to serialize the high-speed link packet and forward the serialized high-speed link racket to the inter-cluster interface for transmission to the non-local interconnection controller via an inter-cluster link, wherein the transceiver is further configured to encapsulate both an intra-cluster packet and an ACK packet in a single high-speed link packet.

16. An interconnection controller, comprising:

an intra-cluster interface configured for coupling with intra-cluster links to a plurality of local processors arranged in a point-to-point architecture in a local cluster;

an inter-cluster interface configured for coupling with an inter-cluster link to a non-local interconnection controller in a non-local cluster;

a transceiver configured to receive an intra-cluster packet from the local processors via the intra-cluster links of the coherent interface and encapsulate the intra-cluster packet as a high-speed link packet by adding a sequence identifier and a packet type identifier to the intra-cluster packet; and

a serializer/deserializer configured to serialize the high-speed link packet and forward the serialized high-speed link packet to the inter-cluster interface for transmission to the non-local interconnection controller via an inter-cluster link, wherein the transceiver is further configured to form high-speed link packets having lengths that are integral multiples of 96 bits.

17. A computer system comprising a plurality of processor clusters interconnected by a plurality of point-to-point inter-cluster links, each processor cluster comprising nodes including a plurality of local processors and an interconnection controller interconnected by a plurality of point-to-point intra-cluster links, communications within a cluster being made via an intra-cluster protocol that uses intra-cluster packets, wherein the interconnection controller in each cluster is operable to map locally-generated communications directed to others of the clusters to the point-to-point inter-cluster links and to map remotely-generated communications directed to the local nodes to the point-to-point intra-cluster links, communications between clusters being made via an inter-cluster protocol that uses inter-cluster packets, an inter-cluster packet encapsulating at least one intra-cluster packet, each interconnection controller configured to:

receive an intra-cluster packet from the local processors via the intra-cluster links of the coherent interface;

encapsulate the intra-cluster packet as a high-speed link packet by adding a sequence identifier and a packet type identifier to the intra-cluster packet;

compute a cyclic redundancy code value based only on the high-speed link packet;

encode the cyclic redundancy code value in the high-speed link packet;

serialize the high-speed link packet; and

forward the serialized high-speed link packet to the inter-cluster interface for transmission to another processor cluster via an inter-cluster lint.

18. Integrated circuits comprising the interconnection controllers of claim 17.

19. A set of semiconductor processing masks representative of at least a portion of the interconnection controllers of claim 17.

20. At least one computer-readable medium having data structures stored therein representative of the interconnection controllers of claim 17.