Synchronized communication between multi-processor clusters of multi-cluster computer systems

• US 7,159,137 B2
• Filed: 08/05/2003
• Issued: 01/02/2007
• Est. Priority Date: 08/05/2003
• Status: Expired due to Fees

• Current Assignee: Sanmina-SCI Corporation
• Original Assignee: Newisys, Inc. (Sanmina-SCI Corporation)
• Inventors: Glasco, David B., Zeitler, Carl, Prasadh, Guru, Kota, Rajesh, Nemawarkar, Shashank
• Primary Examiner(s): Browne, Lynne H.
• Assistant Examiner(s): Sugent, James F.
• Application Number:
  US10/635,705
• Publication Number:
  US 20050034007A1
• Time in Patent Office: 1,246 Days
• Field of Search: 713375-601, 709/222, 709/248, 714/4, 714/12, 714/700, 714/707, 714/775, 714/798
• US Class Current: 713/503
• CPC Class Codes: G06F 1/10 Distribution of clock signals , e.g. skew

Abstract

Improved techniques are provided for detecting and correcting errors and skew in inter-cluster communications within computer systems having a plurality of multi-processor clusters. The local nodes of each cluster include a plurality of processors and an interconnection controller. Intra-cluster links are formed between the local nodes, including the interconnection controller, within a cluster. Inter-cluster links are formed between interconnection controllers of different clusters. Intra-cluster packets may be serialized and encapsulated as inter-cluster packets for transmission on inter-cluster links, preferably with link-layer encapsulation. Each inter-cluster packet may include a sequence identifier and error information computed for that packet. Clock data may be embedded in symbols sent on each bit lane of the inter-cluster links. Copies of transmitted inter-cluster packets may be stored until an acknowledgement is received. The use of inter-cluster packets on an inter-cluster link is preferably transparent to other links and to the protocol layer.

Claims

1. A computer system, comprising:

a first cluster including a first plurality of processors and a first interconnection controller, the first plurality of processors and the first interconnection controller interconnected by first point-to-point intra-cluster links; and

a second cluster including a second plurality of processors and a second interconnection controller, the second plurality of processors and the second interconnection controller interconnected by second point-to-point intra-cluster links, the first interconnection controller coupled to the second interconnection controller by point-to-point inter-cluster links,

the first and second interconnection controllers configured to;

perform an initialization sequence that establishes a characteristic skew pattern between data lanes of the point-to-point inter-cluster links;

encode clock data in each symbol transmitted on the point-to-point inter-cluster links;

recover clock data from each symbol received on the point-to-point inter-cluster links; and

apply the characteristic skew pattern to correct for skew between data lanes of the point-to-point inter-cluster links.

2. The computer system of claim 1, wherein the encoding step comprises encoding 8-bit symbols as 10-bit symbols.

3. The computer system of claim 1, wherein the encoding step comprises encoding 4-bit symbols as 5-bit symbols.

4. The computer system of claim 1, wherein the initialization sequence comprises the use of one or more training sequences having known structures and lengths.

5. The computer system of claim 1, wherein the initialization sequence comprises establishing a phase lock loop for incoming frequency-encoded data.

6. An interconnection controller, comprising:

an intra-cluster interface configured for coupling with intra-cluster links to a plurality of local processors arranged in a point-to-point architecture in a local cluster;

an inter-cluster interface configured for coupling with an inter-cluster link to a non-local interconnection controller in a non-local cluster; and

a transceiver configured to;

perform an initialization sequence with the non-local interconnection controller that establishes a characteristic skew pattern between data lanes of the point-to-point inter-cluster links;

recover clock data from symbols received on the point-to-point inter-cluster links; and

apply the characteristic skew pattern to correct for skew between data lanes of the point-to-point inter-cluster links.

7. The interconnection controller of claim 6, further configured to encode clock data in symbols transmitted on the point-to-point inter-cluster links.

8. The interconnection controller of claim 6, further configured to forward symbols to the intra-cluster interface for transmission via the intra-cluster links to at least one of the plurality of local processors.

9. An integrated circuit comprising the interconnection controller of claim 6.

10. A set of semiconductor processing masks configured for fabricating the integrated circuit claim 9.

11. At least one computer-readable medium having data structures stored therein representative of the interconnection controller of claim 6.

12. The integrated circuit of claim 9, wherein the integrated circuit comprises an application-specific integrated circuit.

13. The at least one computer-readable medium of claim 11, wherein the data structures comprise a simulatable representation of the interconnection controller.

14. The at least one computer-readable medium of claim 11, wherein the data structures comprise a code description of the interconnection controller.

15. The at least one computer-readable medium of claim 13, wherein the simulatable representation comprises a netlist.

16. The at least one computer-readable medium of claim 14, wherein the code description corresponds to a hardware description language.