Data processing system, method and interconnect fabric supporting multiple planes of processing nodes

• US 7,818,388 B2
• Filed: 10/07/2005
• Issued: 10/19/2010
• Est. Priority Date: 10/07/2005
• Status: Expired due to Fees

• Current Assignee: LinkedIn Corporation (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Arimilli, Ravi K., Starke, William J., Goodman, Benjiman L., Reddy, Praveen S., Guthrie, Guy L.
• Primary Examiner(s): ISMAIL, SHAWKI SAIF
• Application Number:
  US11/245,887
• Publication Number:
  US 20070081516A1
• Time in Patent Office: 1,838 Days
• Field of Search: 709/208
• US Class Current: 709/208
• CPC Class Codes: G06F 15/16 Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs coordinating program control therefor G06F9/52; in regulating and control system G05B

Abstract

A data processing system includes a first plane including a first plurality of processing nodes, each including multiple processing units, and a second plane including a second plurality of processing nodes, each including multiple processing units. The data processing system also includes a plurality of point-to-point first tier links. Each of the first plurality and second plurality of processing nodes includes one or more first tier links among the plurality of first tier links, where the first tier link(s) within each processing node connect a pair of processing units in the same processing node for communication. The data processing system further includes a plurality of point-to-point second tier links. At least a first of the plurality of second tier links connects processing units in different ones of the first plurality of processing nodes, at least a second of the plurality of second tier links connects processing units in different ones of the second plurality of processing nodes, and at least a third of the plurality of second tier links connects a processing unit in the first plane to a processing unit in the second plane.

Claims

1. A data processing system, comprising:

a first plane including a first plurality of processing nodes each including multiple processing units and a second plane including a second plurality of processing nodes each including multiple processing units;

a plurality of point-to-point type first tier links, wherein each of said first plurality and second plurality of processing nodes includes one or more of first tier links, and wherein a first tier link within a processing node connects solely a pair of processing units in a same processing node for communication; and

a plurality of point-to-point type second tier links, wherein;

at least a first of said plurality of second tier links connects solely two processing units disposed in different ones of said first plurality of processing nodes;

at least a second of said plurality of second tier links connects solely two processing units disposed in different ones of said second plurality of processing nodes; and

at least a third of said plurality of second tier links solely connects a processing unit in said first plane to a processing unit in said second plane;

wherein;

said processing units include interconnect logic that processes a plurality of concurrently pending broadcast operations of differing broadcast scope, wherein at least a first of said plurality of concurrently pending broadcast operations has a first scope including processing nodes in said first and second planes and a second of said plurality of concurrently pending broadcast operations has a second scope restricted to at least one processing node in a single one of said first and second planes;

said first scope comprises a system-wide scope including all processing units in said data processing system;

said interconnect logic places a scope indicator indicating a broadcast scope in at least a request of each operation among said plurality of concurrently pending broadcast operations;

for an operation of system-wide scope, a native local master processing unit in said first plane distributes said operation to each processing unit in said first plane via particular ones of said first and second tier links, and distributes said operation, via a second tier link, to a foreign local master processing unit in said second plane, wherein said foreign local master processing unit distributes said operation to each other processing unit in said second plane via others of said first and second tier links;

said foreign local master processing unit transmits a collected partial response representing all partial responses of processing units in said second plane to a native local hub processing unit in said first plane;

a native local hub processing unit in said first plane transmits a collected partial response representing all partial responses of processing units in said first plane to said foreign local master processing unit in said second plane; and

said foreign local master processing unit determines a combined response representing a system-wide response to said operation based at least in part upon said collected partial response of said first plane.

2. A data processing system, comprising:

a first plane including a first plurality of processing nodes each including multiple processing units and a second plane including a second plurality of processing nodes each including multiple processing units;

a plurality of point-to-point type first tier links, wherein each of said first plurality and second plurality of processing nodes includes one or more of first tier links, and wherein a first tier link within a processing node connects solely a pair of processing units in a same processing node for communication; and

a plurality of point-to-point type second tier links, wherein;

at least a first of said plurality of second tier links connects solely two processing units disposed in different ones of said first plurality of processing nodes;

at least a second of said plurality of second tier links connects solely two processing units disposed in different ones of said second plurality of processing nodes; and

at least a third of said plurality of second tier links solely connects a processing unit in said first plane to a processing unit in said second plane;

wherein;

at least some of the processing units in the data processing system have associated cache memory;

the data processing system is cache coherent; and

wherein for an operation of system-wide scope, a native local master processing unit in said first plane distributes said operation to each processing unit in said first plane via particular ones of said first and second tier links, and distributes said operation, via a second tier link, to a foreign local master processing unit in said second plane, wherein said foreign local master processing unit distributes said operation to each other processing unit in said second plane via others of said first and second tier links.

3. The data processing system of claim 2, wherein:

said processing units include interconnect logic that processes a plurality of concurrently pending broadcast operations of differing broadcast scope, wherein at least a first of said plurality of concurrently pending broadcast operations has a first scope including processing nodes in said first and second planes and a second of said plurality of concurrently pending broadcast operations has a second scope restricted to at least one processing node in a single one of said first and second planes.

4. The data processing system of claim 3, wherein said first scope comprises a system-wide scope including all processing units in said data processing system.

5. The data processing system of claim 3, wherein said interconnect logic places a scope indicator indicating a broadcast scope in at least a request of each operation among said plurality of concurrently pending broadcast operations.

6. The data processing system of claim 2, wherein said at least a third of said plurality of second tier links includes multiple second tier links each connecting a respective one of said plurality of first processing nodes in said first plane to a single one of said plurality of second processing nodes in said second plane.

7. The data processing system of claim 2, wherein said foreign local master processing unit transmits a collected partial response representing all partial responses of processing units in said second plane to a native local hub processing unit in said first plane.

8. The data processing system of claim 7, wherein a native local hub processing unit in said first plane transmits a collected partial response representing all partial responses of processing units in said first plane to said foreign local master processing unit in said second plane.

9. The data processing system of claim 8, wherein said foreign local master processing unit determines a combined response representing a system-wide coherence response to said operation based at least in part upon said collected partial response of said first plane.

10. A method of data processing in a data processing system including a first plane containing a first plurality of processing nodes each including multiple processing units and a second plane containing a second plurality of processing nodes each including multiple processing units, said method comprising:

communicating operations between processing units within a same processing node via a plurality of point-to-point first tier links, wherein each of said first plurality and second plurality of processing nodes includes one or more first tier links among said plurality of first tier links, and wherein each first tier link connects solely a pair of processing units in a same processing node for communication; and

communicating operations between processing units in different processing nodes via a plurality of point-to-point second tier links, wherein;

at least a first of said plurality of second tier links connects solely two processing units in different ones of said first plurality of processing nodes;

at least a second of said plurality of second tier links connects solely processing units in different ones of said second plurality of processing nodes; and

at least a third of said plurality of second tier links connects solely a processing unit in said first plane to a processing unit in said second plane;

wherein;

at least some of the processing units in the data processing system have associated cache memory;

the data processing system is cache coherent; and

said steps of communicating operations between processing units within a same processing node and communicating operations between processing units in different processing nodes comprise;

transmitting an operation of system-wide scope, wherein said transmitting includes;

a native local master processing unit in said first plane distributing said operation to each processing unit in said first plane via particular ones of said first and second tier links; and

distributing said operation, via a second tier link, to a foreign local master processing unit in said second plane, wherein said foreign local master processing unit distributes said operation to each other processing unit in said second plane via others of said first and second tier links.

11. The method of claim 10, wherein:

interconnect logic in said processing units processing a plurality of concurrently pending broadcast operations of differing broadcast scope, wherein at least a first of said plurality of concurrently pending broadcast operations has a first scope including processing nodes in said first and second planes and a second of said plurality of concurrently pending broadcast operations has a second scope restricted to at least one processing node in a single one of said first and second planes.

12. The method of claim 11, wherein said first scope comprises a system-wide scope including all processing units in said data processing system.

13. The method of claim 11, and further comprising said interconnect logic placing a scope indicator indicating a broadcast scope in at least a request of each operation among said plurality of concurrently pending broadcast operations.

14. The method of claim 10, wherein:

said at least a third of said plurality of second tier links includes multiple second tier links each connecting a respective one of said plurality of first processing nodes in said first plane to a single one of said plurality of second processing nodes in said second plane; and

said step of communicating operations between processing units in different processing nodes via a plurality of point-to-point second tier links comprises communicating operations between said first and second planes via said multiple second tier links.

15. The method of claim 10, and further comprising said foreign local master processing unit transmitting a collected partial response representing all partial responses of processing units in said second plane to a native local hub processing unit in said first plane.

16. The method of claim 15, and further comprising a native local hub processing unit in said first plane transmitting a collected partial response representing all partial responses of processing units in said first plane to said foreign local master processing unit in said second plane.

17. The method of claim 16, and further comprising said foreign local master processing unit determining a combined response representing a system-wide coherence response to said operation based at least in part upon said collected partial response of said first plane.