Synchronized communication in a data processing system

• US 8,103,791 B2
• Filed: 08/20/2008
• Issued: 01/24/2012
• Est. Priority Date: 02/10/2005
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Goodman, Benjiman L., Guthrie, Guy L., Reddy, Praveen S., Starke, William J., Stuecheli, Jeffrey A.
• Primary Examiner(s): Sheikh, Ayaz
• Assistant Examiner(s): VU, HOANG-CHUONG Q
• Application Number:
  US12/195,130
• Publication Number:
  US 20080307137A1
• Time in Patent Office: 1,252 Days
• Field of Search: 370/241, 370/242, 710/240, 710/241, 710/52, 710/110, 710/107, 710/100, 710/244, 710/316, 709/213, 709/248, 709238-244, 709/208, 712/32, 713/375, 713/400, 713/401
• US Class Current: 709/238
• CPC Class Codes: G06F 15/16 Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs coordinating program control therefor G06F9/52; in regulating and control system G05B

Abstract

A data processing system includes a plurality of processing units, including at least a local master and a local hub, which are coupled for communication via a communication link. The local master includes a master capable of initiating an operation, a snooper capable of receiving an operation, and interconnect logic coupled to a communication link coupling the local master to the local hub. The interconnect logic includes request logic that synchronizes internal transmission of a request of the master to the snooper with transmission, via the communication link, of the request to the local hub.

Claims

1. A data processing system, comprising:

a plurality of processing units coupled for communication, said plurality of processing units including at least a local master processing unit and multiple local hub processing units each coupled to the local master processing unit by a respective one of a plurality of communication links, wherein said local master processing unit includes;

a master that initiates memory access requests;

a snooper that receives memory access requests; and

interconnect logic coupled to the plurality of communication links, wherein said interconnect logic includes request logic that synchronizes internal delivery of a memory access request of said master to said snooper with delivery, via the plurality of communication links, of the memory access request to snoopers in the plurality of local hub processing units, wherein the request logic synchronizes the internal delivery by applying a greater delay to the memory access request to be delivered to the snooper than to that to be delivered to the plurality of local hub processing units, wherein the request logic applies the delay to the memory access request to be delivered to the snooper after transmission by the request logic of the memory access request to the plurality of local hub processing units and prior to delivery of the memory access request to the snooper.

2. The data processing system of claim 1, wherein said data processing system further comprises:

a remote hub processing unit coupled to a particular local hub processing unit among the multiple local hub processing units by a communication link, wherein said particular local hub processing unit includes interconnect logic that transmits said memory access request to said remote hub processing unit in response to receipt of the memory access request from the local master processing unit.

3. The data processing system of claim 1, wherein the local master processing unit includes request logic including a launch buffer comprising:

a first ordered data structure that orders requests generated by said master;

multiple second ordered data structures each ordering requests received from a respective one of said multiple local hub processing units; and

an arbiter that selects requests for transmission to a remote hub processing unit among the plurality of processing units by reference to said first and said second ordered data structures.

4. The data processing system of claim 3, wherein said launch buffer further comprises map logic that routes requests from said master to said first ordered data structure and routes requests of said local hub processing units to said second ordered data structure by reference to relative physical positions of said local master processing unit and said multiple local hub processing units.

5. The data processing system of claim 4, wherein:

each of said multiple local hub processing units includes request logic comprising a launch buffer and arbiter; and

said arbiter at each of said local hub processing units selects requests for transmission from said launch buffer of said each local hub processing unit in accordance with selections of requests by said arbiter at said local master processing unit.

6. The data processing system of claim 1, wherein:

said data processing system includes response generation logic that generates a combined response of said data processing system to the request by said master of said local master processing unit;

said interconnect logic of said local master processing unit includes combined response phase logic coupled to master and to said snooper, wherein said combined response phase logic presents said combined response to said master prior to said snooper.

7. The data processing system of claim 6, wherein said interconnect logic includes a time-ordered tag data structure that records a master tag of said request of said master upon transmission of the request to said snooper and outputs the master tag in response to transmission of said combined response.

8. The data processing system of claim 1, wherein:

said memory access request is a first request and identifies a memory block with a request address;

said data processing system includes response generation logic that generates a combined response of said data processing system to the first request;

said plurality of processing units includes an owning processing node holding coherency ownership of said memory block immediately prior to receipt of said first request;

said plurality of processing units are coupled for communication such that a latency of a second request for access to said memory block from any competing processing unit among said plurality of processing units to said owning processing unit is no greater than the sum of a latency of the second request from the competing processing unit to the local master processing unit plus the latency from a time said combined response is observed at the local master processing unit to a time said combined response is observed at said owning processing unit.

9. A data processing system, comprising:

a plurality of processing units coupled for communication, said plurality of processing units including at least a local master processing unit and a local hub processing unit coupled by a communication link characterized by a link latency, wherein said local master processing unit includes;

a master that initiates a memory access request;

a snooper that receives the memory access request; and

interconnect logic coupled to the communication link, wherein said interconnect logic includes request logic that synchronizes internal transmission of the memory access request of said master to said snooper with transmission, via the communication link, of the memory access request to the local hub processing unit;

response generation logic that generates a combined response of said data processing system to the memory access request of said master;

wherein the interconnect logic includes combined response phase logic, coupled to the master and to the snooper, that transmits said combined response to the master and snooper and from said local master processing unit to said local hub processing unit, wherein said combined response phase logic presents the combined response to said snooper approximately the link latency later than presentation of the combined response to said master.

10. A processing unit for a data processing system, said processing unit comprising:

a master that initiates memory access requests;

a snooper that receives memory access requests; and

interconnect logic coupled to a plurality of communication links each coupling the processing unit to a respective one of multiple local hub processing units within the data processing system, wherein said interconnect logic includes request logic that synchronizes internal delivery of a memory access request of said master to said snooper with delivery, via the plurality of communication links, of the memory access request to snoopers in the plurality of local hub processing units, wherein the request logic synchronizes the internal delivery by applying a greater delay to the memory access request to be delivered to the snooper than to that to be delivered to the plurality of local hub processing units, wherein the request logic applies the delay to the memory access request to be delivered to the snooper after transmission by the request logic of the memory access request to the plurality of local hub processing units and prior to delivery of the memory access request to the snooper.

11. The processing unit of claim 10, wherein the processing unit includes request logic including a launch buffer comprising:

a first ordered data structure that orders requests generated by said master;

multiple second ordered data structures each ordering requests received from a respective one of said multiple hub processing units; and

an arbiter that selects requests for transmission to a remote hub processing unit by reference to said first and said second ordered data structures.

12. The processing unit of claim 10, wherein:

said data processing system includes response generation logic that generates a combined response of said data processing system to the request by said master of said processing unit; and

said interconnect logic includes combined response phase logic coupled to master and to said snooper, wherein said combined response phase logic presents said combined response to said master prior to said snooper.

13. The data processing system of claim 12, wherein:

said combined response phase logic transmits said combined response to said multiple local hub processing units subject to an inter-chip link latency; and

said combined response phase logic presents the combined response to said snooper approximately an inter-chip link latency later than presentation of the combined response to said master.

14. The processing unit of claim 13, wherein said interconnect logic includes a time-ordered tag data structure that records a master tag of said request of said master upon transmission of the request to said snooper and outputs the master tag in response to transmission of said combined response.

15. A multi-chip module, comprising:

a package;

processing unit according to claim 10;

the plurality of communication links; and

the multiple local hub processing units.

16. A processing unit for a multi-processor data processing system, said processing unit comprising:

a master that initiates a request;

a snooper that receives the request; and

interconnect logic coupled to a plurality of communication links each coupling the processing unit to a respective one of multiple local hub processing units of the data processing system, wherein said interconnect logic includes request logic that synchronizes internal delivery of a request of said master to said snooper with delivery, via the communication link, of the request in the plurality of local hub processing units;

request logic including a launch buffer including;

a first ordered data structure that orders requests generated by said master;

multiple second ordered data structure each ordering requests received from one of the multiple local hub processing units; and

an arbiter that selects requests for transmission to by reference to the first and the second ordered data structures;

wherein said launch buffer further comprises map logic that routes requests from said processing unit to said first ordered data structure and routes requests of said multiple local hub processing units to said multiple second ordered data structures by reference to relative physical positions of said processing unit and said multiple local hub processing units.

17. A method of data processing in a data processing system including a plurality of processing units coupled for communication, said plurality of processing units including at least a local master processing unit and multiple local hub processing units, said method comprising:

initiating, by a master within said local master processing unit, an operation by issuing a memory access request; and

transmitting, by request logic within said local master, said memory access request to an internal snooper of said local master processing unit and transmitting the memory access request to each of said multiple local hub processing units via a respective one of a plurality of external communication links, wherein said request logic synchronizes internal delivery of said memory access request to said snooper with transmission of the memory access request to the multiple local hub processing units, wherein the request logic synchronizes the internal delivery by applying a greater delay to the memory access request to be delivered to the snooper than to that to be delivered to the plurality of local hub processing units, wherein the request logic applies the delay to the memory access request to be delivered to the snooper after transmission by the request logic of the memory access request to the plurality of local hub processing units and prior to delivery of the memory access request to the snooper.

18. The method of claim 17, wherein:

the plurality of processing units further includes a remote hub processing unit coupled to a particular local hub processing unit among said multiple local hub processing units by a communication link; and

said method further comprises said particular local hub transmitting said memory access request to said remote hub processing unit in response to receipt of the memory access request from the local master processing unit.

19. The method of claim 17, wherein:

said method further comprises;

buffering in a first ordered data structure of said local master processing unit requests generated by said master;

buffering in each of multiple second ordered data structures of said local master processing unit requests received from a respective one of said multiple local hub processing units; and

selecting requests for transmission from said local master processing unit to a remote hub processing unit among the plurality of processing units by reference to said first and said second ordered data structures.

20. The method of claim 19, and further comprising routing requests from said master to said first ordered data structure and routing requests of said multiple local hub processing units to said multiple second ordered data structures by reference to relative physical positions of said local master processing units and said multiple local hub processing units.

21. The method of claim 20, and further comprising:

buffering requests in first and second ordered data structures within said multiple local hub processing units; and

at said local hub processing unit, selecting requests for transmission from said first and second ordered data structures in accordance with selections of requests for transmission at said local master processing unit.

22. The method of claim 17, wherein:

generating a combined response of said data processing system to the request by said master of said local master processing unit; and

presenting said combined response to said master prior to said snooper.

23. The method of claim 22, wherein:

transmitting said combined response from said local master processing unit to said local hub processing unit subject to an inter-chip link latency; and

said presenting comprises presenting the combined response to said snooper approximately an inter-chip link latency later than to said master.

24. The method of claim 22, and further comprising:

recording, within a time-ordered tag data structure, a master tag of said request of said master upon transmission of the request to said snooper; and

outputting the master tag from said time-ordered tag data structure in response to transmission of said combined response.

25. A method of data processing in a data processing system including a plurality of processing units coupled for communication, said plurality of processing units including at least a local master processing unit and a local hub processing unit, said method comprising:

initiating, by a master within said local master processing unit, an operation by issuing a first request that identifies a memory block with a request address, wherein the plurality of processing units includes an owning processing unit that holds coherency ownership of said memory block immediately prior to receipt by the owning processing unit of said first request;

transmitting, by request logic within said local master processing unit, the request to an internal snooper of said local master processing unit and transmitting the request to the local hub processing unit via an external communication link, wherein said request logic synchronizes internal delivery of said request to said snooper with delivery of the request in the local hub,

generating, by response generation logic, a combined response of said data processing system to the first request; and

communicating said combined response to said plurality of processing units such that a latency of a second request for access to said memory block from any competing processing unit among said plurality of processing units to said owning processing unit is no greater than the sum of a latency of the second request from the competing processing unit to the local master processing unit plus a latency from a time said combined response is observed at the local master processing unit to a time said combined response is observed at said owning processing unit.

26. The data processing system, comprising:

a plurality of processing units coupled for communication, said plurality of processing units including at least a local master processing unit and multiple local hub processing units each coupled to the local master processing unit by a respective one of a plurality of communication links, wherein said local master processing unit includes;

a master that initiates memory access requests;

a snooper that receives memory access requests; and

interconnect logic coupled to the plurality of communication links, wherein said interconnect logic includes;

means for synchronizing internal delivery of a memory access request of said master to said snooper with delivery, via the plurality of communication links, of the memory access request to snoopers in the plurality of local hub processing units, wherein the means for synchronizing synchronizes the internal delivery by applying a greater delay to the memory access request to be delivered to the snooper than to that to be delivered to the plurality of local hub processing units, wherein the means for synchronizing applies the delay to the memory access request to be delivered to the snooper after transmission of the memory access request to the plurality of local hub processing units and prior to delivery of the memory access request to the snooper.