Cluster-wide system clock in a multi-tiered full-graph interconnect architecture

• US 7,921,316 B2
• Filed: 09/11/2007
• Issued: 04/05/2011
• Est. Priority Date: 09/11/2007
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Joyner, Jody B., Drerup, Bernard C., Arimilli, Ravi K., Arimilli, Lakshminarayana B., Lewis, Jerry D.
• Primary Examiner(s): CONNOLLY, MARK A
• Application Number:
  US11/853,522
• Publication Number:
  US 20090070617A1
• Time in Patent Office: 1,302 Days
• Field of Search: 713/375, 713/400, 713/500, 713/600
• US Class Current: 713/375
• CPC Class Codes: G06F 1/10 Distribution of clock signals , e.g. skew
• CPC Class Codes: G06F 1/12 Synchronisation of different clock signals provided by a plurality of clock generators

Abstract

Mechanisms for providing a cluster-wide system clock in a multi-tiered full graph (MTFG) interconnect architecture are provided. Heartbeat signals transmitted by each of the processor chips in the computing cluster are synchronized. Internal system clock signals are generated in each of the processor chips based on the synchronized heartbeat signals. As a result, the internal system clock signals of each of the processor chips are synchronized since the heartbeat signals, that are the basis for the internal system clock signals, are synchronized. Mechanisms are provided for performing such synchronization using direct couplings of processor chips within the same processor book, different processor books in the same supernode, and different processor books in different supernodes of the MTFG interconnect architecture.

Claims

1. A method, in a computing cluster comprising a plurality of processor chips, for providing a cluster-wide system clock signal, comprising:

synchronizing heartbeat signals transmitted by each of the processor chips in the plurality of processor chips such that each processor chip in the plurality of processor chips transmits a heartbeat signal at approximately a same time; and

generating, in each processor chip of the plurality of processor chips, an internal system clock signal based on a heartbeat signal transmitted by the processor chip, wherein through synchronization of the heartbeat signals the internal system clock signals of each processor chip of the plurality of processor chips are synchronized.

2. The method of claim 1, wherein the computing cluster comprises a multi-tiered full graph interconnect architecture data processing system having a plurality of supernodes, each supernode having one or more processor books, and each processor book having one or more processor chips of the plurality of processor chips.

3. The method of claim 1, wherein synchronizing heartbeat signals transmitted by each of the processor chips comprises:

transmitting, from a first processor chip, a first heartbeat signal to at least one second processor chip directly coupled to the first processor chip; and

transmitting, in response to receiving the first heartbeat signal, from the at least one second processor chip, a second heartbeat signal to at least one third processor chip directly coupled to the at least one second processor chip.

4. The method of claim 3, wherein the second heartbeat signal is timed to be transmitted at approximately a same time as the first heartbeat signal is re-transmitted to the at least one second processor chip.

5. The method of claim 3, wherein transmitting the first heartbeat signal to at least one second processor chip comprises transmitting the first heartbeat signal on the ports or phys associated with the first processor chip to second processor chips coupled to the ports or phys.

6. The method of claim 3, wherein transmitting the first heartbeat signal to at least one second processor chip comprises:

performing a lookup operation in at least one routing table data structure to identify second processor chips directly coupled to the first processor chip; and

transmitting the first heartbeat signal to second processor chips that are directly coupled to the first processor chip based on results of the lookup operation.

7. The method of claim 6, wherein performing the lookup operation further comprises:

determining if a processor chip that is directly coupled to the first processor chip is in an error state; and

eliminating the processor chip that is in an error state from the second processor chips.

8. The method of claim 3, wherein transmitting a second heartbeat signal comprises:

predicting a time at which the first processor chip will re-transmit the first heartbeat signal; and

timing the transmitting of the second heartbeat signal based on the predicted time at which the first processor chip will re-transmit the first heartbeat signal.

9. The method of claim 8, wherein the first heartbeat signal includes a timestamp of when the heartbeat signal was transmitted by the first processor chip, and wherein predicting a time at which the first processor chip will re-transmit the first heartbeat signal comprises determining a delay during transmission of the first heartbeat signal and using the delay to predict when the first heartbeat signal will be re-transmitted.

10. The method of claim 1, wherein the method is initiated in response to one of a user input requesting the method to be performed, a type of application or job that is to be processed by the computing cluster, or a determination that synchronization of at least two processor chips in the computing cluster are sufficiently out of synchronization with each other.

11. The method of claim 1, wherein the heartbeat signal comprises header information identifying at least one identifier of a source processor chip of the heartbeat signal.

12. The method of claim 1, further comprising:

using the internal system clock signal to time processing of instructions by processors of the processor chip.

13. The method of claim 1, further comprising:

monitoring, in each processor chip, receipt of heartbeat signals from directly coupled other processor chips in the plurality of processor chips;

determining if a heartbeat signal from any of the directly coupled other processor chips is out of synchronization with a heartbeat signal of the processor chip; and

initiating the synchronizing heartbeat signals operation in response to the determination indicating that a heartbeat signal of a directly coupled other processor chip is out of synchronization with the heartbeat signal of the processor chip.

14. The method of claim 13, wherein determining if a heartbeat signal from any of the directly coupled other processor chips is out of synchronization with a heartbeat signal of the processor chip comprises determining a difference in timestamps of the heartbeat signals and comparing the difference to a threshold, wherein if the difference is equal to or greater than the threshold, the synchronizing heartbeat signals operation is initiated.

15. The method of claim 1, further comprising:

initiating, by a master processor chip in the plurality of processor chips, a heartbeat synchronization operation;

performing the heartbeat synchronization operation; and

broadcasting an end of the heartbeat synchronization operation in response to the performing of the heartbeat synchronization operation being completed, wherein generating the internal system clock signal is performed in response to receiving the end of the heartbeat synchronization operation.

16. The method of claim 1, wherein one or more processor chips in the plurality of processor chips are provided in a first processor book and one or more processor chips in the plurality of processor chips are provided in a second processor book.

17. The method of claim 1, wherein the first processor book is in a first supernode and the second processor book is in a second supernode.

18. A computer program product comprising a computer useable storage medium having a computer readable program stored thereon, wherein the computer readable program, when executed in a data processing system, causes the data processing system to:

synchronize heartbeat signals transmitted by each processor chip in a plurality of processor chips such that each processor chip in the plurality of processor chips transmits a heartbeat signal at approximately a same time; and

generates, in each processor chip of the plurality of processor chips, an internal system clock signal based on a heartbeat signal transmitted by the processor chip, wherein through synchronization of the heartbeat signals the internal system clock signals of each processor chip of the plurality of processor chips are synchronized.

19. The computer program product of claim 18, wherein the computer readable program causes the data processing system to synchronize heartbeat signals transmitted by each of the processor chips by:

transmitting, from a first processor chip, a first heartbeat signal to at least one second processor chip directly coupled to the first processor chip; and

transmitting, in response to receiving the first heartbeat signal, from the at least one second processor chip, a second heartbeat signal to at least one third processor chip directly coupled to the at least one second processor chip.

20. The computer program product of claim 18, wherein the computer readable program causes the data processing system to transmit a second heartbeat signal by:

predicting a time at which the first processor chip will re-transmit the first heartbeat signal; and

timing the transmitting of the second heartbeat signal based on the predicted time at which the first processor chip will re-transmit the first heartbeat signal.