System for Providing a Cluster-Wide System Clock in a Multi-Tiered Full-Graph Interconnect Architecture

• US 20090063886A1
• Filed: 08/31/2007
• Published: 03/05/2009
• Est. Priority Date: 08/31/2007
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Arimilli, Ravi K., Arimilli, Lakshminarayana B., Lewis, Jerry D., Joyner, Jody B., Drerup, Bernard C.
• Granted Patent: US 7,827,428 B2
• Time in Patent Office: Days
• US Class Current: 713/400
• CPC Class Codes: G06F 1/12 Synchronisation of different clock signals provided by a plurality of clock generators
• CPC Class Codes: H04J 3/0685 Clock or time synchronisation in a node; Intranode synchronisation
• CPC Class Codes: H04Q 2213/13092 Scanning of subscriber lines, monitoring
• CPC Class Codes: H04Q 2213/13214 Clock signals
• CPC Class Codes: H04Q 2213/13322 Integrated circuits
• CPC Class Codes: H04Q 2213/1336 Synchronisation

Abstract

A system for providing a cluster-wide system clock in a multi-tiered full graph (MTFG) interconnect architecture are provided. Heartbeat signals transmitted by each of the processor chips in the computing cluster are synchronized. Internal system clock signals are generated in each of the processor chips based on the synchronized heartbeat signals. As a result, the internal system clock signals of each of the processor chips are synchronized since the heartbeat signals, that are the basis for the internal system clock signals, are synchronized. Mechanisms are provided for performing such synchronization using direct couplings of processor chips within the same processor book, different processor books in the same supernode, and different processor books in different supernodes of the MTFG interconnect architecture.

Claims

1. A system, comprising:

a plurality of processor chips, each processor chip in the plurality of processor chips having heartbeat signal generation logic; and

at least one communication link between the plurality of processor chips, wherein;

the at least one communication link couples the plurality of processor chips into a computing cluster,

heartbeat signal generation logic of a first processor chip synchronizes a first heartbeat signal transmitted by the first processor chip with second heartbeat signals of other second processor chips in the plurality of processor chips such that each processor chip in the plurality of processor chips transmits a heartbeat signal at approximately a same time, and

the first processor chip in the plurality of processor chips generates an internal system clock signal based on the first heartbeat signal such that, through synchronization of the first heartbeat signal with the second heartbeat signals of the other second processor chips of the plurality of processor chips, the internal system clock signals of each processor chip of the plurality of processor chips are synchronized.

2. The system of claim 1, wherein the computing cluster comprises a multi-tiered full graph interconnect architecture data processing system having a plurality of supernodes, each supernode having one or more processor books, and each processor book having one or more processor chips of the plurality of processor chips.

3. The system of claim 1, wherein the heartbeat signal generation logic of the first processor chip synchronizes the first heartbeat signal by:

transmitting, from the first processor chip, the first heartbeat signal to at least one second processor chip directly coupled to the processor chip, wherein, in response to receiving the first heartbeat signal, the at least one second processor chip transmits a second heartbeat signal to at least one third processor chip directly coupled to the at least one second processor chip.

4. The system of claim 3, wherein the second heartbeat signal is timed to be transmitted at approximately a same time as the first heartbeat signal is re-transmitted to the at least one second processor chip.

5. The system of claim 3, wherein the first processor chip transmits the first heartbeat signal to at least one second processor chip by transmitting the first heartbeat signal on the ports or phys associated with the first processor chip to second processor chips coupled to the ports or phys.

6. The system of claim 3, wherein the first processor chip transmits the first heartbeat signal to at least one second processor chip by:

performing a lookup operation in at least one routing table data structure to identify second processor chips directly coupled to the first processor chip; and

transmitting the first heartbeat signal to second processor chips that are directly coupled to the first processor chip based on results of the lookup operation.

7. The system of claim 6, wherein the first processor chip performs the lookup operation by:

determining if a processor chip that is directly coupled to the first processor chip is in an error state; and

eliminating the processor chip that is in an error state from the second processor chips.

8. The system of claim 3, wherein the at least one second processor chip transmits a second heartbeat signal by:

predicting a time at which the first processor chip will re-transmit the first heartbeat signal; and

timing the transmitting of the second heartbeat signal based on the predicted time at which the first processor chip will re-transmit the first heartbeat signal.

9. The system of claim 8, wherein the first heartbeat signal includes a timestamp of when the first heartbeat signal was transmitted by the first processor chip, and wherein predicting a time at which the first processor chip will re-transmit the first heartbeat signal comprises determining a delay during transmission of the first heartbeat signal and using the delay to predict when the first heartbeat signal will be re-transmitted.

10. The system of claim 1, wherein the first processor chip initiates synchronizing the first heartbeat signal with the second heartbeat signals in response to one of a user input, a type of application or job that is to be processed by the computing cluster, or a determination that synchronization of at least two processor chips in the computing cluster are sufficiently out of synchronization with each other.

11. The system of claim 1, wherein the first and second heartbeat signals comprise header information identifying at least one identifier of a source processor chip of the first and second heartbeat signals.

12. The system of claim 1, wherein the first processor chip uses the internal system clock signal to time processing of instructions by processors of the first processor chip.

13. The system of claim 1, wherein the first processor chip:

monitors receipt of heartbeat signals from directly coupled other processor chips in the plurality of processor chips;

determines if a heartbeat signal from any of the directly coupled other processor chips is out of synchronization with a heartbeat signal of the processor chip; and

initiates synchronizing heartbeat signals in response to the determination indicating that a heartbeat signal of a directly coupled other processor chip is out of synchronization with the heartbeat signal of the processor chip.

14. The system of claim 13, wherein determining if a heartbeat signal from any of the directly coupled other processor chips is out of synchronization with a heartbeat signal of the processor chip comprises determining a difference in timestamps of the heartbeat signals and comparing the difference to a threshold, wherein if the difference is equal to or greater than the threshold, the synchronizing heartbeat signals operation is initiated.

15. The system of claim 1, wherein one or more processor chips in the plurality of processor chips are provided in a first processor book and one or more processor chips in the plurality of processor chips are provided in a second processor book, and wherein the first processor book is in a first supernode and the second processor book is in a second supernode.

16. A processor chip, comprising:

one or more processors; and

heartbeat signal generation logic, wherein;

the heartbeat signal generation logic synchronizes a first heartbeat signal transmitted by the processor chip with second heartbeat signals of other second processor chips in a plurality of processor chips such that each processor chip in the plurality of processor chips transmits a heartbeat signal at approximately a same time, and

the processor chip generates an internal system clock signal for use by the one or more processors based on the first heartbeat signal such that, through synchronization of the first heartbeat signal with the second heartbeat signals of the other second processor chips of the plurality of processor chips, the internal system clock signals of each processor chip of the plurality of processor chips are synchronized.

17. The processor chip of claim 16, wherein the heartbeat signal generation logic of the processor chip synchronizes the first heartbeat signal by:

transmitting, from the processor chip, the first heartbeat signal to at least one second processor chip directly coupled to the processor chip, wherein, in response to receiving the first heartbeat signal, the at least one second processor chip transmits a second heartbeat signal to at least one third processor chip directly coupled to the at least one second processor chip.

18. The processor chip of claim 17, wherein the second heartbeat signal is timed to be transmitted at approximately a same time as the first heartbeat signal is re-transmitted to the at least one second processor chip.

19. The processor chip of claim 17, wherein the processor chip transmits the first heartbeat signal to at least one second processor chip by:

performing a lookup operation in at least one routing table data structure to identify second processor chips directly coupled to the processor chip; and

transmitting the first heartbeat signal to second processor chips that are directly coupled to the processor chip based on results of the lookup operation.

20. The processor chip of claim 19, wherein the processor chip performs the lookup operation by:

determining if a second processor chip that is directly coupled to the processor chip is in an error state; and

eliminating the second processor chip that is in an error state from the at least one second processor chip to which the first heartbeat signal is transmitted.

21. The processor chip of claim 17, wherein the at least one second processor chip transmits a second heartbeat signal by:

predicting a time at which the processor chip will re-transmit the first heartbeat signal; and

timing the transmitting of the second heartbeat signal based on the predicted time at which the processor chip will re-transmit the first heartbeat signal.

22. The processor chip of claim 16, wherein the processor chip initiates synchronizing the first heartbeat signal with the second heartbeat signals in response to one of a user input, a type of application or job that is to be processed by the processor chip, or a determination that synchronization of at least two processor chips in a computing cluster are sufficiently out of synchronization with each other.

23. The processor chip of claim 16, wherein the first and second heartbeat signals comprise header information identifying at least one identifier of a source processor chip of the first and second heartbeat signals.

24. The processor chip of claim 16, wherein the processor chip uses the internal system clock signal to time processing of instructions by processors of the first processor chip.

25. The processor chip of claim 16, wherein the processor chip:

monitors receipt of heartbeat signals from directly coupled other processor chips in the plurality of processor chips;

determines if a heartbeat signal from any of the directly coupled other processor chips is out of synchronization with a heartbeat signal of the processor chip; and

initiates synchronizing heartbeat signals in response to the determination indicating that a heartbeat signal of a directly coupled other processor chip is out of synchronization with the heartbeat signal of the processor chip.