Synchronization in a Multi-Tile, Multi-Chip Processing Arrangement

• US 20200133914A1
• Filed: 12/23/2019
• Published: 04/30/2020
• Est. Priority Date: 10/20/2017
• Status: Active Grant

• Current Assignee: Graphcore Limited
• Original Assignee: Graphcore Limited
• Inventors: Wilkinson, Daniel John Pelham, Felix, Stephen, Osborne, Richard Luke Southwell, Knowles, Simon Christian, Alexander, Alan Graham, Quinn, Ian James
• Granted Patent: US 11,023,413 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 15/17325 Synchronisation; Hardware support therefor intertask synchronisation G06F9/52
• CPC Class Codes: G06F 15/80 comprising an array of processing units with common control, e.g. single instruction multiple data processors G06F15/82 takes precedence ; for correlation function computation G06F17/15
• CPC Class Codes: G06F 2209/505 Clust
• CPC Class Codes: G06F 9/522 Barrier synchronisation

Abstract

A method of operating a system comprising multiple processor tiles divided into a plurality of domains wherein within each domain the tiles are connected to one another via a respective instance of a time-deterministic interconnect and between domains the tiles are connected to one another via a non-time-deterministic interconnect. The method comprises: performing a compute stage, then performing a respective internal barrier synchronization within each domain, then performing an internal exchange phase within each domain, then performing an external barrier synchronization to synchronize between different domains, then performing an external exchange phase between the domains.

Claims

1. A method of operating a system comprising a plurality of processor tiles divided into a plurality of domains wherein each domain has inter-tile connections via a time-deterministic interconnect, and inter-domain tile connections are made via a non-time-deterministic interconnect;

the method comprising;

performing a first compute phase at a first tile in a first domain;

performing a second compute phase at a second tile in the first domain;

performing an internal barrier synchronization within the first domain to require that the first tile has completed the first compute phase and the second tile has completed the second compute phase before proceeding to a first internal exchange phase at the first domain;

following the internal barrier synchronization, performing the first internal exchange phase between the first tile and the second tile within the first domain, in which the first tile communicates results of its computations to the second tile via the time-deterministic interconnect, wherein the first internal exchange phase does not include communicating computation results from the first domain to a second domain;

performing an external barrier synchronization to require the first tile and the second tile of the first domain have completed the first internal exchange phase and a third tile of the second domain has completed a second internal exchange phase at the second domain before any of the first tile, the second tile, or the third tile is allowed to proceed to an external exchange phase; and

following the external barrier synchronization, performing the external exchange phase in which the first tile communicates results of its computations with the third tile via the non-time-deterministic interconnect.

2. The method of claim 1, wherein communications via the non-time-deterministic interconnect are queued.

3. The method of claim 1, wherein communications between the first tile and the second tile via the time-deterministic interconnect are not queued.

4. The method of claim 1, wherein communications between the first tile and the second tile via the time-deterministic interconnect are performed by:

transmitting a message from the first tile, and

controlling the first tile to listen to an address of the second tile at a time interval after a transmission by the second tile.

5. The method of claim 1, wherein communications between the first tile and the second tile via the time-deterministic interconnect are performed by:

controlling the first tile to listen to an address of the second tile at a time interval after a transmission by the second tile, wherein the time interval is equal to a total delay between the second tile and the first tile, the time interval being set by a compiler.

6. The method of claim 1, wherein the time-deterministic interconnect comprises an external lossless inter-chip interconnect.

7. The method of claim 1, wherein the external barrier synchronization includes executing a synchronization instruction comprising an opcode and an operand, wherein the operand specifies a mode of the synchronization instruction as being external, and wherein the opcode, when executed, causes hardware logic in the time-deterministic interconnect to coordinate performance of the internal barrier synchronization.

8. The method of claim 1, wherein the internal barrier synchronization includes executing a synchronization instruction comprising an opcode and an operand, wherein the operand specifies a mode of the synchronization instruction as being internal, and wherein the opcode, when executed, causes hardware logic in the non-time-deterministic interconnect to coordinate performance of the external barrier synchronization.

9. The method of claim 1, wherein in the external exchange phase, the first tile also communicates computation results with a host processor via the the non-time-deterministic interconnect, the host processor being implemented on a separate host processor chip.

10. At least one non-transitory computer-readable storage having encoded thereon code configured so as when executed on a plurality of tiles performs operations including:

perform a first compute phase at a first tile in a first domain;

perform a second compute phase at a second tile in the first domain;

perform an internal barrier synchronization within the first domain to require that the first tile has completed the first compute phase and the second tile has completed the second compute phase before proceeding to a first internal exchange phase at the first domain;

following the internal barrier synchronization, perform the first internal exchange phase between the first tile and the second tile within the first domain, in which the first tile communicates results of its computations to the second tile via a time-deterministic interconnect, wherein the first internal exchange phase does not include communicating computation results from the first domain to a second domain;

perform an external barrier synchronization to require the first tile and the second tile of the first domain have completed the first internal exchange phase and a third tile of the second domain has completed a second internal exchange phase at the second domain before any of the first tile, the second tile, or the third tile is allowed to proceed to an external exchange phase; and

following the external barrier synchronization, perform the external exchange phase in which the first tile communicates results of its computations with the third tile via a non-time-deterministic interconnect between the first domain and the second domain.

11. The non-transitory computer-readable storage of claim 10, wherein communications via the non-time-deterministic interconnect are queued.

12. The non-transitory computer-readable storage of claim 10, wherein communications between the first tile and the second tile via the time-deterministic interconnect are not queued.

13. The non-transitory computer-readable storage of claim 10, wherein communications between the first tile and the second tile via the time-deterministic interconnect are performed by:

transmitting a message from the first tile, and

controlling the first tile to listen to an address of the second tile at a time interval after a transmission by the second tile.

14. The non-transitory computer-readable storage of claim 10, wherein communications between the first tile and the second tile via the time-deterministic interconnect are performed by:

controlling the first tile to listen to an address of the second tile at a time interval after a transmission by the second tile, wherein the time interval is equal to a total delay between the second tile and the first tile, the time interval being set by a compiler.

15. The non-transitory computer-readable storage of claim 10, wherein the external barrier synchronization includes executing a synchronization instruction comprising an opcode and an operand, wherein the operand specifies a mode of the synchronization instruction as being external, and wherein the opcode, when executed, causes hardware logic in the time-deterministic interconnect to coordinate performance of the internal barrier synchronization.

16. The non-transitory computer-readable storage of claim 10, wherein the internal barrier synchronization includes executing a synchronization instruction comprising an opcode and an operand, wherein the operand specifies a mode of the synchronization instruction as being internal, and wherein the opcode, when executed, causes hardware logic in the non-time-deterministic interconnect to coordinate performance of the external barrier synchronization.

17. The non-transitory computer-readable storage of claim 10, wherein in the external exchange phase, the first tile also communicates computation results with a host processor via the non-time-deterministic interconnect, the host processor being implemented on a separate host processor chip.

18. A system comprising a plurality of processor tiles divided into a plurality of domains wherein each domain has inter-tile connections via a time-deterministic interconnect, and inter-domain tile connections are made via a non-time-deterministic interconnect;

the system being programmed to perform operations of;

performing a first compute phase at a first tile in a first domain;

performing a second compute phase at a second tile in the first domain;

performing an internal barrier synchronization within the first domain to require that the first tile has completed the first compute phase and the second tile has completed the second compute phase before proceeding to a first internal exchange phase at the first domain;

following the internal barrier synchronization, performing the first internal exchange phase between the first tile and the second tile within the first domain, in which the first tile communicates results of its computations to the second tile via the time-deterministic interconnect, wherein the first internal exchange phase does not include communicating computation results from the first domain to a second domain;

performing an external barrier synchronization to require the first tile and the second tile of the first domain have completed the first internal exchange phase and a third tile of the second domain has completed a second internal exchange phase at the second domain before any of the first tile, the second tile, or the third tile is allowed to proceed to an external exchange phase; and

following the external barrier synchronization, performing the external exchange phase in which the first tile communicates results of its computations with the third tile via the non-time-deterministic interconnect.

19. The system of claim 18, wherein the time-deterministic interconnect comprises an external lossless inter-chip interconnect.

20. The system of claim 18, wherein the time-deterministic interconnect is lossless.