Synchronization in a multi-tile, multi-chip processing arrangement

• US 10,579,585 B2
• Filed: 02/01/2018
• Issued: 03/03/2020
• Est. Priority Date: 10/20/2017
• Status: Active Grant

• Current Assignee: Graphcore Limited
• Original Assignee: Graphcore Limited
• Inventors: Wilkinson, Daniel John Pelham, Felix, Stephen, Osborne, Richard Luke Southwell, Knowles, Simon Christian, Alexander, Alan Graham, Quinn, Ian James
• Primary Examiner(s): Coleman, Eric
• Application Number:
  US15/886,138
• Publication Number:
  US 20190121784A1
• Time in Patent Office: 761 Days
• Field of Search: None
• CPC Class Codes: G06F 15/17325 Synchronisation; Hardware support therefor intertask synchronisation G06F9/52
• CPC Class Codes: G06F 15/80 comprising an array of processing units with common control, e.g. single instruction multiple data processors G06F15/82 takes precedence ; for correlation function computation G06F17/15
• CPC Class Codes: G06F 2209/505 Clust
• CPC Class Codes: G06F 9/522 Barrier synchronisation

Abstract

A method of operating a system comprising multiple processor tiles divided into a plurality of domains wherein within each domain the tiles are connected to one another via a respective instance of a time-deterministic interconnect and between domains the tiles are connected to one another via a non-time-deterministic interconnect. The method comprises: performing a compute stage, then performing a respective internal barrier synchronization within each domain, then performing an internal exchange phase within each domain, then performing an external barrier synchronization to synchronize between different domains, then performing an external exchange phase between the domains.

Claims

1. A method of operating a system comprising multiple processor tiles divided into a plurality of domains wherein within each domain the tiles are connected to one another via a respective instance of a time-deterministic interconnect and between domains the tiles are connected to one another via a non-time-deterministic interconnect;

the method comprising;

on each respective one of a participating group of some or all of the tiles across the domains, performing a compute stage in which the respective tile performs a respective one or more on-tile computations, but communicates computation results neither to nor from any others of the tiles in the group;

within each respective one of said one or more domains, performing a respective internal barrier synchronization to require that all the participating tiles in the respective domain have completed the compute phase before any of the participating tiles in the respective domain is allowed to proceed to an internal exchange phase, thereby establishing a common time-reference between all participating tiles internally within each individual one of said one or more domains;

following the respective internal barrier synchronization, performing said internal exchange phase within each of said one or more domains, in which each participating tile within the respective domain communicates one or more results of its respective computations to and/or from one or more others of the participating tiles within the same domain via the time-deterministic interconnect, but communicates computation results neither to nor from any others of said domains;

performing an external barrier synchronization to require all the participating tiles of said domains to have completed the internal exchange phase before any of the participating tiles is allowed to proceed to an external exchange phase, thereby establishing a common time-reference between all participating tiles across the domains; and

following said external barrier synchronization, performing the external exchange phase in which one or more of the participating tiles communicate one or more of said computation results with another of the domains via the non-time-deterministic interconnect.

2. The method of claim 1, wherein the communications via the non-time-deterministic interconnect are queued, but the communications between tiles via the time-deterministic interconnect are not queued.

3. The method of claim 1, wherein over the time-deterministic interconnect, the communication between each pair of transmitting and receiving tile is performed by:

transmitting a message from the receiving tile, and

controlling the receiving tile to listen to an address of the transmitting tile at a predetermined time interval after the transmission by the transmitting tile,

wherein the predetermined time interval is equal to a total predetermined delay between the transmitting and receiving tile, the time interval being set by a compiler having predetermined information on the delay.

4. The method of claim 1, wherein:

the time-deterministic interconnect is lossless, whilst the non-time-deterministic interconnect is lossy at a physical layer, transport layer or networking layer.

5. The method of claim 1, wherein each of the domains is a different respective chip, the time-deterministic interconnect being an internal on-chip interconnect and the non-time-deterministic interconnect is an external interconnect between the chips.

6. The method of claim 1, wherein each of the domains comprises multiple chips, the time-deterministic interconnect being an external lossless inter-chip interconnect and the non-time-deterministic interconnect is an external lossy interconnect.

7. The method of claim 1, comprising performing a series of repeating iterations, each comprising a respective instance of the compute stage, followed by a respective instance of the internal barrier synchronization, followed by a respective instance of the internal exchange phase, followed by a respective instance of the external barrier synchronization, followed by a respective instance of the external exchange phase;

where each successive iteration is not allowed to proceed until the external barrier synchronization of the immediately preceding iteration has been performed.

8. The method of claim 1, comprising performing a sequence of instances of the compute phase each followed by a corresponding instance of the internal exchange phase and then a corresponding instance of the internal barrier synchronization, wherein the external barrier synchronization follows the last the compute phase in said sequence.

9. The method of claim 7, comprising performing a sequence of instances of the compute phase each followed by a corresponding instance of the internal exchange phase and then a corresponding instance of the internal barrier synchronization, wherein the external barrier synchronization follows the last the compute phase in said sequence;

wherein each of one or more of said iterations comprises a respective sequence of multiple instances of the compute phase each followed by a corresponding instance of the internal exchange phase and then a corresponding instance of the internal barrier synchronization, wherein the respective external barrier synchronization follows the last instance of the compute phase in the respective sequence.

10. The method of claim 1, wherein each of the internal and external barrier synchronizations is performed by executing a synchronization instruction comprising an opcode and an operand, wherein the operand specifies a mode of the synchronization instruction as being either internal or external, and wherein the opcode, when executed, causes hardware logic in the time-deterministic interconnect to coordinate the performance of the internal barrier synchronization when the operand specifies the internal mode, and causes hardware logic in the non-time-deterministic interconnect to coordinate the performance of the external barrier synchronization when the operand specifies the external mode.

11. The method of claim 1, comprising selecting one of plurality of predefined zones as the participating tiles, each zone comprising a different set or subset of said multiple domains.

12. The method of claim 11, wherein the zones are hierarchical, with at least two lower level zones being nested within at least one higher level zone.

13. The method of claim 10, comprising selecting one of plurality of predefined zones as the participating tiles, each zone comprising a different set or subset of said multiple domains;

wherein the operand of the synchronisation instruction specifies which of a plurality of different possible variants of the external mode the external barrier synchronization applies to, each corresponding to a different one of said zones.

14. The method of claim 13, wherein the zones are hierarchical, with at least two lower level zones being nested within at least one higher level zone;

and wherein the variants of the external mode specify at least which hierarchical level of zone the external barrier synchronization applies to.

15. The method of claim 12, wherein the external synchronization and exchange comprises:

first performing a first-level external synchronization and then exchange constrained within a first, lower-level one of said hierarchical zones; and

following the first level synchronization and exchange, performing a second-level external synchronization and exchange across a second, higher level one of said zones.

16. The method of claim 11, wherein one of said hierarchical zones consists of all tiles on in a same IC package, but none beyond;

and/or one of said hierarchical zones consists of all tiles in a same card, but none beyond; and

/or one of said hierarchical zones consist of all tiles in a same chassis, but none beyond.

17. The method of claim 1, comprising executing an abstain instruction on one or some of the tiles, the opcode of the abstain instruction causing the tile or tiles on which it is executed to be opted-out of said group.

18. The method of claim 1, wherein in the external exchange phase, one or more of the participating tiles also communicate one or more of said computation results with a host processor via said external interconnect, the host processor being implemented on a separate host processor chip.

19. The method of claim 1 wherein, in the compute phase, some or all of the participating tiles each run a batch of worker threads in an interleaved manner, and the internal barrier synchronization requires all the worker threads in every batch to have exited.

20. The method of claim 1, comprising using the system to perform a machine intelligence algorithm in which each node in a graph has one or more respective input vertices and one or more respective output vertices with the input vertices of at least some of the nodes being the output vertices of at least some others of the nodes, each node comprising a respective function relating its output vertices to its input vertices, with each respective function being parameterized by one or more respective parameters, and each of the respective parameters having an associated error, such that the graph converges toward a solution as the errors in some or all of the parameters reduce;

wherein each of the tiles models a respective one or more of the nodes in the graph.

21. The method of claim 5, wherein in the external exchange phase, one or more of the participating tiles also communicate one or more of said computation results with a host processor via said external interconnect, the host processor being implemented on a separate host processor chip;

and wherein said chips are Al accelerator chips assisting the host processor.

22. At least one non-transitory computer-readable storage having encoded thereon code configured so as when executed on said tiles to perform operations in accordance with claim 1.

23. A system comprising multiple processor tiles divided into a plurality of domains wherein within each domain the tiles are connected to one another via a respective instance of a time-deterministic interconnect and between domains the tiles are connected to one another via a non-time-deterministic interconnect;

the system being programmed to perform operations of;

on each respective one of a participating group of some or all of the tiles across the domains, performing a compute stage in which the respective tile performs a respective one or more on-tile computations, but communicates computation results neither to nor from any others of the tiles in the group;

within each respective one of said one or more domains, performing a respective internal barrier synchronization to require that all the participating tiles in the respective domain have completed the compute phase before any of the participating tiles in the respective domain is allowed to proceed to an internal exchange phase, thereby establishing a common time-reference between all participating tiles internally within each individual one of said one or more domains;

following the respective internal barrier synchronization, performing said internal exchange phase within each of said one or more domains, in which each participating tile within the respective domain communicates one or more results of its respective computations to and/or from one or more others of the participating tiles within the same domain via the time-deterministic interconnect, but communicates computation results neither to nor from any others of said domains;

performing an external barrier synchronization to require all the participating tiles of said domains to have completed the internal exchange phase before any of the participating tiles is allowed to proceed to an external exchange phase, thereby establishing a common time-reference between all participating tiles across the domains; and

following said external barrier synchronization, performing the external exchange phase in which one or more of the participating tiles communicate one or more of said computation results with another of the domains via the non-time-deterministic interconnect.