Dataflow triggered tasks for accelerated deep learning

• US 10,614,357 B2
• Filed: 04/15/2018
• Issued: 04/07/2020
• Est. Priority Date: 04/17/2017
• Status: Active Grant

• Current Assignee: Cerebras Systems, Inc.
• Original Assignee: Cerebras Systems, Inc.
• Inventors: Lie, Sean, Lauterbach, Gary R., James, Michael Edwin, Morrison, Michael, Arekapudi, Srikanth
• Primary Examiner(s): Vincent, David R
• Application Number:
  US16/463,091
• Publication Number:
  US 20190286987A1
• Time in Patent Office: 723 Days
• Field of Search: 706 15, 706 45
• CPC Class Codes: G06F 13/00 Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units interface circuits for specific input/output devices G06F3/00 ; multiprogram control therefor G06F9/46; multiprocessor systems G06F15/16
• CPC Class Codes: G06F 15/825 Dataflow computers
• CPC Class Codes: G06F 17/16 Matrix or vector computation , e.g. matrix-matrix or matrix-vector multiplication, matrix factorization matrix transposition G06F7/78
• CPC Class Codes: G06F 9/30036 Instructions to perform operations on packed data, e.g. vector, tile or matrix operations
• CPC Class Codes: G06F 9/3005 to perform operations for flow control
• CPC Class Codes: G06F 9/3016 Decoding the operand specifier, e.g. specifier format
• CPC Class Codes: G06F 9/30192 according to data descriptor, e.g. dynamic data typing
• CPC Class Codes: G06F 9/324 using program counter relative addressing
• CPC Class Codes: G06F 9/3802 Instruction prefetching
• CPC Class Codes: G06F 9/3836 Instruction issuing, e.g. dynamic instruction scheduling or out of order instruction execution
• CPC Class Codes: G06F 9/3851 from multiple instruction streams, e.g. multistreaming
• CPC Class Codes: G06F 9/3887 controlled by a single instruction for multiple data lanes [SIMD]
• CPC Class Codes: G06F 9/45533 Hypervisors; Virtual machine monitors
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/5077 Logical partitioning of resources; Management or configuration of virtualized resources specific details on emulation or internal functioning of virtual machines G06F9/455
• CPC Class Codes: G06F 9/52 Program synchronisation; Mutual exclusion, e.g. by means of semaphores
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/048 : Activation functions
• CPC Class Codes: G06N 3/063 : using electronic means
• CPC Class Codes: G06N 3/08 : Learning methods
• CPC Class Codes: G06N 3/084 : Backpropagation, e.g. using gradient descent

Abstract

Techniques in advanced deep learning provide improvements in one or more of accuracy, performance, and energy efficiency. An array of processing elements performs flow based computations on wavelets of data. Each processing element has a compute element and a routing element. Each compute element has memory. Each router enables communication via wavelets with nearest neighbors in a 2D mesh. Routing is controlled by respective virtual channel specifiers in each wavelet and routing configuration information in each router. A compute element receives a particular wavelet comprising a particular virtual channel specifier and a particular data element. Instructions are read from the memory of the compute element based at least in part on the particular virtual channel specifier. The particular data element is used as an input operand to execute at least one of the instructions.

Claims

1. A method comprising:

sending a fabric packet by a sending processing element to a fabric, the fabric packet comprising a virtual channel specifier and a fabric packet payload;

routing the fabric packet via the fabric from the sending processing element to a receiving processing element via zero or more routing processing elements, the routing in accordance with the virtual channel specifier;

in the receiving processing element, receiving the fabric packet from the fabric, reading one or more instructions from a memory of the receiving processing element at an address based at least in part on the virtual channel specifier, and using at least a portion of the fabric packet payload as an input operand to execute at least one of the one or more instructions;

wherein the virtual channel specifier is one of a plurality of virtual channel specifiers, each of the plurality of virtual channel specifiers is associated with a respective set of one or more sets of fabric packets, and the receiving comprises associating the fabric packet with the respective set associated with the virtual channel specifier; and

wherein a block/unblock state is maintained for each of the virtual channel specifiers, and the block/unblock state of a particular one of the virtual channel specifiers is set to a block state in response to a block instruction specifying the particular one of the virtual channel specifiers and the block/unblock state of the particular one of the virtual channel specifiers is set to an unblock state in response to an unblock instruction specifying the particular one of the virtual channel specifiers.

2. The method of claim 1, wherein at least one of the reading and the using are responsive to the receiving processing element selecting the fabric packet for processing, and the selecting is performed at least in part by a scheduler of the receiving processing element.

3. The method of claim 2, wherein the selecting comprises selecting the virtual channel specifier from the virtual channel specifiers in the unblock state, and the fabric packet is older than any other fabric packets associated with the respective set associated with the virtual channel specifier.

4. The method of claim 3, wherein the selecting the virtual channel specifier is based at least in part upon the virtual channel specifier and a virtual channel specifier of a previously selected fabric packet and in accordance with a round-robin scheduling policy.

5. The method of claim 3, wherein the selecting the virtual channel specifier is based at least in part upon the virtual channel specifier and a virtual channel specifier of a previously selected fabric packet and in accordance with selecting the virtual channel specifier of the previously selected fabric packet until a fabric packet comprising a control specifier set to a predetermined value is selected.

6. The method of claim 3, wherein the selecting the virtual channel specifier is based at least in part upon the virtual channel specifier being one of one or more predetermined prioritized virtual channel specifiers of the plurality of virtual channel specifiers.

7. The method of claim 1, wherein the sending processing element and the receiving processing element implement at least portions of a neuron of a neural network.

8. The method of claim 1, wherein the sending processing element, the routing processing elements, and the receiving processing element are fabricated via wafer-scale integration.

9. A method comprising:

sending a fabric packet by a sending processing element to a fabric, the fabric packet comprising a virtual channel specifier and a fabric packet payload;

routing the fabric packet via the fabric from the sending processing element to a receiving processing element via zero or more routing processing elements, the routing in accordance with the virtual channel specifier;

in the receiving processing element, receiving the fabric packet from the fabric, reading one or more instructions from a memory of the receiving processing element at an address based at least in part on the virtual channel specifier, and using at least a portion of the fabric packet payload as an input operand to execute at least one of the one or more instructions; and

wherein the portion of the fabric packet payload comprises one or more data elements and the one or more data elements comprise at least a portion of one of;

a weight of a neural network, an activation of a neural network, a partial sum of activations of a neural network, an error of a neural network, a gradient estimate of a neural network, and a weight update of a neural network.

10. The method of claim 9, wherein the sending processing element, the routing processing elements, and the receiving processing element are fabricated via wafer-scale integration.

11. The method of claim 9, wherein the sending processing element and the receiving processing element implement at least portions of a neuron of a neural network.

12. A method comprising:

sending a fabric packet by a sending processing element to a fabric, the fabric packet comprising a virtual channel specifier and a fabric packet payload;

routing the fabric packet via the fabric from the sending processing element to a receiving processing element via zero or more routing processing elements, the routing in accordance with the virtual channel specifier;

in the receiving processing element, receiving the fabric packet from the fabric, reading one or more instructions from a memory of the receiving processing element at an address based at least in part on the virtual channel specifier, and using at least a portion of the fabric packet payload as an input operand to execute at least one of the one or more instructions; and

wherein the one or more instructions implement at least a portion of one or more of;

computing an activation of a neural network, computing a partial sum of activations of a neural network, computing an error of a neural network, computing a gradient estimate of a neural network, and updating a weight of a neural network.

13. The method of claim 12, wherein the routing is a first routing, the receiving processing element is a first receiving processing element, the zero or more routing processing elements are first zero or more routing processing elements, the receiving is a first receiving, the reading is a first reading, the using is a first using, and the one or more instructions are first one or more instructions;

and further comprising a second routing the fabric packet via the fabric from the sending processing element to at least a second receiving processing element via second zero or more routing processing elements, the second routing in accordance with the virtual channel specifier, and in the second receiving processing element a second receiving the fabric packet from the fabric, a second reading a second one or more instructions from a memory of the second receiving processing element at an address based at least in part on the virtual channel specifier, and a second using at least a portion of the fabric packet payload as an input operand to execute at least one of the second one or more instructions.

14. The method of claim 12, wherein the sending processing element, the routing processing elements, and the receiving processing element are fabricated via wafer-scale integration.

15. The method of claim 12, wherein the sending processing element and the receiving processing element implement at least portions of a neuron of a neural network.

16. A method comprising:

sending a fabric packet by a sending processing element to a fabric, the fabric packet comprising a virtual channel specifier and a fabric packet payload;

routing the fabric packet via the fabric from the sending processing element to a receiving processing element via zero or more routing processing elements, the routing in accordance with the virtual channel specifier;

in the receiving processing element, receiving the fabric packet from the fabric, reading one or more instructions from a memory of the receiving processing element at an address based at least in part on the virtual channel specifier, and using at least a portion of the fabric packet payload as an input operand to execute at least one of the one or more instructions; and

wherein a virtual channel associated with the virtual channel specifier is used for communicating at least one of control and data associated with one or more of;

computing an activation of a neural network, computing a partial sum of activations of a neural network, computing an error of a neural network, computing a gradient estimate of a neural network, and updating a weight of a neural network.

17. The method of claim 16, wherein the virtual channel specifier is one of a plurality of virtual channel specifiers, each of the plurality of virtual channel specifiers is associated with a respective set of one or more sets of fabric packets, and the receiving comprises associating the fabric packet with the respective set associated with the virtual channel specifier.

18. The method of claim 16, wherein the sending processing element, the routing processing elements, and the receiving processing element are fabricated via wafer-scale integration.

19. The method of claim 16, wherein the sending processing element and the receiving processing element implement at least portions of a neuron of a neural network.

20. A system comprising:

means for sending a fabric packet by a sending processing element to a fabric, the fabric packet comprising a virtual channel specifier and a fabric packet payload;

means for routing the fabric packet via the fabric from the sending processing element to a receiving processing element via zero or more routing processing elements, the means for routing operating in accordance with the virtual channel specifier;

in the receiving processing element, means for receiving the fabric packet from the fabric, means for reading one or more instructions from a memory of the receiving processing element at an address based at least in part on the virtual channel specifier, and means for using at least a portion of the fabric packet payload as an input operand to execute at least one of the one or more instructions;

wherein the virtual channel specifier is one of a plurality of virtual channel specifiers, each of the plurality of virtual channel specifiers associated with a respective set of one or more sets of fabric packets, and the means for receiving comprises means for associating the fabric packet with the respective set associated with the virtual channel specifier; and

further comprising means for maintaining respective block/unblock state for each of the virtual channel specifiers; and

wherein the block/unblock state of a particular one of the virtual channel specifiers is set to a block state in response to a block instruction specifying the particular one of the virtual channel specifiers and wherein the block/unblock state of the particular one of the virtual channel specifiers is set to an unblock state in response to an unblock instruction specifying the particular one of the virtual channel specifiers.

21. The system of claim 20, wherein at least one of the means for reading and the means for using are responsive to the receiving processing element selecting the fabric packet for processing, and the selecting is performed at least in part by a scheduler of the receiving processing element.

22. The system of claim 21, wherein the selecting comprises selecting the virtual channel specifier from the virtual channel specifiers in the unblock state, and the fabric packet is older than any other fabric packets associated with the respective set associated with the virtual channel specifier.

23. The system of claim 22, wherein the selecting the virtual channel specifier is based at least in part upon the virtual channel specifier and a virtual channel specifier of a previously selected fabric packet and in accordance with a round-robin scheduling policy.

24. The system of claim 22, wherein the selecting the virtual channel specifier is based at least in part upon the virtual channel specifier and a virtual channel specifier of a previously selected fabric packet and in accordance with selecting the virtual channel specifier of the previously selected fabric packet until a fabric packet comprising a control specifier set to a predetermined value is selected.

25. The system of claim 22, wherein the selecting the virtual channel specifier is based at least in part upon the virtual channel specifier being one of one or more predetermined prioritized virtual channel specifiers of the plurality of virtual channel specifiers.

26. The system of claim 20, wherein the sending processing element and the receiving processing element implement at least portions of a neuron of a neural network.

27. The system of claim 20, wherein the sending processing element, the routing processing elements, and the receiving processing element are fabricated via wafer-scale integration.

28. A system comprising:

means for sending a fabric packet by a sending processing element to a fabric, the fabric packet comprising a virtual channel specifier and a fabric packet payload;

means for routing the fabric packet via the fabric from the sending processing element to a receiving processing element via zero or more routing processing elements, the means for routing operating in accordance with the virtual channel specifier;

in the receiving processing element, means for receiving the fabric packet from the fabric, means for reading one or more instructions from a memory of the receiving processing element at an address based at least in part on the virtual channel specifier, and means for using at least a portion of the fabric packet payload as an input operand to execute at least one of the one or more instructions; and

wherein the portion of the fabric packet payload comprises one or more data elements and the one or more data elements comprise at least a portion of one of;

a weight of a neural network, an activation of a neural network, a partial sum of activations of a neural network, an error of a neural network, a gradient estimate of a neural network, and a weight update of a neural network.

29. The system of claim 28, wherein the sending processing element, the routing processing elements, and the receiving processing element are fabricated via wafer-scale integration.

30. The system of claim 28, wherein the sending processing element and the receiving processing element implement at least portions of a neuron of a neural network.

31. A system comprising:

means for sending a fabric packet by a sending processing element to a fabric, the fabric packet comprising a virtual channel specifier and a fabric packet payload;

means for routing the fabric packet via the fabric from the sending processing element to a receiving processing element via zero or more routing processing elements, the means for routing operating in accordance with the virtual channel specifier;

in the receiving processing element, means for receiving the fabric packet from the fabric, means for reading one or more instructions from a memory of the receiving processing element at an address based at least in part on the virtual channel specifier, and means for using at least a portion of the fabric packet payload as an input operand to execute at least one of the one or more instructions; and

wherein the one or more instructions implement at least a portion of one or more of;

computing an activation of a neural network, computing a partial sum of activations of a neural network, computing an error of a neural network, computing a gradient estimate of a neural network, and updating a weight of a neural network.

32. The system of claim 31, wherein the means for routing is a first means for routing, the receiving processing element is a first receiving processing element, the zero or more routing processing elements are first zero or more routing processing elements, the means for receiving is a first means for receiving, the means for reading is a first means for reading, the means for using is a first means for using, and the one or more instructions are first one or more instructions;

and further comprising a second means for routing the fabric packet via the fabric from the sending processing element to at least a second receiving processing element via second zero or more routing processing elements, the second means for routing in accordance with the virtual channel specifier, and in the second receiving processing element a second means for receiving the fabric packet from the fabric, a second means for reading a second one or more instructions from a memory of the second receiving processing element at an address based at least in part on the virtual channel specifier, and a second means for using at least a portion of the fabric packet payload as an input operand to execute at least one of the second one or more instructions.

33. The system of claim 31, wherein the sending processing element, the routing processing elements, and the receiving processing element are fabricated via wafer-scale integration.

34. The system of claim 31, wherein the sending processing element and the receiving processing element implement at least portions of a neuron of a neural network.

35. A system comprising:

means for sending a fabric packet by a sending processing element to a fabric, the fabric packet comprising a virtual channel specifier and a fabric packet payload;

means for routing the fabric packet via the fabric from the sending processing element to a receiving processing element via zero or more routing processing elements, the means for routing operating in accordance with the virtual channel specifier;

in the receiving processing element, means for receiving the fabric packet from the fabric, means for reading one or more instructions from a memory of the receiving processing element at an address based at least in part on the virtual channel specifier, and means for using at least a portion of the fabric packet payload as an input operand to execute at least one of the one or more instructions; and

wherein a virtual channel associated with the virtual channel specifier is used for communicating at least one of control and data associated with one or more of;

computing an activation of a neural network, computing a partial sum of activations of a neural network, computing an error of a neural network, computing a gradient estimate of a neural network, and updating a weight of a neural network.

36. The system of claim 35, wherein the virtual channel specifier is one of a plurality of virtual channel specifiers, each of the plurality of virtual channel specifiers associated with a respective set of one or more sets of fabric packets, and the means for receiving comprises means for associating the fabric packet with the respective set associated with the virtual channel specifier.

37. The system of claim 35, wherein the sending processing element, the routing processing elements, and the receiving processing element are fabricated via wafer-scale integration.

38. The system of claim 35, wherein the sending processing element and the receiving processing element implement at least portions of a neuron of a neural network.

39. A system comprising:

a sending processing element, one or more routing elements, and a receiving processing element, all comprising respective couplings to a fabric, and the receiving processing element further comprising a memory, an instruction fetcher, a data sequencer, and an instruction execution unit;

wherein the sending processing element is enabled to transmit a fabric packet comprising a virtual channel specifier and a fabric packet payload to the fabric via the coupling of the sending processing element;

wherein the one or more routing elements are enabled to route the fabric packet in accordance with the virtual channel specifier to the receiving processing element via the couplings of the one or more routing elements;

wherein the receiving processing element is enabled to receive the fabric packet via the coupling of the receiving processing element;

wherein the memory is enabled to store one or more instructions at an address based at least in part on the virtual channel specifier;

wherein the instruction fetcher is enabled to read the one or more instructions at the address;

wherein the data sequencer is enabled to provide a portion of the fabric packet payload to the instruction execution unit;

wherein the instruction execution unit is enabled to execute at least one of the one or more instructions;

wherein the virtual channel specifier is one of a plurality of virtual channel specifiers, each of the plurality of virtual channel specifiers associated with a respective set of one or more sets of fabric packets, and the receiving comprises associating the fabric packet with the respective set associated with the virtual channel specifier; and

further comprising respective block/unblock state maintained for each of the virtual channel specifiers; and

wherein the block/unblock state of a particular one of the virtual channel specifiers is set to a block state in response to a block instruction specifying the particular one of the virtual channel specifiers and wherein the block/unblock state of the particular one of the virtual channel specifiers is set to an unblock state in response to an unblock instruction specifying the particular one of the virtual channel specifiers.

40. The system of claim 39, further comprising a scheduler of the receiving processing element;

and wherein at least one of the reading and the providing are responsive to the receiving processing element selecting the fabric packet for processing, and the selecting is performed at least in part by the scheduler.

41. The system of claim 40, wherein the selecting comprises selecting the virtual channel specifier from the virtual channel specifiers in the unblock state, and the fabric packet is older than any other fabric packets associated with the respective set associated with the virtual channel specifier.

42. The system of claim 41, wherein the selecting the virtual channel specifier is based at least in part upon the virtual channel specifier and a virtual channel specifier of a previously selected fabric packet and in accordance with a round-robin scheduling policy.

43. The system of claim 41, wherein the selecting the virtual channel specifier is based at least in part upon the virtual channel specifier and a virtual channel specifier of a previously selected fabric packet and in accordance with selecting the virtual channel specifier of the previously selected fabric packet until a fabric packet comprising a control specifier set to a predetermined value is selected.

44. The system of claim 41, wherein the selecting the virtual channel specifier is based at least in part upon the virtual channel specifier being one of one or more predetermined prioritized virtual channel specifiers of the plurality of virtual channel specifiers.

45. The system of claim 39, wherein the sending processing element and the receiving processing element implement at least portions of a neuron of a neural network.

46. The system of claim 39, wherein the sending processing element, the routing elements, and the receiving processing element are fabricated via wafer-scale integration.

47. A system comprising:

a sending processing element, one or more routing elements, and a receiving processing element, all comprising respective couplings to a fabric, and the receiving processing element further comprising a memory, an instruction fetcher, a data sequencer, and an instruction execution unit;

wherein the sending processing element is enabled to transmit a fabric packet comprising a virtual channel specifier and a fabric packet payload to the fabric via the coupling of the sending processing element;

wherein the one or more routing elements are enabled to route the fabric packet in accordance with the virtual channel specifier to the receiving processing element via the couplings of the one or more routing elements;

wherein the receiving processing element is enabled to receive the fabric packet via the coupling of the receiving processing element;

wherein the memory is enabled to store one or more instructions at an address based at least in part on the virtual channel specifier;

wherein the instruction fetcher is enabled to read the one or more instructions at the address;

wherein the data sequencer is enabled to provide a portion of the fabric packet payload to the instruction execution unit;

wherein the instruction execution unit is enabled to execute at least one of the one or more instructions; and

wherein the portion of the fabric packet payload comprises one or more data elements and the one or more data elements comprise at least a portion of one of;

a weight of a neural network, an activation of a neural network, a partial sum of activations of a neural network, an error of a neural network, a gradient estimate of a neural network, and a weight update of a neural network.

48. The system of claim 47, wherein the sending processing element, the routing elements, and the receiving processing element are fabricated via wafer-scale integration.

49. The system of claim 47, wherein the sending processing element and the receiving processing element implement at least portions of a neuron of a neural network.

50. A system comprising:

a sending processing element, one or more routing elements, and a receiving processing element, all comprising respective couplings to a fabric, and the receiving processing element further comprising a memory, an instruction fetcher, a data sequencer, and an instruction execution unit;

wherein the sending processing element is enabled to transmit a fabric packet comprising a virtual channel specifier and a fabric packet payload to the fabric via the coupling of the sending processing element;

wherein the one or more routing elements are enabled to route the fabric packet in accordance with the virtual channel specifier to the receiving processing element via the couplings of the one or more routing elements;

wherein the receiving processing element is enabled to receive the fabric packet via the coupling of the receiving processing element;

wherein the memory is enabled to store one or more instructions at an address based at least in part on the virtual channel specifier;

wherein the instruction fetcher is enabled to read the one or more instructions at the address;

wherein the data sequencer is enabled to provide a portion of the fabric packet payload to the instruction execution unit;

wherein the instruction execution unit is enabled to execute at least one of the one or more instructions; and

wherein the one or more instructions implement at least a portion of one or more of;

computing an activation of a neural network, computing a partial sum of activations of a neural network, computing an error of a neural network, computing a gradient estimate of a neural network, and updating a weight of a neural network.

51. The system of claim 50, wherein the sending processing element, the routing elements, and the receiving processing element are fabricated via wafer-scale integration.

52. The system of claim 50, wherein the sending processing element and the receiving processing element implement at least portions of a neuron of a neural network.

53. A system comprising:

a sending processing element, one or more routing elements, and a receiving processing element, all comprising respective couplings to a fabric, and the receiving processing element further comprising a memory, an instruction fetcher, a data sequencer, and an instruction execution unit;

wherein the sending processing element is enabled to transmit a fabric packet comprising a virtual channel specifier and a fabric packet payload to the fabric via the coupling of the sending processing element;

wherein the one or more routing elements are enabled to route the fabric packet in accordance with the virtual channel specifier to the receiving processing element via the couplings of the one or more routing elements;

wherein the receiving processing element is enabled to receive the fabric packet via the coupling of the receiving processing element;

wherein the memory is enabled to store one or more instructions at an address based at least in part on the virtual channel specifier;

wherein the instruction fetcher is enabled to read the one or more instructions at the address;

wherein the data sequencer is enabled to provide a portion of the fabric packet payload to the instruction execution unit;

wherein the instruction execution unit is enabled to execute at least one of the one or more instructions; and

wherein a virtual channel associated with the virtual channel specifier is used for communicating at least one of control and data associated with one or more of;

computing an activation of a neural network, computing a partial sum of activations of a neural network, computing an error of a neural network, computing a gradient estimate of a neural network, and updating a weight of a neural network.

54. The system of claim 53, wherein the virtual channel specifier is one of a plurality of virtual channel specifiers, each of the plurality of virtual channel specifiers associated with a respective set of one or more sets of fabric packets, and the receiving comprises associating the fabric packet with the respective set associated with the virtual channel specifier.

55. The system of claim 53, wherein the sending processing element comprises at least one of the one or more routing elements.

56. The system of claim 53, wherein the sending processing element, the routing elements, and the receiving processing element are fabricated via wafer-scale integration.

57. The system of claim 53, wherein the sending processing element and the receiving processing element implement at least portions of a neuron of a neural network.