MULTICAST NETWORK AND MEMORY TRANSFER OPTIMIZATIONS FOR NEURAL NETWORK HARDWARE ACCELERATION

• US 20170337468A1
• Filed: 05/05/2017
• Published: 11/23/2017
• Est. Priority Date: 05/07/2016
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: 1026 Labs, Inc. (Intel Corporation)
• Inventors: Bruestle, Jeremy, Ng, Choong
• Granted Patent: US 11,120,329 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 12/06 Addressing a physical block of locations, e.g. base addressing, module addressing, memory dedication G06F12/08 takes precedence
• CPC Class Codes: G06F 9/345 of multiple operands or results addressing multiple banks G06F12/06
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: H04L 15/00 Apparatus or local circuits for transmitting or receiving dot-and-dash codes, e.g. Morse code teaching apparatus therefor G09B; telegraph tapping keys H01H21/86
• CPC Class Codes: H04L 49/1507 Distribute and route fabrics, e.g. sorting-routing or Batcher-Banyan

Abstract

Neural network specific hardware acceleration optimizations are disclosed, including an optimized multicast network and an optimized DRAM transfer unit to perform in constant or linear time. The multicast network is a set of switch nodes organized into layers and configured to operate as a Bene{hacek over (s)} network. Configuration data may be accessed by all switch nodes in the network. Each layer is configured to perform a Bene{hacek over (s)} network transformation of the -previous layer within a computer instruction. Since the computer instructions are pipelined, the entire network of switch nodes may be configured in constant or linear time. Similarly a DRAM transfer unit configured to access memory in strides organizes memory into banks indexed by prime or relatively prime number amounts. The index value is selected as not to cause memory address collisions. Upon receiving a memory specification, the DRAM transfer unit may calculate out strides thereby accessing an entire tile of a tensor in constant or linear time.

Claims

1. A system to configure input data for multicast to data receivers, comprising:

a computer memory storing configuration data in a known address and input data;

a set of switch nodes configured into a Bene{hacek over (s)} multicast network ordered into multiple layers, each layer with a plurality of switch nodes, the first layer proximate to the input data and the last layer proximate to a data receiver, each switch node storing received input data comprised in a plurality of entries, a configuration indicator, and a controller indicator,

the configuration indicator to specify whether to perform a broadcast mode whether input data is to be forwarded according to the configuration data, or a passthru mode wherein input data is to be forwarded regardless of the configuration data, and

the controller indicator to specify whether to update at least one switch node entry; and

a set of control registers communicatively connected to each switch node in the set of switch nodes, the set of registers configured to store received configuration data, such that the set of nodes is configured within two operations, the first operation to read the configuration data from the known address in the computer memory, and a second operation to populate the set of control registers, and subsequent operations comprising multicast operations by the set of switch nodes according to the populated set of control registers.

2. The system of claim 1, wherein the data receiver is a set of execution units.

3. The system of claim 2, comprising:

a second set of switch nodes configured into a Bene{hacek over (s)} multicast network ordered into multiple layers, each layer with a plurality of switch nodes, the first layer proximate to the input data and the last layer proximate to the set of execution units to receive data, each switch node in the second set of switch nodes storing received input data comprised in a plurality of entries, a configuration indicator, and a controller indicator,

the configuration indicator to specify whether to perform a broadcast mode whether input data is to be forwarded according to the configuration data, or a passthru mode wherein input data is to be forwarded regardless of the configuration data, and

the controller indicator to specify whether to update at least one switch node entry; and

wherein each switch node of the second set of switch nodes is communicatively connected to the control registers.

4. The system of claim 3, wherein the last layer of the first set of switch nodes connects to the set of execution units in a different permutation than the last layer of the second set of switch nodes.

5. The system of claim 4, wherein the permutation used by the last layer of the first set of switch nodes to connect to the set of execution units is modulo, and the permutation used by the last layer of the second set of switch nodes to connect to the set of execution units is grouped fanout.

6. The system of claim 1, wherein the set of switch nodes is configured to operate via a pipeline containing a series of processor instructions, wherein a first processor instruction in the pipeline is to configure the set of control registers, a second processor instruction in the pipeline is to execute a Bene{hacek over (s)} multicast for the first layer of switch nodes, and each subsequent processor instruction is to execute a Bene{hacek over (s)} multicast for each respective subsequent layer of switch nodes, such that the configuration of the set of switch nodes may be performed within the first and the second processor instructions in the pipeline.

7. The system of claim 6, wherein the processor instructions in the pipeline to execute a Bene{hacek over (s)} multicast for a layer of switch nodes is specific to the topology of the switch nodes configured into the Bene{hacek over (s)} network.

8. The system of claim 6, wherein the pipeline contains no-ops to guarantee the pipeline will not stall during configuring the set of switch nodes.

9. The system of claim 1, wherein the set of switch nodes is configured to interpret the configuration indicator being set to passthru as operating via a pipeline containing a series of processor instructions, a first processor instruction in the pipeline is to execute a Bene{hacek over (s)} multicast for the first layer of switch nodes, and each subsequent processor instruction is to execute a Bene{hacek over (s)} multicast for each respective subsequent layer of switch nodes, such that the configuration of the set of switch nodes may be performed within the first processor instruction in the pipeline.

10. A system to deterministically transfer partitions of contiguous computer readable data in constant time, comprising:

a computer readable memory organized into D banks, containing contiguous data containing a plurality of data elements of size M which are constituent data elements of a vector with N data elements, the data elements starting at an offset address O; and

a modulo address generator configured to generate the addresses of the data elements of a vector with i data elements stored in the computer readable memory, and the modulo address generator comprised of at least one forward permutaton configured to permute data elements with addresses of the form O+M*i where 0←

i<

N.

11. The system of claim 10, comprising:

an enqueuing controller, communicatively coupled to the modulo address generator;

a plurality of address queues, each configured to store a plurality of memory addresses respectively, the address queues coupled to the enqueuing controllers;

a plurality of data decoders, a decoder coupled to and corresponding to each address queue respectively, the plurality of data decoders configured to decode the a memory address to retrieve the data stored at the memory address;

a plurality of data queues, each configured to store a plurality of data elements respectively, the data queues coupled and corresponding to each data decoder respectively; and

a dequeuing controller communicatively coupled to the plurality of data queues, and communicatively coupled to the enqueuing controller via a control queue configured to contain control data to ensure synchronization.

12. The system of claim 11, comprising:

a reverse permutaton, communicatively coupled to the dequeuing controller, configured to restore any rotations performed.

13. The system of claim 11, wherein the number of banks D relatively prime to the stride of the data elements M, and the number of data elements N in a vector is less than D.

14. The system of claim 11, wherein the number of banks D is prime and the stride of the data elements M is not a strict multiple of D.

15. The system of claim 11, wherein the control data in the control queue includes information as to the number of rotations a permutaton is to perform based at least on a generator integer g, and provided at compile time.

16. The system of claim 11, comprising a hardware indicator configured to perform at least one of the following:

track which element in a memory bank to access;

or

track if there are to be multiple accesses of a memory bank.

17. A method to configure input data for multicast to data receivers, comprising:

retrieving configuration data retrieved from a known address in a computer memory,

the configuration data to configure a set of switch nodes configured into a Bene{hacek over (s)} multicast network ordered into multiple layers, each layer with a plurality of switch nodes, the first layer proximate to the input data and the last layer proximate to a data receiver,

each switch node storing received input data comprised in a plurality of entries, a configuration indicator, and a controller indicator,

wherein the configuration data is comprised of a configuration indicator to specify whether to perform a broadcast mode whether input data is to be forwarded according to the configuration data, or a passthru mode wherein input data is to be forwarded regardless of the configuration data, and a controller indicator to specify whether to update at least one switch node entry;

storing the retrieved configuration data in a set of control registers;

populating the configuration indicator and controller indicator at each switch node from the set of control registers; and

starting a Bene{hacek over (s)} multicast operation at the first level of switch nodes based at least on the configuration indicators and controller indicators of the first level switch nodes.

18. The method of claim 17, comprising:

completing the Bene{hacek over (s)} multicast operations sequentially through each level of switch nodes;

performing at the last level of switch nodes a Bene{hacek over (s)} multicast to data receivers.

19. A method to access computer readable data stored in a computer readable memory in strides, comprising:

receiving a parameter o where o=O% D where O is the initial offset address for a computer readable memory storing a number of data elements in contiguous memory and organized into D banks;

receiving a parameter r which specifies the number of rotations to perform for a cyclic group less than D, wherein r is based at least on the discrete log of generator g;

performing a rotation of data in the data in data banks D−

1 to D; and

performing a rotation of the data in data banks in O through D.

20. A method to access computer readable data stored in a computer readable memory in strides, comprising:

receiving at a modulo address generator, a memory address and a length;

generating by the modulo address generator, a set of memory addresses corresponding to data elements stored in a computer readable memory separated by strides;

receiving at an enqueuing controller the generated set of memory addresses;

queuing by the enqueuing controller addresses of data elements into corresponding address queues respectively, and concurrently queuing by the enqueuing controller control data in a control queue;

decoding the memory addresses of the data elements into data at a decoder and queuing into a corresponding data queue respectively;

at a dequeuing controller receiving data elements from the data queues and forwarding the received data elements to a reverse permutaton, based at least on control data received from the control queue;

restoring via the reverse permutaton, the data received from the dequeuing controller; and

forwarding at the reverse permutaton, the restored data to data out.