NEURAL NETWORK PROCESSOR BASED ON APPLICATION SPECIFIC SYNTHESIS SPECIALIZATION PARAMETERS

0Associated
Cases 
0Associated
Defendants 
0Accused
Products 
1Forward
Citation 
0
Petitions 
1
Assignment
First Claim
1. A method, implemented by a processor, for synthesizing a neural network processor comprising a plurality of tile engines, wherein each of the plurality of tile engines is configured to process matrix elements and vector elements, the method comprising:
 using the processor analyzing a neural network model corresponding to an application to determine;
(1) a first minimum number of units required to express a shared exponent value required to satisfy a first precision requirement corresponding to each of the matrix elements and corresponding to each of the vector elements, (2) a second minimum number of units required to express a first mantissa value required to satisfy a second precision requirement corresponding to the each of the matrix elements, and (3) a third minimum number of units required to express a second mantissa value required to satisfy a third precision requirement corresponding to the each of the vector elements;
obtaining code representative of at least a portion of at least one hardware node for implementing the neural network processor;
obtaining a synthesis model comprising a plurality of synthesis specialization parameters including;
(1) a first synthesis specialization parameter corresponding to a first native dimension of the each of the matrix elements, (2) a second synthesis specialization parameter corresponding to a second native dimension of the each of the vector elements, and (3) a third synthesis specialization parameter corresponding to a number of the plurality of tile engines, wherein each of a first value corresponding to the first synthesis specialization parameter, a second value corresponding to the second synthesis specialization parameter, and a third value corresponding to the third synthesis specialization parameter is selected to meet or exceed a performance metric associated with the at least one hardware node; and
using the processor modifying the code, based on at least the first minimum number of units, the second minimum number of units, the third minimum number of units and at least the first value and the second value, to generate a modified version of the code and storing a modified version of the code.
1 Assignment
0 Petitions
Accused Products
Abstract
Neural network processors that have been customized based on application specific synthesis specialization parameters and related methods are described. Certain example neural network processors and methods described in the present disclosure expose several major synthesis specialization parameters that can be used for specializing a microarchitecture instance of a neural network processor to specific neural network models including: (1) aligning the native vector dimension to the parameters of the model to minimize padding and waste during model evaluation, (2) increasing lane widths to drive up intrarowlevel parallelism, or (3) increasing matrix multiply tiles to exploit submatrix parallelism for large neural network models.
1 Citation
Reconfigurable multiprecision integer dotproduct hardware accelerator for machinelearning applications  
Patent #
US 10,642,614 B2
Filed 09/29/2018

Current Assignee
Intel Corporation

Sponsoring Entity
Intel Corporation

No References
20 Claims
 1. A method, implemented by a processor, for synthesizing a neural network processor comprising a plurality of tile engines, wherein each of the plurality of tile engines is configured to process matrix elements and vector elements, the method comprising:
using the processor analyzing a neural network model corresponding to an application to determine;
(1) a first minimum number of units required to express a shared exponent value required to satisfy a first precision requirement corresponding to each of the matrix elements and corresponding to each of the vector elements, (2) a second minimum number of units required to express a first mantissa value required to satisfy a second precision requirement corresponding to the each of the matrix elements, and (3) a third minimum number of units required to express a second mantissa value required to satisfy a third precision requirement corresponding to the each of the vector elements;obtaining code representative of at least a portion of at least one hardware node for implementing the neural network processor; obtaining a synthesis model comprising a plurality of synthesis specialization parameters including;
(1) a first synthesis specialization parameter corresponding to a first native dimension of the each of the matrix elements, (2) a second synthesis specialization parameter corresponding to a second native dimension of the each of the vector elements, and (3) a third synthesis specialization parameter corresponding to a number of the plurality of tile engines, wherein each of a first value corresponding to the first synthesis specialization parameter, a second value corresponding to the second synthesis specialization parameter, and a third value corresponding to the third synthesis specialization parameter is selected to meet or exceed a performance metric associated with the at least one hardware node; andusing the processor modifying the code, based on at least the first minimum number of units, the second minimum number of units, the third minimum number of units and at least the first value and the second value, to generate a modified version of the code and storing a modified version of the code.  View Dependent Claims (2, 3, 4, 5, 6, 7)
 8. A system comprising:
a processor; and a memory comprising;
(1) code representative of at least a portion of at least one hardware node for implementing the neural network processor comprising a plurality of tile engines, wherein each of the plurality of tile engines is configured to process matrix elements and vector elements, (2) a synthesis model comprising a plurality of synthesis specialization parameters including;
(a) a first synthesis specialization parameter corresponding to a first native dimension of the each of the matrix elements and, (b) a second synthesis specialization parameter corresponding to a second native dimension of the each of the vector elements, and (c) a third synthesis specialization parameter corresponding to a number of the plurality of tile engines, wherein each of a first value corresponding to the first synthesis specialization parameter, a second value corresponding to the second synthesis specialization parameter, and a third value corresponding to the third synthesis specialization parameter is selected to meet or exceed a performance metric associated with the at least one hardware node, and (3) instructions for synthesizing a neural network processor comprising a plurality of tile engines, wherein each of the plurality of tile engines is configured to process matrix elements and vector elements, the instructions configured to;using the processor analyze a neural network model corresponding to an application to determine;
(1) a first minimum number of units required to express a shared exponent value required to satisfy a first precision requirement corresponding to each of the matrix elements and corresponding each of the vector elements, (2) a second minimum number of units required to express a first mantissa value required to satisfy a second precision requirement corresponding to the each of the matrix elements, and (3) a third minimum number of units required to express a second mantissa value required to satisfy a third precision requirement corresponding to the each of the vector elements, andusing the processor modify the code, based on at least the first minimum number of units, the second minimum number of units, the third minimum number of units and at least the first value, the second value, and the third value, to generate a modified version of the code and store a modified version of the code.  View Dependent Claims (9, 10, 11, 12, 13, 14)
 15. A method, implemented by a processor, for synthesizing a neural network processor comprising a plurality of tile engines, wherein each of the plurality of tile engines is configured to process matrix elements and vector elements, and wherein each of the plurality of tile engines comprises a plurality of dot product units and wherein each of the dot product units is configured to receive the matrix elements from a matrix register file, the method comprising:
using the processor analyzing a neural network model corresponding to an application to determine;
(1) a first minimum number of units required to express a shared exponent value required to satisfy a first precision requirement corresponding to each of the matrix elements and corresponding to each of the vector elements, (2) a second minimum number of units required to express a first mantissa value required to satisfy a second precision requirement corresponding to the each of the matrix elements, and (3) a third minimum number of units required to express a second mantissa value required to satisfy a third precision requirement corresponding to the each of the vector elements;obtaining code representative of at least a portion of at least one hardware node for implementing the neural network processor; obtaining a synthesis model comprising a plurality of synthesis specialization parameters including;
(1) a first synthesis specialization parameter corresponding to whether the matrix register file is private to each one of the plurality of tile engines or whether the matrix register file is shared among the plurality of tile engines and (2) a second synthesis specialization parameter corresponding to whether each of the plurality of dot product units comprises an addreduction tree; andusing the processor modifying the code, based on at least the first minimum number of units, the second minimum number of units, the third minimum number of units and at least the first synthesis specialization parameter and the second synthesis specialization parameter, and storing a modified version of the code.  View Dependent Claims (16, 17, 18, 19, 20)
1 Specification
Neural network technology is used to perform complex tasks such as reading comprehension, language translation, image recognition, or speech recognition. Machine learning services, such as those based on Recurrent Neural Networks (RNNs), Convolution Neural Networks (CNNs), Long Short Term Memory (LSTM) neural networks, or Gated Recurrent Units (GRUs) have been deployed to perform such complex tasks. While these types of neural networks have been deployed, there is a need for continued improvement in the underlying architecture and corresponding instructions to perform these complex tasks.
In one example, the present disclosure relates to a method, implemented by a processor, for synthesizing a neural network processor comprising a plurality of tile engines, where each of the plurality of tile engines is configured to process matrix elements and vector elements. The method may include using the processor analyzing a neural network model corresponding to an application to determine: (1) a first minimum number of units required to express a shared exponent value required to satisfy a first precision requirement corresponding to each of the matrix elements and corresponding to each of the vector elements, (2) a second minimum number of units required to express a first mantissa value required to satisfy a second precision requirement corresponding to the each of the matrix elements, and (3) a third minimum number of units required to express a second mantissa value required to satisfy a third precision requirement corresponding to the each of the vector elements.
The method may further include obtaining code representative of at least a portion of at least one hardware node for implementing the neural network processor. The method may further include obtaining a synthesis model comprising a plurality of synthesis specialization parameters including: (1) a first synthesis specialization parameter corresponding to a first native dimension of the each of the matrix elements, (2) a second synthesis specialization parameter corresponding to a second native dimension of the each of the vector elements, and (3) a third synthesis specialization parameter corresponding to a number of the plurality of tile engines, where each of a first value corresponding to the first synthesis specialization parameter, a second value corresponding to the second synthesis specialization parameter, and a third value corresponding to the third synthesis specialization parameter is selected to meet or exceed a performance metric associated with the at least one hardware node.
The method may further include using the processor modifying the code, based on at least the first minimum number of units, the second minimum number of units, the third minimum number of units and at least the first value and the second value, to generate a modified version of the code and storing a modified version of the code.
In another example, the present disclosure relates to a system comprising a processor and a memory comprising: (1) code representative of at least a portion of at least one hardware node for implementing the neural network processor comprising a plurality of tile engines, where each of the plurality of tile engines is configured to process matrix elements and vector elements, (2) a synthesis model comprising a plurality of synthesis specialization parameters including: (a) a first synthesis specialization parameter corresponding to a first native dimension of the each of the matrix elements and, (b) a second synthesis specialization parameter corresponding to a second native dimension of the each of the vector elements, and (c) a third synthesis specialization parameter corresponding to a number of the plurality of tile engines, where each of a first value corresponding to the first synthesis specialization parameter, a second value corresponding to the second synthesis specialization parameter, and a third value corresponding to the third synthesis specialization parameter is selected to meet or exceed a performance metric associated with the at least one hardware node, and (3) instructions for synthesizing a neural network processor comprising a plurality of tile engines, where each of the plurality of tile engines is configured to process matrix elements and vector elements.
The instructions may be configured to using the processor analyze a neural network model corresponding to an application to determine: (1) a first minimum number of units required to express a shared exponent value required to satisfy a first precision requirement corresponding to each of the matrix elements and corresponding each of the vector elements, (2) a second minimum number of units required to express a first mantissa value required to satisfy a second precision requirement corresponding to the each of the matrix elements, and (3) a third minimum number of units required to express a second mantissa value required to satisfy a third precision requirement corresponding to the each of the vector elements.
The instructions may further be configured to using the processor modify the code, based on at least the first minimum number of units, the second minimum number of units, the third minimum number of units and at least the first value, the second value, and the third value, to generate a modified version of the code and store a modified version of the code.
In yet another example, the present disclosure relates to a method for synthesizing a neural network processor comprising a plurality of tile engines, where each of the plurality of tile engines is configured to process matrix elements and vector elements, and where each of the plurality of tile engines comprises a plurality of dot product units and where each of the dot product units is configured to receive the matrix elements from a matrix register file.
The method may include using the processor analyzing a neural network model corresponding to an application to determine: (1) a first minimum number of units required to express a shared exponent value required to satisfy a first precision requirement corresponding to each of the matrix elements and corresponding to each of the vector elements, (2) a second minimum number of units required to express a first mantissa value required to satisfy a second precision requirement corresponding to the each of the matrix elements, and (3) a third minimum number of units required to express a second mantissa value required to satisfy a third precision requirement corresponding to the each of the vector elements.
The method may further include obtaining code representative of at least a portion of at least one hardware node for implementing the neural network processor. The method may further include obtaining a synthesis model comprising a plurality of synthesis specialization parameters including: (1) a first synthesis specialization parameter corresponding to whether the matrix register file is private to each one of the plurality of tile engines or whether the matrix register file is shared among the plurality of tile engines and (2) a second synthesis specialization parameter corresponding to whether each of the plurality of dot product units comprises an addreduction tree.
The method may further include using the processor modifying the code, based on at least the first minimum number of units, the second minimum number of units, the third minimum number of units and at least the first synthesis specialization parameter and the second synthesis specialization parameter, and storing a modified version of the code.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
The present disclosure is illustrated by way of example and is not limited by the accompanying figures, in which like references indicate similar elements. Elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale.
Examples disclosed in the present example relate to neural network processors that have been customized based on application specific synthesis specialization parameters. Certain example neural network processors described in the present disclosure expose several major parameters that can be used for specializing a microarchitecture instance of a neural network processor to specific models including: (1) aligning the native vector dimension to parameters of the model tends to minimize padding and waste during model evaluation, (2) increasing lane widths can be used to drive up intrarowlevel parallelism, and (3) increasing matrix multiply tiles that can exploit submatrix parallelism for large models.
In addition, certain examples of neural network processors described in the present disclosure employ a narrow precision block floating point (BFP) format that shares exponents across groups of floating point numbers at the native vector level. In these examples, using a narrow form of BFP, mantissas could be trimmed to as low as 23 bits without negatively impacting the accuracy of model training of deep neural networks (DNN) models and large ImageNet models (e.g., ResNet50). With shared exponents, the cost of implementing floating point formatbased computations may drop to similar levels as the cost of implementing fixed integer arithmetic. As part of the synthesis specialization for FPGAbased neural network processors, narrow precision may be further exploited. As an example, packing 2 or 3 bit multiplications into DSP blocks combined with soft logic instantiations may permit the deployment of a large number of multiplyaccumulate (MAC) units on an FPGA.
Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) are both widely used computational kernels in mainstream machine learning. CNNs and RNNs may be efficiently expressed in terms of matrixvector multiplication, however the parallelism and data structures inherent to each differs significantly. Therefore, it is challenging to produce a single teraflop scale computer architecture that efficiently computes both CNNs and RNNs This problem is compounded when real time latency requirements are placed on the design. As a result, previous solutions have specialized for CNNs or RNNs without prioritizing strong performance on both. Certain examples disclosed in the present disclosure relate to using system, methods, and components that provide for efficient computation for both CNNs and RNNs.
As an example, the present disclosure describes a neural network processor that leverages the parallelism between individual output activations in a CNN to perform a limited form of matrixmatrix multiplication within an individual CNN evaluation. This parallelism is mapped onto a circuit in the form of an array of quasiindependent matrixvector multiplication tile engines that receive the same matrix data but different vector data. This approach allows for high utilization at batch=1 for CNN inputs, which in turn delivers high throughput at low latency. One of the ways to enable this approach is the use of a multiport vector register file (VRF), which allows many simultaneous vectorized reads and writes to a shared memory space. This approach is also enabled by a CNNaware instruction set architecture (ISA) that provides an informationdense expression of CNNs in the same assembly level code that can be used to express RNNs.
The neural network processors described in this disclosure may be implemented using portions or combinations of Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Erasable and/or Complex programmable logic devices (PLDs), Programmable Array Logic (PAL) devices, and Generic Array Logic (GAL) devices. An image file may be used to configure or reconfigure the neural network processors, such as FPGAs. The image file or similar file or program may be delivered via a network link or a local link (e.g., PCIe) from a host CPU. Information included in an image file can be used to program hardware blocks of a neural network processor (e.g., logic blocks and reconfigurable interconnects of an FPGA) to implement desired functionality. Desired functionality can be implemented to support any service that can be offered via a combination of computing, networking, and storage resources such as via a data center or other infrastructure for delivering a service.
In one example, neural network processors (e.g., FPGAs) or groups of such neural network processors may be coupled to each other via a low latency network. A converged platform leveraging hundreds to thousands of such neural network processors (e.g., FPGAs) may advantageously offer: (1) significantly reduced training times from exploiting parallelism across hundreds of thousands of nodes, (2) enabling new training scenarios such as online learning insitu on live data, and (3) training models of unprecedented scale while leveraging flexible and fungible homogeneous FPGA resources in a hyperscale datacenter spanning hundreds of thousands of servers. In one example, such advantages may be obtained by exploiting unconventional data representations that may leverage the architecture of neural network processors, such as FPGAs.
The described aspects can also be implemented in cloud computing environments. Cloud computing may refer to a model for enabling ondemand network access to a shared pool of configurable computing resources. For example, cloud computing can be employed in the marketplace to offer ubiquitous and convenient ondemand access to the shared pool of configurable computing resources. The shared pool of configurable computing resources can be rapidly provisioned via virtualization and released with low management effort or service provider interaction, and then scaled accordingly. A cloud computing model can be composed of various characteristics such as, for example, ondemand selfservice, broad network access, resource pooling, rapid elasticity, measured service, and so forth. A cloud computing model may be used to expose various service models, such as, for example, Hardware as a Service (“HaaS”), Software as a Service (“SaaS”), Platform as a Service (“PaaS”), and Infrastructure as a Service (“IaaS”). A cloud computing model can also be deployed using different deployment models such as private cloud, community cloud, public cloud, hybrid cloud, and so forth.
Machine learning services, such as those based on Recurrent Neural Networks (RNNs), Convolution Neural Networks (CNNs), Long Short Term Memory (LSTM) neural networks, or Gated Recurrent Units (GRUs) may be implemented using the neural network processors described in this disclosure. In one example, the servicerelated content or other information, such as words, sentences, images, videos, or other such content/information may be translated into a vector representation. The vector representation may correspond to techniques such as RNN, CNN, LSTM, or GRU. The deep learning models may be trained offline before service initialization and then may be deployed using the systems and neural network processors described in this disclosure.
In one example, the neural network model may comprise of many layers and each layer may be encoded as matrices or vectors of weights expressed in the form of coefficients or constants that have been obtained via offline training of a neural network. Programmable hardware logic blocks in the nodes may process the matrices or vectors to perform various operations, including multiply, add, and other operations against input vectors representing encoded information related to the service. In one example, the matrices or vectors of weights may be partitioned and pinned across multiple nodes by using techniques such as graph partitioning. As part of this process, a large neural network may be translated into an intermediate representation (e.g., a graph) and then the intermediate representation may be carved into smaller representations (e.g., subgraphs) and each of the matrices of weights corresponding to each subgraph may be pinned to a node'"'"'s onchip memories. In one example, the models may be translated into fixedsize matrices and vectors. This way, the nodes'"'"' resources may operate on the fixedsize matrices and vectors in parallel.
Taking the LSTM example, an LSTM network may comprise a sequence of repeating RNN layers or other types of layers. Each layer of the LSTM network may consume an input at a given time step, e.g., a layer'"'"'s state from a previous time step, and may produce a new set of outputs or states. In case of using the LSTM, a single chunk of content may be encoded into a single vector or multiple vectors. As an example, a word or a combination of words (e.g., a phrase, a sentence, or a paragraph) may be encoded as a single vector. Each chunk may be encoded into an individual layer (e.g., a particular time step) of an LSTM network. An LSTM layer may be described using a set of equations, such as the ones below:
i_{t}=σ(W_{xi}xt++W_{hi}h_{t1}+W_{ci}c_{t1}+b_{i }
f_{t}=σ(W_{xf}x_{t}+W_{hf}h_{t1}+W_{cf}c_{t1}+b_{f})
c_{t}=f_{t}c_{t1}i_{t }tan h(W_{xc}x_{t}+W_{hc}h_{t1}+b_{c})
o_{t}=σ(W_{xo}x_{t}+W_{ho}h_{t1}+W_{co}c_{t}+b_{o})
h_{t}=o_{t }tan h(c_{t})
In this example, inside each LSTM layer, the inputs and hidden states may be processed using a combination of vector operations (e.g., dotproduct, inner product, or vector addition) and nonlinear functions (e.g., sigmoids, hyperbolic, and tangents). In certain cases, the most compute intensive operations may arise from the dot products, which may be implemented using dense matrixvector and matrixmatrix multiplication routines. In one example, the processing of the vector operations and nonlinear functions may be performed in parallel.
In one example, individual neural network processors may send messages comprising packets directly to each other and thus this may allow the partitioning of even a single neural network across multiple neural network processors without incurring unacceptable latencies. For communicating the neural network processors may use a lightweight protocol, including, for example, RDMA. Parallelization could also be performed within a layer of a neural network by splitting neural weights across multiple neural network processors. As an example, a single CNN or RNN model (e.g., including LSTM weight matrices) may be partitioned and processed using neural network processors.
With continued reference to
MVM 110 may include a vector register file (VRF) 112, a matrix register file (MRF) 120, and tile engines (e.g., tile engines 114, 116, and 118). Tile engines may receive input matrix and input vector data from VRF 112. MVM 110 may further include format converters, as needed, including block floating point (BFP) to floating point (FP) converters. In one example, two internal BFP formats may be used by MVM 110 for expressing its input and output: BFP short, for vector and matrix storage, and BFP long for accumulation. In one example of MVM 110, BFP short may use q1.15 fixed point values with a shared 5 bit exponent, and BFP long may use q34.40 fixed point values with a shared 5 bit exponent. In this example, the matrixvector multiplication may result in BFP long, which may be converted back to a floatingpoint format as a final output stage. Thus, the example MVM 110 shown in
The matrix data may be communicated between Matrix DRAM 184 and Matrix Memory manager 170 using M number of channels. Vector memory manager may move vector data to over C number of channels.
With continued reference to
Still referring to
Neural network processor 100 may be used to enable issuance of instructions that can trigger millions of operations using a small number of instructions. As an example, Table 1 below shows instructions corresponding to a fully parameterized LSTM:
Although Table 1 shows a certain number of instructions having a certain format, neural network processor 100 may execute more or fewer instructions having a different format to accomplish the same objectives.
Table 2 below shows how to compute a 1×1 convolution as part of a CNN evaluation.
As shown in the table above, the number of iterations over a chain of instructions for the computation may be specified. Next, as needed, the native dimension of each instruction chain may be scaled by a column scaling factor. And after reading the vector data from the vector register file it may be multiplied with the weights retrieved from the matrix register file. After performing additional operations as required by the CNN evaluation, the output may be provided. As an example, a pointwise Rectified Linear Unit (ReLU) operation may be performed for each element of the vector data.
Table 3 below shows how to compute an N×N convolution as part of a CNN evaluation. The instructions below that are similar to the 1×1 convolution are not described again. The Set2dWindows instruction may be used to set the total window size and then Set1terations instruction may be used to slide that window across the input volume. The *_inc instructions (e.g., v_rd_inc and v_add_inc) may be used to increment the instruction'"'"'s address based on the stride. As an example, a stride of 2 may result in skipping of every other vector in the vector register file that is used to store vector data for operations, such as addition.
MRF 220 may include several matrix register files that may be configured to supply matrix data or elements to dot product units within each tile. Each multiplier may receive one vector element from VRF 210 per cycle and one matrix element from one of the matrix register files per cycle. The matrix elements may be delivered by a dedicated port of the matrix register file positioned adjacent to that multiplier. MRF 220 may be organized as follows: stored matrices may be divided into nativesized tiles and each tile may be stored in only a single tile engine. The matrix stored in a given tile engine may be viewed as an MRF bank. Each dot product unit may be associated with a subbank of the MRF that holds one row of each matrix tile in that MRF bank. Rows may be statically assigned to dot product units, such that the first dot product unit contains the first row of every matrix tile in the MRF bank. Finally, the elements of the row may be interleaved in an SRAM such that the SRAM read port can be directly connected to multiplier lanes by wires alone. The writes to the matrix register file may be handled differently since matrix data for writing to MRF 220 may come from offchip memory, such as DRAM. Although
From an operational point of view, as described above, MVM 200 instantiates a series of matrixvector tiles, each of which are designed to accelerate a nativesized MVM. In turn, each tile engine includes a series of dot product engines. In one example, this may be accomplished using a hierarchical decode and dispatch architecture. Thus, in a case where neural network processor 100 is implemented based on an FPGA, a control processor may be realized using an offtheshelf Nios II/f processor that is paired with custom code. A toplevel scheduler associated with the control processor may receive a stream of instructions that may be grouped in chains. After decoding the instructions, the toplevel scheduler may dispatch distributed control signals to a set of secondlevel schedulers and to another set of secondlevel decoders. These secondlevel schedulers and decoders may dispatch additional distributed control signals to the lowest level decoders. In the example implementation using the Nios processor, the Nios processor may stream T iterations of N instructions into the toplevel scheduler. Next, the toplevel scheduler may dispatch the MVMspecific portion of instructions to a secondlevel scheduler, which may expand operations along the target matrix'"'"'s N rows and N columns. These MVM schedules may be mapped to matrixvector tile engines and the operations may be dispatched to a set of decoders each for the tile engines and their associated vector register files and accumulation units. The set of decoders may generate control signals that fan out into the data plane, with each tile engine dispatcher fanning out to hundreds of dot product units that may read the vector data from the vector register file and write the vector data back to the vector register file.
In certain examples of the present disclosure synthesis specialization techniques may be used to scale up the microarchitecture of neural network processor 100 for a given application. As an example, first, the numeric representation for the application at hand may be analyzed to determine how many multiplyaccumulate resources are available. Second, the multiplyaccumulate resources may be organized into matrixvector multiplication structures based on the available parallelism in the application.
In one example, a neural network model corresponding to an application to be executed by the neural network processor may be analyzed to determine: (1) a first minimum number of bits required to express a shared exponent value required to satisfy a first precision requirement corresponding to a first native dimension of each of the matrix elements and corresponding to a second native dimension of each of the vector elements, (2) a second minimum number of bits required to express a first mantissa value required to satisfy a second precision requirement corresponding to the each of the matrix elements, and (3) a third minimum number of bits required to express a second mantissa value required to satisfy a third precision requirement corresponding to the each of the vector elements. In a specific example, the neural network model for the application may be analyzed and finetuned to determine the narrowest precision representation that can be used for each the vector and matrix values. Narrower precisions require fewer resources to implement, which in turn may advantageously allow more multiplyaccumulate resources to be provisioned in a hardware node implementing the processor. In one example, neural network processor 100 may express numbers using a block floating point format where the block size is equal to the native dimension. Therefore, in this example, each native 1 by N vector has a shared exponent, and each row of an N by N matrix has a shared exponent. Each element in the block may have a Bbit two'"'"'s complement mantissa, and the mantissa size for the vector (B_{V}) and matrix elements (B_{M}) can be different.
In some examples, the native dimension (N) may be the central synthesis parameter for specialization. Native vectors may have a size of 1 by N and native matrices may have a size of N by N, and all instructions for neural network processor 100 may operate only on nativesized data. Logical vectors and matrices in applications may often be larger than the native size; in these cases, the vectors and matrices may be broken up into nativesized tiles. Table 4 below shows a nonexhaustive list of synthesis parameters:
Intratile column parallelism may exploit the fact that each multiplication within a row of an MVM (e.g., each dot product) is independent. In this example, the synthesis parameter Lanes (L) indicates the number of parallel multipliers that can process Lelements of vector and matrix input data that will result in a partial dotproduct, as depicted in the topright corner of
Intratile row parallelism may exploit the fact that every row in an MVM (e.g., each dot product) is independent from the others. Each tile engine in the processor may be responsible for a matrixvector multiplication of a vector and matrix of sizes equal to the native dimension (N). As explained earlier, within each tile engine the processor may instantiate an array of N dot product units, each of which may handle a row in the MVM operation. The dot product units may be laneparallel, therefore each tile engine may compute N*L multiplyaccumulate operations in parallel. Therefore, at the end of a single matrixvector multiplication, a tile engine may produce a vector result of size N over N/L clock cycles. Dot product unit parallelism may be limited by the number of parallel dot products available in a nativesized MVM tile (e.g., N should not significantly exceed the dimensions of an application'"'"'s logical matrices).
Intertile column parallelism may exploit the fact that each column tile in a tiled MVM can be computed independently if the subresults are then summed together. In accordance with one example,
Another synthesis specialization parameter may be related to the number of independent parallel channels (C) of tile engines in a matrix vector multiplier. This parameter may correspond to vectorbatch parallelism, which may be useful in applications that have abundant independent vectors that need to be multiplied by the same matrix data. In this case, the neural network processor may instantiate parallel channels (C) of the tile engines and multifunction units that operate on different vectors, but on the same matrix data. In accordance with one example,
Yet another synthesis specialization parameter may be related to the independent engine parallelism, which operates on the principle that MVMs of completely different vectors and matrices are independent. A neural network processor can use this by instantiating parallel channels that operate on different matrix data.
Mixing engine parallelization strategies can be productive when an application has characteristics that are intermediate between the ideals for intertile row parallelism and intertile column parallelism. For example, an application may have a moderate number (e.g., 2) of moderately sized matrices (e.g., 2N by 2N). In accordance with one example,
Step 1020 may include obtaining code representative of at least a portion of at least one hardware node for implementing the neural network processor. In one example, the code may be register transferlevel (RTL) code for at least one FPGA that may be used to implement the neural network processor.
Step 1030 may include obtaining a synthesis model comprising a plurality of synthesis specialization parameters including: (1) a first synthesis specialization parameter corresponding to a first native dimension of the each of the matrix elements and, (2) a second synthesis specialization parameter corresponding to a second native dimension of the each of the vector elements, and (2) a third synthesis specialization parameter corresponding to a number of the plurality of tile engines, where each of a first value corresponding to the first synthesis specialization parameter, a second value corresponding to the second synthesis specialization parameter, and a third value corresponding to the third synthesis specialization parameter is selected to meet or exceed a performance metric associated with the at least one hardware node. In this example, the block size of the BFP format data may be equal to the native dimension. Therefore, each native 1 by N vector may have a shared exponent, and each row of an N by N matrix may have a shared exponent. Each of the vector data and the matrix data may have a two'"'"'s complement mantissa portion, and the mantissa size for the vector data and the matrix data may be different.
In one example, the hardware node may be implemented using at least one FPGA that may include adaptive logic modules, digital signal processors, and randomaccess memories (e.g., BRAMs). In this example, the performance metric may relate to an area required to implement the adaptive logic modules, digital signal processors, and randomaccess memories (e.g., BRAMs) as part of the at least one FPGA. In one example, the area required to implement these FGPA portions may be a function of the area of the tile engines, the vector register file, the matrix register files, and other components of a neural network processor, such as the ones described earlier. In one example, the performance metric may correspond to a total tile area of each of the plurality of tile engines, which may be computed based on the equation and the variables shown in Table 5 below:
Thus, as an example, Table 6 below shows various possible configurations of a hardware node (e.g., an FPGA) that may be used to implement a neural network processor and the corresponding approximate numbers of adaptive logic modules (ALMs), randomaccess memories (RAMs), and digital signal processors (DSPs).
When the hardware node is implemented using a specific FPGA (e.g., Stratix), the performance metric related to the area may further be evaluated using information provided by the supplier of the FPGA.
Step 1040 may include using the processor modifying the code, based on at least the first minimum number of units, the second minimum number of units, the third minimum number of units and at least the first value and the second value, to generate a modified version of the code and storing a modified version of the code. As part of this step, in one example, the code representative of the at least one portion of the FPGA may be modified such that the FPGA is now synthesisspecialized corresponding to a neural network processor is tailored to provide inference related to a specific application, such as an image classification application.
Although
In conclusion, the present disclosure relates to a method, implemented by a processor, for synthesizing a neural network processor comprising a plurality of tile engines, where each of the plurality of tile engines is configured to process matrix elements and vector elements. The method may include using the processor analyzing a neural network model corresponding to an application to determine: (1) a first minimum number of units required to express a shared exponent value required to satisfy a first precision requirement corresponding to each of the matrix elements and corresponding to each of the vector elements, (2) a second minimum number of units required to express a first mantissa value required to satisfy a second precision requirement corresponding to the each of the matrix elements, and (3) a third minimum number of units required to express a second mantissa value required to satisfy a third precision requirement corresponding to the each of the vector elements.
The method may further include obtaining code representative of at least a portion of at least one hardware node for implementing the neural network processor. The method may further include obtaining a synthesis model comprising a plurality of synthesis specialization parameters including: (1) a first synthesis specialization parameter corresponding to a first native dimension of the each of the matrix elements, (2) a second synthesis specialization parameter corresponding to a second native dimension of the each of the vector elements, and (3) a third synthesis specialization parameter corresponding to a number of the plurality of tile engines, where each of a first value corresponding to the first synthesis specialization parameter, a second value corresponding to the second synthesis specialization parameter, and a third value corresponding to the third synthesis specialization parameter is selected to meet or exceed a performance metric associated with the at least one hardware node.
The method may further include using the processor modifying the code, based on at least the first minimum number of units, the second minimum number of units, the third minimum number of units and at least the first value and the second value, to generate a modified version of the code and storing a modified version of the code.
The synthesis specialization parameters may further comprise a fourth synthesis specialization parameter corresponding to a number of parallel multipliers that can process the matrix elements and the vector elements to produce a partial dotproduct. The synthesis specialization parameters may further comprise a fifth synthesis specialization parameter corresponding to a number of independent parallel channels of the plurality of tile engines. The synthesis specialization parameters may further comprise a sixth synthesis specialization parameter corresponding to a number of groups, where each of the groups has a group size equal to the number of the plurality of tile engines divided by the number of the independent parallel channels.
In this example, the hardware node may comprise a field programmable gate array (FPGA) including adaptive logic modules, digital signal processors, and randomaccess memories, and the performance metric may correspond to an area required to implement the adaptive logic modules, the digital signal processors, and the randomaccess memories as part of the FPGA.
The tile engines may include dot product units and where each of the dot product units may be configured to receive the matrix elements from a matrix register file, and where the synthesis specialization parameters may further comprise an eighth synthesis specialization parameter corresponding to whether the matrix register file is private to each one of the plurality of tile engines or whether the matrix register file is shared among the plurality of tile engines. In another example, the tile engines may include dot product units and where each of the dot product units and the synthesis specialization parameters may further comprise a ninth synthesis specialization parameter corresponding to whether each of the plurality of dot product units comprises an addreduction tree.
In another example, the present disclosure relates to a system comprising a processor and a memory comprising: (1) code representative of at least a portion of at least one hardware node for implementing the neural network processor comprising a plurality of tile engines, where each of the plurality of tile engines is configured to process matrix elements and vector elements, (2) a synthesis model comprising a plurality of synthesis specialization parameters including: (a) a first synthesis specialization parameter corresponding to a first native dimension of the each of the matrix elements and, (b) a second synthesis specialization parameter corresponding to a second native dimension of the each of the vector elements, and (c) a third synthesis specialization parameter corresponding to a number of the plurality of tile engines, where each of a first value corresponding to the first synthesis specialization parameter, a second value corresponding to the second synthesis specialization parameter, and a third value corresponding to the third synthesis specialization parameter is selected to meet or exceed a performance metric associated with the at least one hardware node, and (3) instructions for synthesizing a neural network processor comprising a plurality of tile engines, where each of the plurality of tile engines is configured to process matrix elements and vector elements.
The instructions may be configured to using the processor analyze a neural network model corresponding to an application to determine: (1) a first minimum number of units required to express a shared exponent value required to satisfy a first precision requirement corresponding to each of the matrix elements and corresponding each of the vector elements, (2) a second minimum number of units required to express a first mantissa value required to satisfy a second precision requirement corresponding to the each of the matrix elements, and (3) a third minimum number of units required to express a second mantissa value required to satisfy a third precision requirement corresponding to the each of the vector elements.
The instructions may further be configured to using the processor modify the code, based on at least the first minimum number of units, the second minimum number of units, the third minimum number of units and at least the first value, the second value, and the third value, to generate a modified version of the code and store a modified version of the code.
The synthesis specialization parameters may further comprise a fourth synthesis specialization parameter corresponding to a number of parallel multipliers that can process the matrix elements and the vector elements to produce a partial dotproduct. The synthesis specialization parameters may further comprise a fifth synthesis specialization parameter corresponding to a number of independent parallel channels of the plurality of tile engines. The synthesis specialization parameters may further comprise a sixth synthesis specialization parameter corresponding to a number of groups, where each of the groups has a group size equal to the number of the plurality of tile engines divided by the number of the independent parallel channels.
In this example, the hardware node may comprise a field programmable gate array (FPGA) including adaptive logic modules, digital signal processors, and randomaccess memories, and the performance metric may correspond to an area required to implement the adaptive logic modules, the digital signal processors, and the randomaccess memories as part of the FPGA.
The tile engines may include dot product units and where each of the dot product units may be configured to receive the matrix elements from a matrix register file, and where the synthesis specialization parameters may further comprise an eighth synthesis specialization parameter corresponding to whether the matrix register file is private to each one of the plurality of tile engines or whether the matrix register file is shared among the plurality of tile engines. In another example, the tile engines may include dot product units and where each of the dot product units and the synthesis specialization parameters may further comprise a ninth synthesis specialization parameter corresponding to whether each of the plurality of dot product units comprises an addreduction tree.
In yet another example, the present disclosure relates to a method for synthesizing a neural network processor comprising a plurality of tile engines, where each of the plurality of tile engines is configured to process matrix elements and vector elements, and where each of the plurality of tile engines comprises a plurality of dot product units and where each of the dot product units is configured to receive the matrix elements from a matrix register file.
The method may include using the processor analyzing a neural network model corresponding to an application to determine: (1) a first minimum number of units required to express a shared exponent value required to satisfy a first precision requirement corresponding to each of the matrix elements and corresponding to each of the vector elements, (2) a second minimum number of units required to express a first mantissa value required to satisfy a second precision requirement corresponding to the each of the matrix elements, and (3) a third minimum number of units required to express a second mantissa value required to satisfy a third precision requirement corresponding to the each of the vector elements.
The method may further include obtaining code representative of at least a portion of at least one hardware node for implementing the neural network processor. The method may further include obtaining a synthesis model comprising a plurality of synthesis specialization parameters including: (1) a first synthesis specialization parameter corresponding to whether the matrix register file is private to each one of the plurality of tile engines or whether the matrix register file is shared among the plurality of tile engines and (2) a second synthesis specialization parameter corresponding to whether each of the plurality of dot product units comprises an addreduction tree.
The method may further include using the processor modifying the code, based on at least the first minimum number of units, the second minimum number of units, the third minimum number of units and at least the first synthesis specialization parameter and the second synthesis specialization parameter, and storing a modified version of the code.
The synthesis specialization parameters may further comprise a third synthesis specialization parameter corresponding to a number of parallel multipliers that can process the matrix elements and the vector elements to produce a partial dotproduct. The synthesis specialization parameters may further comprise a fourth synthesis specialization parameter corresponding to a number of independent parallel channels of the plurality of tile engines. The synthesis specialization parameters may further comprise a fifth synthesis specialization parameter corresponding to a number of groups, where each of the groups has a group size equal to the number of the plurality of tile engines divided by the number of the independent parallel channels of the plurality of tile engines.
In this example, the hardware node may comprise a field programmable gate array (FPGA) including adaptive logic modules, digital signal processors, and randomaccess memories, and the performance metric may correspond to an area required to implement the adaptive logic modules, the digital signal processors, and the randomaccess memories as part of the FPGA. In this example, the performance metric may correspond to an area required to implement each of the plurality of tile engines
It is to be understood that the methods, modules, and components depicted herein are merely exemplary. Alternatively, or in addition, the functionally described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include FieldProgrammable Gate Arrays (FPGAs), ApplicationSpecific Integrated Circuits (ASICs), ApplicationSpecific Standard Products (ASSPs), SystemonaChip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc. In an abstract, but still definite sense, any arrangement of components to achieve the same functionality is effectively “associated” such that the desired functionality is achieved. Hence, any two components herein combined to achieve a particular functionality can be seen as “associated with” each other such that the desired functionality is achieved, irrespective of architectures or intermedial components. Likewise, any two components so associated can also be viewed as being “operably connected,” or “coupled,” to each other to achieve the desired functionality.
The functionality associated with some examples described in this disclosure can also include instructions stored in a nontransitory media. The term “nontransitory media” as used herein refers to any media storing data and/or instructions that cause a machine to operate in a specific manner. Exemplary nontransitory media include nonvolatile media and/or volatile media. Nonvolatile media include, for example, a hard disk, a solidstate drive, a magnetic disk or tape, an optical disk or tape, a flash memory, an EPROM, NVRAM, PRAM, or other such media, or networked versions of such media. Volatile media include, for example, dynamic memory, such as, DRAM, SRAM, a cache, or other such media. Nontransitory media is distinct from, but can be used in conjunction with transmission media. Transmission media is used for transferring data and/or instruction to or from a machine. Exemplary transmission media, include coaxial cables, fiberoptic cables, copper wires, and wireless media, such as radio waves.
Furthermore, those skilled in the art will recognize that boundaries between the functionality of the above described operations are merely illustrative. The functionality of multiple operations may be combined into a single operation, and/or the functionality of a single operation may be distributed in additional operations. Moreover, alternative embodiments may include multiple instances of a particular operation, and the order of operations may be altered in various other embodiments.
Although the disclosure provides specific examples, various modifications and changes can be made without departing from the scope of the disclosure as set forth in the claims below. Accordingly, the specification and figures are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present disclosure. Any benefits, advantages, or solutions to problems that are described herein with regard to a specific example are not intended to be construed as a critical, required, or essential feature or element of any or all the claims.
Furthermore, the terms “a” or “an,” as used herein, are defined as one or more than one. Also, the use of introductory phrases such as “at least one” and “one or more” in the claims should not be construed to imply that the introduction of another claim element by the indefinite articles “a” or “an” limits any particular claim containing such introduced claim element to inventions containing only one such element, even when the same claim includes the introductory phrases “one or more” or “at least one” and indefinite articles such as “a” or “an.” The same holds true for the use of definite articles.
Unless stated otherwise, terms such as “first” and “second” are used to arbitrarily distinguish between the elements such terms describe. Thus, these terms are not necessarily intended to indicate temporal or other prioritization of such elements.