VECTOR COMPUTATION UNIT IN A NEURAL NETWORK PROCESSOR

• US 20160342889A1
• Filed: 09/03/2015
• Published: 11/24/2016
• Est. Priority Date: 05/21/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Clark, Christopher Aaron, Luu, Dan, Thorson, Gregory Michael
• Granted Patent: US 10,192,162 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 5/08 having a sequence of storage locations, the intermediate ones not being accessible for either enqueue or dequeue operations, e.g. using a shift register G06F5/065 takes precedence; shift registers per se G11C19/00
• CPC Class Codes: G06F 7/544 for evaluating functions by calculation G06F7/4824 takes precedence
• CPC Class Codes: G06N 3/0464 Convolutional networks [CNN, ConvNet]
• CPC Class Codes: G06N 3/048 Activation functions
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/04 Inference or reasoning models

Abstract

A circuit for performing neural network computations for a neural network comprising a plurality of layers, the circuit comprising: activation circuitry configured to receive a vector of accumulated values and configured to apply a function to each accumulated value to generate a vector of activation values; and normalization circuitry coupled to the activation circuitry and configured to generate a respective normalized value from each activation value.

Claims

1. A circuit for performing neural network computations for a neural network comprising a plurality of layers, the circuit comprising:

activation circuitry configured to receive a vector of accumulated values and configured to apply a function to each accumulated value to generate a vector of activation values; and

normalization circuitry coupled to the activation circuitry and configured to generate a respective normalized value for each activation value.

2. The circuit of claim 1, where the activation circuitry receives the vector of accumulated values from a systolic array in the circuit.

3. The circuit of claim 1, where the normalization circuitry comprises a plurality of normalization register columns, each normalization register column comprising a plurality of normalization registers connected in series, each normalization register column configured to receive a respective activation value, where the normalization circuitry is configured to form groups around one or more normalization registers, each group corresponding to a normalization unit, and each normalization unit configured to calculate a respective normalized value for the respective activation value.

4. The circuit of claim 3, where each normalization register is configured to pass the distinct activation value to an adjacent normalization column.

5. The circuit of claim 3, where each group is formed using a normalization radius parameter.

6. The circuit of claim 3, where each normalization unit is configured to:

receive the respective activation value;

generate a respective intermediate normalized value from the respective activation value; and

send the respective intermediate normalized value to one or more neighboring normalization units.

7. The circuit of claim 6, where generating the respective intermediate normalized value comprises generating a square of the respective activation value.

8. The circuit of claim 6, where each normalization unit is further configured to:

receive, from one or more neighboring normalization units, one or more intermediate normalized values generated from activation values;

sum each intermediate normalized value to generate an index;

use the index to access one or more values from a lookup table;

generate a scaling factor from the one or more values and the index; and

generate the respective normalized value from the scaling factor and the respective activation value.

9. The circuit of claim 1, further comprising pooling circuitry configured to receive the normalized values and configured to pool the normalized values to generate a pooled value.

10. The circuit of claim 6, where the pooling circuitry is configured to store the plurality of normalized values in a plurality of registers and a plurality of memory units,

where the plurality of registers and the plurality of memory units are connected in series, where each register stores one normalized value and each memory unit stores a plurality of normalized values,

where the pooling circuitry is configured to, after every clock cycle, shift a given normalized value to a subsequent register or memory unit, and

where the pooling circuitry is configured to generate the pooled value from the normalized values.

11. The circuit of claim 1, further comprising pooling circuitry configured to receive the activation values and configured to pool the activation values to generate a pooled value.

12. The circuit of claim 11, where the pooling circuitry is configured to store the plurality of activation values in a plurality of registers and a plurality of memory units,

where the plurality of registers and the plurality of memory units are connected in series, where each register stores one normalized value and each memory unit stores a plurality of activation values,

where the pooling circuitry is configured to, after every clock cycle, shift a given activation value to a subsequent register or memory unit, and

where the pooling circuitry is configured to generate the pooled value from the activation values.