Vector computation unit in a neural network processor

• US 10,192,162 B2
• Filed: 09/03/2015
• Issued: 01/29/2019
• Est. Priority Date: 05/21/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Thorson, Gregory Michael, Clark, Christopher Aaron, Luu, Dan
• Primary Examiner(s): Rifkin, Ben M
• Application Number:
  US14/845,117
• Publication Number:
  US 20160342889A1
• Time in Patent Office: 1,244 Days
• Field of Search: None
• CPC Class Codes: G06F 5/08 having a sequence of storage locations, the intermediate ones not being accessible for either enqueue or dequeue operations, e.g. using a shift register G06F5/065 takes precedence; shift registers per se G11C19/00
• CPC Class Codes: G06F 7/544 for evaluating functions by calculation G06F7/4824 takes precedence
• CPC Class Codes: G06N 3/0464 Convolutional networks [CNN, ConvNet]
• CPC Class Codes: G06N 3/048 Activation functions
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/04 Inference or reasoning models

Abstract

A circuit for performing neural network computations for a neural network comprising a plurality of layers, the circuit comprising: activation circuitry configured to receive a vector of accumulated values and configured to apply a function to each accumulated value to generate a vector of activation values; and normalization circuitry coupled to the activation circuitry and configured to generate a respective normalized value from each activation value.

Claims

1. A system for performing neural network computations for a neural network having a plurality of neural network layers, the system comprising:

normalization circuitry comprising;

M sets of normalization registers, wherein M is a positive integer greater than one, and wherein each set of normalization registers comprises multiple normalization registers, each normalization register configured to;

receive, in a staggered manner over multiple clock cycles, a set of activated values for a neural network layer; and

store subsets of the activated values in the normalization register; and

multiple normalization units, each normalization unit communicatively coupled to multiple sets of normalization registers and each normalization unit configured to;

obtain, from two or more sets of the normalization registers, the subsets of activated values stored in the normalization registers; and

normalize the subsets of activated values obtained from the two or more sets of normalization registers to generate a normalized value for determining an activation input for a subsequent neural network layer of the neural network.

2. The system of claim 1, further comprising:

activation circuitry configured to;

receive multiple input vectors, wherein each input vector includes multiple accumulated values; and

determine, based on the multiple input vectors, multiple output vectors, wherein each output vector includes multiple activated values.

3. The system of claim 2, wherein the activation circuitry is configured to determine the set of activated values based on at least one output vector of the multiple output vectors.

4. The system of claim 1, wherein at least two of the normalization registers are connected in series, such that an activated value provided by a first normalization register of the multiple normalization registers is received by a second normalization register of the multiple normalization registers.

5. The system of claim 1, wherein the set of activated values is represented by data that includes multiple activated values of the set of activated values.

6. The system of claim 1, wherein the set of activated values are represented by data that includes a square of multiple activated values of the set of activated values.

7. The system of claim 1, wherein normalizing the subsets of activated values to generate the normalized value comprises determining a sum of the activated values in the subsets.

8. The system of claim 7, wherein normalizing the subsets of activated values to generate the normalized value comprises:

determining a multiplication factor based on the sum of the activated values in the subsets; and

multiplying an activation value by the multiplication factor to determine the normalized value.

9. The system of claim 1, wherein normalizing the subsets of activated values to generate the normalized value comprises normalizing the subsets of activated values using a lookup table.

10. The system of claim 1, wherein a normalization unit of the multiple normalization units is configured to provide the normalized value to another normalization unit of the multiple normalization units.

11. The system of claim 1, wherein the system further comprises pooling circuitry configured to aggregate multiple normalized values to generate a pooled value.

12. The system of claim 11, wherein aggregating the multiple normalized values comprises applying an aggregation function to the multiple normalized values.

13. The system of claim 11, wherein the pooling circuitry comprises multiple registers and multiple memory units connected in series.

14. A method for performing neural network computations for a neural network having a plurality of neural network layers using normalization circuitry that includes M sets of normalization registers and multiple normalization units, each normalization unit communicatively coupled to multiple sets of normalization registers, wherein M is a positive integer greater than one, and wherein each set of normalization registers include multiple normalization registers, the method comprising:

receiving, by each set of the normalization registers in a staggered manner over multiple clock cycles, a set of activated values for a neural network layer; and

storing, by each set of the normalization registers, subsets of the activated values in the normalization registers of the set; and

obtaining, by each normalization unit of the multiple normalization units and from two or more sets of the normalization registers, the subsets of activated values stored in the normalization registers; and

normalizing, by each normalization unit of the multiple normalization units, the subsets of activated values obtained from the two or more sets of normalization registers to generate a normalized value for determining an activation input for a subsequent neural network layer of the neural network.

15. The method of claim 14, wherein the set of activated values is represented by data that includes multiple activated values of the set of activated values.

16. The method of claim 14, wherein the set of activated values is represented by data that includes a square of multiple activated values of the set of activated values.

17. The method of claim 14, wherein normalizing the subsets of activated values to generate the normalized value comprises determining a sum of the activated values in the subsets.

18. A normalization circuitry comprising:

M sets of normalization registers, wherein M is a positive integer greater than one, and wherein each set of normalization registers comprises multiple normalization registers, each normalization register configured to;

receive, in a staggered manner over multiple clock cycles, a set of activated values for a neural network layer; and

store subsets of the activated values in the normalization register; and

multiple normalization units, each normalization unit communicatively coupled to multiple sets of normalization registers and each normalization unit configured to;

obtain, from two or more sets of the normalization registers, the subsets of activated values stored in the normalization registers; and

normalize the subsets of activated values obtained from the two or more sets of normalization registers to generate a normalized value for determining an activation input for a subsequent neural network layer of the neural network.

19. The normalization circuitry of claim 18, wherein at least two of the normalization registers are connected in series, such that an activated value provided by a first normalization register of the multiple normalization registers is received by a second normalization register of the multiple normalization registers.

20. The normalization circuitry of claim 18, wherein normalizing the subsets of activated values to generate the normalized value comprises determining a sum of the activated values in the subsets.