Configurable neural network integrated circuit

• US 5,956,703 A
• Filed: 07/28/1995
• Issued: 09/21/1999
• Est. Priority Date: 07/28/1995
• Status: Expired due to Term

• Current Assignee: Delphi Technologies, Inc. (BorgWarner Incorporated)
• Original Assignee: Delco Electronics Corporation (Motors Liquidation Co. GUC Trust)
• Inventors: Turner, Douglas D., Breuer, Gabriela
• Primary Examiner(s): MacDonald, Allen R.
• Assistant Examiner(s): Shah, Sanjiv
• Application Number:
  US08/508,637
• Time in Patent Office: 1,516 Days
• Field of Search: 395/27, 395/22, 395/24, 395/11, 382/197
• US Class Current: 706/27
• CPC Class Codes: G06N 3/063 using electronic means

Abstract

A neural network IC 31 includes n dedicated processing elements (PEs) 62, an output register 66 for storing the PEs'"'"' outputs so that they are immediately accessible to all of the PEs, a number of output circuits 78 that are connected to selected PEs to provide binary outputs, and a timing circuit 74. Each of the PEs includes a weight memory 90 for storing input, output and bias weight arrays, a first in first out (FIFO) memory 88 for storing input data, a dot product circuit 92 and an activation circuit 94. The dot product circuit computes a dot product of the input weight array and the contents of the FIFO memory, a dot product of the output weight array and the contents of the output register, a dot product of the bias value and a constant, and sums the three results. The activation circuit maps the output of the dot product circuit through an activation function to produce the PE'"'"'s output. The inclusion of a memory 90 that stores both input and output weight arrays in conjunction with the output register 66 allows the PEs to be configured to implement arbitrary feed-forward and recurrent neural network architectures.

Claims

1. A neural network integrated circuit (IC) having a programmable architecture, comprising:

a plurality of processing elements (PEs) that generate output values at successive computation cycles, each PE comprising;

a data memory,

an input weight memory,

an output weight memory,

a processing circuit that computes an intermediate value that is a function of the contents of said data and input weight memories, the contents of the output weight memory and the PEs'"'"' output values from a previous cycle, and

an activation circuit that maps the intermediate value into said output value in accordance with a desired activation function;

an output memory for storing said output values so that said stored output values are accessible to each PE'"'"'s processing circuit at the next computation cycle;

a data bus for writing input into the data memory;

a timing circuit that is enabled when data is written into said data memories, counts a fixed amount of time per subcycle, and outputs an interrupt when p+1 subcycles have been completed; and

an output circuit that reads out L of said PEs'"'"' output values in response to said interrupt,

the input and output weight memories being programmed with arrays of respective input and output weights configuring the PEs to include and compute in p+1 computation subcycles p hidden layers that extract features from the input data, and an output layer that weights the respective features.

2. The neural network of claim 1, wherein said output circuit includes L logic circuits that compare the output values from L of said PEs in the output layer to respective threshold values to provide binary outputs.

3. A neural network integrated circuit (IC) having a programmable architecture, comprising:

a plurality of processing elements (PEs), including a plurality of PEs arranged in n layers, that generate output values at successive computation cycles, said computation cycles having n subcycles, one for each layer, each PE comprising;

a data memory,

an input weight memory,

an output weight memory,

a processing circuit that computes an intermediate value that is a function of the contents of said data and input weight memories, the contents of the output weight memory and the PE'"'"'s output values from a previous cycle, and

an activation circuit that maps the intermediate value into said output value in accordance with a desired activation function;

an output memory for storing said output values so that said stored output values are accessible to each PE'"'"'s processing circuit at the next computation cycle; and

a PE selection circuit that stores an execution subcycle for each PE and controls said output memory so that the PE'"'"'s output values are only written into said output memory during their respective execution subcycles,

the input and output weight memories being programmable so that the PEs can be interconnected to provide a desired neural network architecture.

4. The neural network of claim 3, further comprising a timing circuit that is enabled at the beginning of each computation cycle and increments a count at the end of each subcycle, said PE selection circuit comprising:

a memory for storing the execution subcycles;

a temporary buffer for storing the PEs'"'"' output values at each subcycle; and

a comparator that compares each of said execution subcycles to said count, and enables said temporary buffer to write the respective output values to the output memory when said execution subcycle equals said count.

5. A neural network processor, comprising:

a neural network integrated circuit that includes a plurality of processing elements (PEs), including a plurality of PEs arranged in n layers, which generate output values at successive computation cycles, said computation cycle having n subcycles, one for each layer, and an output memory that stores the output values so that they are accessible to the PE'"'"'s at the next computation cycle, each PE comprising;

a data memory for storing the input data values,

an input weight memory for storing input weights,

an output weight memory for storing output weights, said input and output weights configuring the PEs to provide a desired neural network architecture, and

a processing circuit that computes said output value as a function of the input data values and input weights, and the output weights and PEs'"'"' output values from a previous cycle;

a microprocessor that writes input data into the PEs'"'"' data memories, and controls a start cycle to initiate the computation of said output values, and an output enable to read the output values;

a timing circuit that receives a clock signal, initiates computation of the PEs in response to said start cycle, counts a fixed number of clock cycles per subcycle, increments a count at the end of each subcycle and generates an interrupt at the end of said computation cycle; and

a PE selection circuit that stores an execution subcycle for each PE and controls said output memory so that the PE'"'"'s output values are only written into said output memory when their respective execution subcycles are the same as the said count.