Pipelined hardware implementation of a neural network circuit

• US 6,836,767 B2
• Filed: 10/03/2001
• Issued: 12/28/2004
• Est. Priority Date: 10/03/2001
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: McBride, Chad B.
• Primary Examiner(s): Davis, George
• Application Number:
  US09/970,002
• Publication Number:
  US 20030065631A1
• Time in Patent Office: 1,182 Days
• Field of Search: 706/41
• US Class Current: 706/41
• CPC Class Codes: G06N 3/063 using electronic means

Abstract

In a first aspect, a pipelined hardware implementation of a neural network circuit includes an input stage, two or more processing stages and an output stage. Each processing stage includes one or more processing units. Each processing unit includes storage for weighted values, a plurality of multipliers for multiplying input values by weighted values, an adder for adding products outputted from product multipliers, a function circuit for applying a non-linear function to the sum outputted by the adder, and a register for storing the output of the function circuit.

Claims

1. A pipelined hardware implementation of a neural network circuit, comprising:

an input stage for receiving and storing input values;

a first processing stage coupled to the input stage, the first processing stage including a plurality of first processing units, each first processing unit including;

(a) a weight store for storing a plurality of weighted values;

(b) a plurality of multipliers each for multiplying an input value by a respective weighted value;

(c) an adder for adding a product outputted from one of the multipliers with at least one product outputted from a respective multiplier of another one of the plurality of first processing units;

(d) a function circuit for receiving a sum outputted by the adder and for generating therefrom a processing unit value; and

(e) a register for storing the processing unit value generated by the function circuit;

an additional processing stage coupled to an upstream processing stage and including a plurality of additional stage processing units, each additional stage processing unit including;

(i) a weight store for storing a plurality of weighted values;

(ii) a plurality of multipliers each for multiplying a processing unit value received from a processing unit of the upstream processing stage by a respective weighted value;

(iii) an adder for adding a product outputted from one of the multipliers of the respective additional stage processing unit with at least one product outputted from a respective multiplier of another one of the plurality of additional stage processing units;

(iv) a function circuit for receiving a sum outputted by the adder of the respective additional stage processing unit and for generating therefrom a processing unit value; and

(v) a register for storing the processing unit value generated by the function circuit of the respective additional stage processing unit; and

an output stage formed from output ports of the registers of the additional processing stage.

2. The apparatus of claim 1, further comprising at least one intervening processing stage coupled between the first processing stage and the additional processing stage.

3. The apparatus of claim 1, wherein the additional processing stage performs calculations with respect to a first set of input values at the same time that the first processing stage performs calculations with respect to a second set of input values.

4. The apparatus of claim 1, further comprising means for loading the weighted values in the weight stores.

5. The apparatus of claim 1, wherein the number of multipliers included in each first processing unit is equal to the number of first processing units included in the first processing stage.

6. The apparatus of claim 1, wherein the number of multipliers included in each additional stage processing unit is equal to the number of additional stage processing units included in the additional processing stage.

7. The apparatus of claim 1, wherein the function circuits apply a sigmoid function to respective sums received by the function circuits.

8. The apparatus of claim 1, wherein the function circuits apply one of a step function, a ramp function and a linear threshold function to the respective sums received by the function circuits.

9. The apparatus of claim 1 wherein the upstream processing stage is the first processing stage.

10. A method of performing a neural network process, comprising:

providing an input stage for receiving and storing input values;

providing a first processing stage coupled to the input stage, the first processing stage including a plurality of first processing units, each first processing unit including;

(a) a weight store for storing a plurality of weighted values;

(b) a plurality of multipliers each for multiplying an input value by a respective weighted value;

(c) an adder for adding a product outputted from one of the multipliers with at least one product outputted from a respective multiplier of another one of the plurality of first processing units;

(d) a function circuit for receiving a sum outputted by the adder and for generating therefrom a processing unit value; and

(e) a register for storing the processing unit value generated by the function circuit;

providing an additional processing stage coupled to an upstream processing stage and including a plurality of additional stage processing units, each additional stage processing unit including;

(i) a weight store for storing a plurality of weighted values;

(ii) a plurality of multipliers each for multiplying a processing unit value received from a processing unit of the upstream processing stage by a respective weighted value;

(iii) an adder for adding a product outputted from one of the multipliers of the respective additional stage processing unit with at least one product outputted from a respective multiplier of another one of the plurality of additional stage processing units;

(iv) a function circuit for receiving a sum outputted by the adder of the respective additional stage processing unit and for generating therefrom a processing unit value; and

(v) a register for storing the processing unit value generated by the function circuit of the respective additional stage processing unit;

forming an output stage from output ports of the registers of the additional processing stage; and

operating the first processing stage and the additional processing stage simultaneously to process respective sets of input values.

11. A pipelined hardware implementation of a neural network circuit, comprising:

an input stage for receiving and storing input values;

a first processing stage coupled to the input stage, the first processing stage including a plurality of first processing units, each first processing unit including;

(a) a weight store for storing a plurality of weighted values;

(b) a plurality of multipliers each for multiplying an input value by a respective weighted value;

(c) an adder for adding a product outputted from one of the multipliers with at least one product outputted from a respective multiplier of another one of the plurality of first processing units;

(d) a function circuit for receiving a sum outputted by the adder and for generating therefrom a processing unit value; and

(e) a register for storing the processing unit value generated by the function circuit;

an additional processing stage coupled to an upstream processing stage and including at least one additional stage processing unit, the additional stage processing unit including;

(i) a weight store for storing a plurality of weighted values;

(ii) a plurality of multipliers each for multiplying a processing unit value received from a processing unit of the upstream processing stage by a respective weighted value;

(iii) an adder for adding at least some of the products outputted from the multipliers of the respective additional stage processing unit;

(iv) a function circuit for receiving a sum outputted by the adder of the respective additional stage processing unit and generating therefrom a processing unit value; and

(v) a register for storing the processing unit value generated by the function circuit of the respective additional stage processing unit; and

an output stage including an output port of the register of the additional processing stage.

12. The apparatus of claim 11, further comprising at least one intervening processing stage coupled between the first processing stage and the additional processing stage.

13. The apparatus of claim 11, wherein the upstream processing stage is the first processing stage.

14. A pipelined hardware implementation of a neural network circuit, comprising:

an input stage adapted to receive and store input values;

a first processing stage coupled to the input stage, the first processing stage including a plurality of first processing units, each first processing unit including;

(a) a weight store adapted to store a plurality of weighted values;

(b) a plurality of multipliers each adapted to multiply an input value by a respective weighted value;

(c) an adder adapted to add a product outputted from one of the multipliers with at least one product outputted from a respective multiplier of another one of the plurality of first processing units;

(d) a function circuit adapted to receive a sum outputted by the adder and to generate therefrom a processing unit value; and

(e) a register adapted to store the processing unit value generated by the function circuit;

an additional processing stage coupled to an upstream processing stage and including a plurality of additional stage processing units, each additional stage processing unit including;

(i) a weight store adapted to store a plurality of weighted values;

(ii) a plurality of multipliers each adapted to multiply a processing unit value received from a processing unit of the upstream processing stage by a respective weighted value;

(iii) an adder adapted to add a product outputted from one of the multipliers of the respective additional stage processing unit with at least one product outputted from a respective multiplier of another one of the plurality of additional stage processing units;

(iv) a function circuit adapted to receive a sum outputted by the adder of the respective additional stage processing unit and to generate therefrom a processing unit value; and

(v) a register adapted to store the processing unit value generated by the function circuit of the respective additional stage processing unit; and

an output stage formed from output ports of the registers of the additional processing stage.

15. A pipelined hardware implementation of a recall-only neural network circuit, comprising:

an input stage adapted to receive and store at least one input value;

a first processing stage coupled to the input stage, the first processing stage including at least one processing unit having;

(a) a weight store adapted to store at least one weighted value;

(b) at least one multiplier adapted to multiply an input value by a respective weighted value;

(c) a function circuit coupled downstream from one or more of the at least one multiplier and adapted to receive a function input and to generate therefrom a processing unit value; and

(d) a register adapted to store the processing unit value generated by the function circuit;

an additional processing stage coupled to an upstream processing stage and including at least one additional stage processing unit having;

(i) a weight store adapted to store at least one weighted value;

(ii) at least one multiplier adapted to multiply a processing unit value received from a processing unit of the upstream processing stage by a weighted value;

(iii) a function circuit coupled downstream from one or more of the at least one multiplier of the respective additional stage processing unit and adapted to receive a function input and to generate therefrom a processing unit value; and

(iv) a register adapted to store the processing unit value generated by the function circuit of the respective additional stage processing unit; and

an output stage including an output port of the register of the additional processing stage.

16. The apparatus of claim 15, wherein each of the processing units includes an adder adapted to add products of multipliers coupled to the adder to produce a sum supplied as the function input to the function circuit of the respective processing unit.

17. The apparatus of claim 16, further comprising circuitry adapted to load the weighted values in the weight stores.