Neural digital processor utilizing an approximation of a non-linear activation function
First Claim
Patent Images
1. A neural digital processor (10), comprising:
- a. an input (13) for receiving digital data and generating an output signal,b. a neural unit (12) coupled to the input for calculating neural potentials, from the output signal, according to a function of the output signal and synaptic coefficients, said synaptic coefficients are weight connections either between neurons or between neurons and the input,c. memory (16) for storing said synaptic coefficients,d. means (14) for subjecting at least one of the neural potentials, designated POT, to at least one approximative non-linear activation function ANLF which is formed by n segments in order to produce at least one neural state, said means (14) comprising another neural digital processor which comprises;
I. means (20) for calculating n combinations, Mj =Hj ·
POT+Thj, wheren is an integer;
j is an integer such that 1≦
j≦
n;
Hj are predetermined synaptic coefficients; and
Thj are thresholds,II. means (22) for calculating states Sj =F(Mj), using another non-linear function CNLF which is formed byA. a segment F(x), where x is a running independent variable, which segment is not constant when x is situated in an interval (-xmin, xmax), andB. two segments F(x)=Fmax and F(x)=-Fmin when x≧
xmax and x≦
-xmin, respectively,III. means (24) for linearly combining the states with further synaptic coefficients Dj in order to produce said at least one neural state STAT.
0 Assignments
0 Petitions
Accused Products
Abstract
A neural digital processor (10) that includes circuitry (14) for applying a function ANLF to neural potentials. ANLF approximates a non-linear activation function NLF. The circuitry includes another neural processor which operates with another non-linear activation function CNLF. CNLF is a simple function, for example a ramp. The circuitry (14) may comprise elements (361, 362, 64) in common with apparatus (75) for calculating a derivative of the approximation function ANLF. The precision of approximation of the non-linear activation function NLF can be predetermined.
40 Citations
17 Claims
-
1. A neural digital processor (10), comprising:
-
a. an input (13) for receiving digital data and generating an output signal, b. a neural unit (12) coupled to the input for calculating neural potentials, from the output signal, according to a function of the output signal and synaptic coefficients, said synaptic coefficients are weight connections either between neurons or between neurons and the input, c. memory (16) for storing said synaptic coefficients, d. means (14) for subjecting at least one of the neural potentials, designated POT, to at least one approximative non-linear activation function ANLF which is formed by n segments in order to produce at least one neural state, said means (14) comprising another neural digital processor which comprises; I. means (20) for calculating n combinations, Mj =Hj ·
POT+Thj, wheren is an integer; j is an integer such that 1≦
j≦
n;Hj are predetermined synaptic coefficients; and Thj are thresholds, II. means (22) for calculating states Sj =F(Mj), using another non-linear function CNLF which is formed by A. a segment F(x), where x is a running independent variable, which segment is not constant when x is situated in an interval (-xmin, xmax), and B. two segments F(x)=Fmax and F(x)=-Fmin when x≧
xmax and x≦
-xmin, respectively,III. means (24) for linearly combining the states with further synaptic coefficients Dj in order to produce said at least one neural state STAT. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10)
-
-
11. A data processing system comprising a neural network comprising:
-
a. a unit for creating a neural potential from input data received by the unit, wherein the neural potential represents a sum of products, each respective one of the products resulting from a respective multiplication of a respective one of the input data by a respective synaptic coefficient; b. non-linear function means for applying a non-linear function to the neural potential to create neuron output data, the non-linear function means comprising; I. a plurality of further units, each respective one of the further units being for A. supplying a respective further product resulting from multiplying the neural potential by a respective factor; and B. applying a respective further non-linear function to the further product to create a respective outcome; and II. combining means for linearly combining the respective outcomes to provide a value of the first non-linear function associated with said neural potential said neural potential being an intermediate signal within a single neuron function whose external signal output is said neuron output data. - View Dependent Claims (12, 13)
-
-
14. A non-linear function device for use in creating at least one neuron output in a neural network processor, the device comprising:
-
a. means for receiving a neural potential signal which results from the sum of the products of a plurality of respective synaptic coefficients to a plurality of respective neuron input signals, which neural potential signal is an intermediate signal within a same neuron function as the at least one neuron output; b. a plurality of means for sequentially applying a plurality of respective basic transfer functions to the neural potential signal for creating respective intermediate outcomes; and c. combining means for linearly combining the respective outcomes to produce at least one non-linear output signal, said at least one non-linear output signal being an output of the at least one neuron.
-
-
15. A neural digital processor comprising:
-
a. input means for receiving digital data; b. first means for storing a first plurality of synaptic coefficients; c. a first neural unit for applying the first plurality of synaptic coefficients to an output signal of the input means, according to a stored neuron configuration to create at least one neural potential signal; d. means for applying at least one respective first non-linear function to each of the at least one neural potential signal to create at least one respective neuron output, said means for applying at least one first non-linear function comprising another neural digital processor coupled to the first neural unit, said other neural digital processor comprising; I. second means for storing a second plurality of synaptic coefficients; II. a second neural unit for applying the second plurality of synaptic coefficients to the output signal of the first neural unit; and III. means for applying at least one second non-linear function to an output signal of the second neural unit, said at least one second non-linear function being simpler than the non-linear function, wherein each of the at least one neural potential signal is an intermediate signal within a respective single neuron function whose ultimate output is created by the means for applying at least one first respective non-linear function. - View Dependent Claims (16, 17)
-
Specification