Spike transmission for neural networks

• US 4,893,255 A
• Filed: 05/31/1988
• Issued: 01/09/1990
• Est. Priority Date: 05/31/1988
• Status: Expired due to Fees

• Current Assignee: Analog Intelligence Corporation
• Original Assignee: Analog Intelligence Corporation
• Inventors: Tomlinson, Max S. Jr.
• Primary Examiner(s): NOT, DEFINED
• Application Number:
  US07/200,604
• Time in Patent Office: 588 Days
• Field of Search: 364/513, 364/200, 364/800, 364/807, 382/14, 382/15
• US Class Current: 706/35
• CPC Class Codes: G06N 3/049 Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs

Abstract

Pulse trains are utilized for the transmission of information in a neural network. A squash function is achieved by logically OR'"'"'ing together pulsed outputs, giving f(x) approximately 1-e^-x. For Back Propagation, as derived by Rumelhart, the derivative of the squash function is available by examining the time when no OR'"'"'ed together pulses are present, being 1-f(x), or e^-x. Logically AND'"'"'ing of the two signals. Mulitplication of input frequencies by weights is accomplished by modulating the width of the output pulses, while keeping the frequency the same.

Claims

1. A neural network for receiving an input vector comprised of input vector elements, each representing an input vector element value, and, in response thereto and as a function thereof, for outputing a network output value;

said network comprising;

a plurality of matrix elements, corresponding in number to the number of elements in the input vector, each comprising;

means for receiving a corresponding element of the input vector as an input data value;

duty cycle conversion means

for receiving said input data value, and

for producing an output data signal, said output data signal being a pulse train whose duty cycle is proportional to said input data value;

and

means for transmitting said output data signal; and

a combining element, comprising;

means for OR'"'"'ing said output data signals from said matrix elements to produce a raw squashed data output signal; and

means for transmitting said raw squashed data output signal as said network output value.

2. The neural network of claim 1, wherein said matrix elements further comprise:

weight value means for providing a weight value; and

wherein said duty cycle conversion means receives said weight value, and

wherein the duty cycle of said output data signal is proportional to the product of said input data value and said weight value.

3. The neural network of claim 2,

wherein the received input vector elements are in the form of pulse trains with the corresponding input vector values being represented by the frequency of the pulse trains;

and

wherein the frequency of said output data signal of said duty cycle conversion means is equal to the frequency of the input vector element pulse train and

wherein the width of a pulse of said output data signal is proportional to said weight value.

4. A neural network for receiving an input vector comprised of input vector elements, each representing an input vector value, and, in response thereto and as a function thereof, for outputing a network output value;

said network comprising;

a plurality of matrix elements, corresponding in number to the number of elements in the input vector, each comprising;

means for receiving a corresponding element of the input vector as an input data value;

duty cycle conversion means

for receiving said input data value, and

for producing an output data signal, said output data signal being a pulse train whose duty cycle is proportional to said input data value;

and

means for transmitting said output data signal;

a combining element, comprising;

means for OR'"'"'ing, said output data signals from said matrix elements to produce a raw squashed data output signal; and

means for transmitting said raw squashed data output signal; and

an output layer comprising;

an output processing element, comprising;

averaging means

for receiving said raw squashed data output signal; and

for producing, as the network output value, an output data signal which is the time average of said raw squashed data output signal.

5. The neural network of claim 4, wherein said matrix elements further comprise:

weight value means for providing a weight value; and

wherein said duty cycle conversion means receives said weight value, and

wherein said duty cycle of said output data signal is proportional to the product of said input data value and said weight value.

6. The neural network of claim 5,

wherein the received input vector elements are in the form of pulse trains with the corrseponding input vector values being represented by the frequency of the pulse trains;

and

wherein the frequency of said output data signal of said duty cycle conversion means is equal to the frequency of the input vector element pulse train and

wherein the width of a pulse of said output data signal is proportional to said weight value.

7. A back propagation neural network

receiving an input vector comprised of input vector elements, each representing an input vector element value,

for receiving an error vector element as an error vector element value,

and, in response thereto and as a function thereof, for outputing a network output value;

said network comprising;

a plurality of matrix elements, corresponding in number to the number of elements in the input vector, each comprising;

means for receiving a corresponding element of the input vector as an input data value;

weight value means for providing a weight value;

means for receiving an error value from a combining element;

duty cycle conversion means

for receiving said input data value,

for producing an output data value dependent on said input data value,

for receiving said weight value,

for producing an output data signal as a pulse train whose duty cycle is proportional to the product of said input vector value and said weight value;

means for transmitting said output data signal; and

weight update means

for producing a weight update value as a function of said error value and said input data value; and

for combining said weight update value with said weight value to produce an updated weight value for said weight value means;

a combining element, comprising;

means for receiving said output data signals from said matrix elements;

means for OR'"'"'ing said output data signals to produce a raw squashed data output signal;

means for transmitting said raw squashed data output signal; and

means for receiving an error value from an output processing element and distributing it to said matrix elements;

an output layer comprising;

an output processing element, comprising;

averaging means

for receiving said raw squashed data output signal, and

for producing, as the network output vector, an output data signal which is the time average of said raw squashed data output signals;

error receiving means for receiving an error vector element value;

error conversion means for producing an error value which is a function of said error vector element value; and

error transmission means for transmitting said error value.

8. The network of claim 7, wherein

said weight update value produced is the product of said error value and said input data value, averaged over the time when no pulses are present in said raw squashed data output signal.

9. The network of claim 7,

wherein said combining element is implemented as multiple transmission lines;

and

wherein said raw squashed data signal and said error value are transmitted seperately on seperate transmission lines.

10. The network of claim 9,

wherein said error value is transmitted during the time when no pulses from said raw squashed data signal are present on said transmission lines.

11. The network of claim 7, wherein

said combining element includes transmission lines and

said raw squashed data signal and said error value are multiplexed onto said transmission lines.

12. A back propagation neural network

for receiving an input vector comprised of input vector elements, each representing an input vector element value;

for receiving an error vector comprised of one or more error vector elements, each representing an error vector element value;

and, in response thereto and as a function thereof,

for transmitting an output vector comprised of one or more output vector elements, corresponding in number to the number of error vector elements, each representing an output vector element value;

said network comprising;

a processing element input layer comprising;

a plurality of input processing elements, corresponding in number to the number of input vector elements, one input processing element being associated with each input vector element;

each input processing element being associated with a particular input layer horizontal combining element means;

each input processing element including;

means for receiving said input processing element'"'"'s associated input vector element value and

means for transmitting a data signal which is a function of said input processing element'"'"'s associated input vector element value to said input processing element'"'"'s associated input layer horizontal combining element means;

a processing element hidden layer comprising;

at least one hidden processing element,

each hidden processing element

being associated with a particular vertical combining element means; and

being associated with a particular hidden layer horizontal combining element means;

each hidden processing element comprising;

forward data propagation means including;

means for receiving a squashed data signal from said hidden processing element'"'"'s associated vertical combining element means; and

for transmitting to said hidden processing element'"'"'s associated hidden layer horizontal combining element means a data signal which is a function of said received squashed data signal; and

error back propagation means including;

means for receiving a combined error signal from said hidden processing element'"'"'s associated hidden layer horizontal combining element means;

means for receiving a squashed data signal from said hidden processing element'"'"'s associated vertical combining element means;

means for producing a feedback error value which is a function of said combined error signal and said squashed data signal; and

means for transmitting said feedback error value to said hidden processing element'"'"'s associated vertical combining element means;

a processing element output layer comprising;

one or more output processing elements, corresponding in number to the number of error vector elements, one output processing element being associated with each error vector element,

each output processing element being associated with a particular vertical combining element means,

each output processing element comprising;

forward data propagation means comprising;

means for receiving a squashed data signal from said output processing element'"'"'s associated vertical combining element means; and

means for transmitting an output vector element value which is a function of said squashed data signal; and

backwards error propagation means including;

means for receiving an error vector element value;

means for producing a feedback error value which is a function of said error vector element value; and

means for transmitting said feedback error value to said output processing element'"'"'s associated vertical combining element means;

a plurality of input matrix elements,

each forming a data flow coupling path between an input layer horizontal combining element means and a vertical combining element means;

there being at least one input matrix element coupled to each input layer horizontal combining element;

each input matrix element comprising;

weight value means for providing a weight value;

forward data propagation means including

means for receiving said data signal from said input matrix element'"'"'s associated input layer horizontal combining element means;

means for receiving said weight value from said input matrix element'"'"'s said weight value means;

duty cycle conversion means for producing a weighted data signal as a pulse train whose duty cycle is proportional to the product of said data signal and said weight value; and

means for transmitting said weight data value to said input matrix element'"'"'s associated vertical combining element means; and

weight update means including;

means for receiving said data signal from said input matrix element'"'"'s associated input layer horizontal combining element means;

means for receiving a feedback error value from said input matrix element'"'"'s associated vertical combining element means;

means for producing a weight update value as a function of said feedback error value and said pulse data signal; and

means for combining said weight update value with said weight value to produce an updated weight value;

one or more hidden matrix elements,

each forming a data flow coupling path between a hidden layer horizontal combining element means and a vertical combining element means;

at least one hidden matrix element being associated with each hidden layer horizontal combining element;

at least one input matrix element or one hidden matrix element being associated with each vertical combining element;

each said hidden matrix element comprising;

weight value means for providing a weight value;

forward data propagation means including

means for receiving said data signal from said hidden matrix element'"'"'s associated hidden layer horizontal combining element means;

means for receiving said weight value from said hidden matrix element'"'"'s said weight value means;

duty cycle conversion means for producing a weighted data signal as a pulse train whose duty cycle is proportional to the product of said data signal and said weight value; and

means for transmitting said weight data value to said input matrix element'"'"'s associated vertical combining element means;

weight update means including;

means for receiving said data signal from said hidden matrix element'"'"'s associated hidden layer horizontal combining element means;

means for receiving a feedback error value from said hidden matrix element'"'"'s associated vertical combining element means;

means for producing a weight update value as a function of said feedback error value and said pulse data signal; and

means for combining said weight update value with said weight value to produce an updated weight value; and

weighted backward error propagation means including;

means for receiving a feedback error value from said hidden matrix element'"'"'s associated hidden layer horizontal combining element means;

means for producing a weighted feedback error value as a function on said weight value and said received feedback error value; and

means for transmitting said weighted feedback error value to said hidden matrix element'"'"'s associated vertical combining element means;

a plurality of vertical combining element means,

each being associated with

a single processing element, and

one or more matrix elements;

each vertical combining element means comprising;

means for OR'"'"'ing said pulsed weighted data signals received from said vertical combining element'"'"'s said associated matrix elements to produce a raw squashed data output signal;

means for transmitting said raw squashed data output signal to said vertical combining element'"'"'s associated processing element;

means for receiving said feedback error value from said vertical combining element'"'"'s associated processing element; and

means for distributing said feedback error value to said vertical combining element'"'"'s associated matrix elements.

a plurality of hidden layer horizontal combining element means,

each being associated with

a single hidden processing element, and

one or more hidden matrix elements,

each hidden layer horizontal combining element means comprising;

means for receiving said data signal from said hidden layer horizontal combining element'"'"'s said associated hidden processing element;

means for transmitting said data signal to said hidden layer horizontal combining element'"'"'s said associated hidden matrix elements;

means for receiving said feedback error values from said hidden layer horizontal combining element'"'"'s said associated hidden matrix elements;

means for producing a combined error value which is a function of said feedback error values; and

means for transmitting said combined error value to said hidden layer horizontal combining element'"'"'s associated hidden processing element; and

a plurality of input layer horizontal combining element means,

each being associated with

a single input layer processing element, and

one or more input matrix elements;

each input layer horizontal combining element means comprising;

means for receiving said data signal from said input layer horizontal combining element'"'"'s said associated input layer processing element; and

means for transmitting said data signal to said input layer horizontal combining element'"'"'s said associated input matrix elements.

13. The neural network of claim 12, wherein:

said data signal transmitted from each said input processing element is in the form of a pulse train, the frequency of said pulse train being a function of said corresponding input vector element value;

wherein said weighted data signal of said input matrix element'"'"'s duty cycle conversion means has a frequency equal to said frequency of its said received data signal from said input vector element pulse train; and

wherein said pulses of said weighted data signal of said input matrix elements has a width proportional to said weight value.

14. The neural network of claim 12, wherein said vertical combining element means comprises a plurality of transmission lines and wherein said raw squashed data output signal and said feedback error value are transmitted on separate transmission lines.

15. The neural network of claim 12,

said backwards error propagation means of an output processing element further includes:

means for receiving a squashed data signal from said output processing element'"'"'s associated vertical combining element means; and

wherein said feedback error value, produced by said means for producing a feedback error value, is also a function of said squashed data signal.

16. The neural network of claim 15, wherein:

said feedback error value, produced by said means for producing a feedback error value of said backwards error propagation means of an output processing element, is proportional to the product of the off-time of said error vector element value and said error vector element value.

17. The neural network of claim 12,

said backwards error propagation means of an output processing element further includes:

means for receiving a squashed data signal from said output processing element'"'"'s associated vertical combining element means; and

wherein said feedback error value, produced by said means for producing a feedback error value, is also a function of said squashed data signal.

18. The neural network of claim 17, wherein:

said feedback error value, produced by said means for producing a feedback error value of said backwards error propagation means of an output processing element, is proportional to the product of the off-time of said error vector element value and said error vector element value.

19. The neural network of claim 12, wherein each said vertical combining element means includes a transmission line and said raw squashed data output signal and said feedback error value are multiplexed onto said transmission line.

20. The neural network of claim 19, wherein said feedback error value is transmitted during the off-time of said raw squashed data signal.

21. A neural network for receiving an input vector comprised of input vector elements, each representing an input vector element value, and, in response thereto and as a function thereof, for outputing a network output value;

said network comprising;

a plurality of excitatory matrix elements, corresponding in number to the number of input vector elements, one excitatory matrix element being associated with each input vector element, each excitatory matrix element comprising;

means for receiving a corresponding input vector element as an input data value;

duty cycle conversion means

for receiving said input data value, and

for producing an excitatory output data signal as a pulse train whose duty cycle is proportional to said input data value; and

means for transmitting said excitatory output data signal;

an excitatory combining element, comprising;

means for receiving said excitatory output data signals from said excitatory matrix elements;

means for OR'"'"'ing said excitatory output data signals to produce a raw excitatory squashed data output signal; and

means for transmitting said raw excitatory squashed data output signal;

a plurality of inhibitory matrix elements, corresponding in number to the number of input vector elements, one being associated with each inhibitory matrix element, each inhibitory matrix element comprising;

means for receiving a corresponding input vector element value as an input data value;

duty cycle conversion means for receiving said input data value, and for producing an inhibitory output data signal, as a pulse train whose duty cycle is proportional to said input data value; and

means for transmitting said inhibitory output data signal;

an inhibitory combining element, comprising;

means for receiving said inhibitory output data signals from said inhibitory matrix elements;

means for OR'"'"'ing said inhibitory output data signals to produce a raw inhibitory squashed data output signal; and

means for transmitting said raw squashed inhibitory data output signal;

an output processing element, comprising;

means for receiving said raw excitatory squashed data output signal;

means for receiving said raw inhibitory squashed data output signal; and

means for producing, as the network output value, an output data signal which is a function of the time average of the logical AND of said raw excitatory squashed data output signal and the complement of said raw inhibitory squashed data output signal.