Neuromorphic learning networks

• US 4,874,963 A
• Filed: 02/11/1988
• Issued: 10/17/1989
• Est. Priority Date: 02/11/1988
• Status: Expired due to Fees

• Current Assignee: TTI Inventions A LLC (Intellectual Ventures LLC)
• Original Assignee: Bell Communications Research, Inc. (Telefonaktiebolaget LM Ericsson)
• Inventors: Alspector, Joshua
• Primary Examiner(s): HUDSPETH, DAVID R
• Application Number:
  US07/155,150
• Time in Patent Office: 614 Days
• Field of Search: 307/201, 364/807, 364/513, 364/200 MS File, 364/900 MS File, 377/2
• US Class Current: 706/34
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: G06N 3/065 Analogue means

Abstract

A neuron network which achieves learning by means of a modified Boltzmann algorithm. The network may comprise interconnected input, hidden and output layers of neurons, the neurons being "on-off" or threshold electronic devices which are symmetrically connected by means of adjustable-weight synapse pairs. The synapses comprise the source-drain circuits of a plurality of paralleled FETs which differ in resistance or conductance in a binary sequence. The synapses are controlled by the output of an Up-Down counter, the reading of which is controlled by the results of a correlation of the states of the two neurons connected by the synapse pairs following the application of a set of plus and minus training signals to selected neurons of said network. A noise generator comprising a thermal noise source is provided for each neuron for the purpose of simulated annealing of the network.

Claims

1. A neuron network comprising interconnected input, hidden and output layers of neurons, said neurons comprising "on-off" or threshold electronic devices,

all of said neurons which are connected being symmetrically connected by means of automatically-adjustable synapse pairs, each of said synapses comprises a plurality of drain-source circuits of field effect transistors, said source-drain circuits having resistances or conductances which differ from each other by ratios of 2 to 1,

whereby different combinations of said source-drain circuits can be switched in parallel to provide any resistance or conductance, in a binary sequence, for each of said synapses, said synapses being controlled by the output of an Up-Down counter, and

means to control the reading of said counter depending on the results of a correlation of the states of each connected pair of said neurons following the application of sets of training signals to said neurons network.

2. The network of claim 1 wherein each neuron is provided with a circuit for accomplishing simulated annealing, said circuit comprising means to perturb the threshold of each of said neurons by means of amplified thermal noise which, during each annealing cycle, starts at a high amplitude and decreases in ramp-fashion to a zero amplitude, whereby the network simultaneously relaxes to a state of minimum energy during the application of said training signals.

3. The network of claim 1 wherein each of said neurons comprises a differential amplifier.

4. An electronic neuron network comprising a plurality of step-function or "on-off" electronic neurons arranged in input, hidden and output layers, which neurons comprise differential amplifiers,

selected pairs of the neurons of said network being connected by means of pairs of synapses,

digital control circuitry for automatically varying the synaptic weight of each of the synapses of said pairs of synapses, in unison,

means to apply plus and minus phase training patterns to the neurons in the input and output layers of said network,

means to apply uncorrelated Gaussian thermal noise to each of said neurons to accomplish simulated annealing during the application of said plus and minus phase training patterns, said noise varying in ramp-fashion from a high to a low amplitude during each annealing cycle,

and a logic circuit provided for each connected pair of neurons, said logic circuit comprising means to measure the correlation of each of the said plus and minus phase training patterns and means to control said digital control circuitry in accordance with the results of said correlations.

5. The network of claim 4 wherein said synaptic weights comprise the source-drain circuits of a plurality of field effect transistors of different resistances or conductances with a conductance sequence of 1:

2;

4 etc., and wherein the synaptic weights are varied by switching different combinations of said source-drain circuits in parallel.

6. A neuron network comprising,

a plurality of step neurons,

means to symmetrically connect said neurons by means of direct and reciprocal variable-weight synapses,

means to apply sets of training signals to selected ones of the neurons of said network,

each of said neurons being provided with a separate and thus uncorrelated source of thermal noise, means to apply said thermal noise to each of said neurons to accomplish simulated annealing of said network during the application of each of said training signals,

means to correlate the states of said connected neurons following the application of each of said training signals, and

means to automatically adjust said synapses in response to said correlations.

7. The network of claim 6 wherein each of said neurons comprises a differential amplifier with the step therein at zero differential input volts and wherein one input of each of said neurons comprises a threshold voltage which is the inverse of the desired threshold of that neuron, said threshold voltage being derived from a permanently "on" true neuron.

8. A method for teaching a neuron network to recognize input patterns repeatedly applied thereto, which network utilizes simulated annealing to relax to a state of low energy, said method comprising the steps of;

correlating the states of each pair of connected neurons following each cycle of simulated annealing, then adjusting the synaptic weights of each of said pairs of neurons by the minimum step of said synaptic weight using only the correlation data obtained from said connected pairs of neurons.

9. A neuron network comprising input, hidden and output layers of neurons, said neurons comprising "on-off" or bistable electronic devices, and wherein all connected neuron pairs are symmetrically connected by means of pairs of automatically adjustable resistors which comprise synapses of adjustable weight;

means to sequentially apply plus and minus phase training signals to said network,

means to achieve simulated annealing of said network during the application of said training signals,

each pair of connected neurons having associated therewith a logic and control circuit for obtaining correlation figures related to the states of each of said pairs of connected neurons following the simulated annealing of said network during each application of said plus and minus training signals, said correlation figures being positive (+1) if the states of said two connected neurons are both the same and negative (-1) if the states thereof are different, and

an Up-Down counter controlled by said control circuit, said counter controlling the weight of each of said pair of synapses,

said logic and control circuitry comprising means to increment said counter and said synaptic weights if the said correlation following said plus phase is greater than the said correlation following said minus phase, and also means to decrement the reading of said counter and the weight of each said pair of synapses if the correlation following said plus phase is less than the said correlation following said minus phase, and leaving said synaptic weights unchanged if the said plus and minus correlations are the same.

10. A neuron network comprising bistable electronic neurons, means to sequentially apply plus and minus phase training signals to said network, means to apply a different electronically-generated noise signal to each of said neurons during each application of said plus and minus phase training signals, each pair of connected neurons being connected by means of adjustable-weight synapses, and means to automatically adjust the synaptic weights of each of said pairs of connected neurons following each cycle of application of said training signals by comparing only the correlation of the states of each said connected neuron pair.

11. A neuron network adapted to utilize a Boltzmann algorithm to accomplish learning, said network comprising means for performing simulated annealing, said means comprising separate electronic noise generators connected to the inputs of each neuron in said network, the amplitude of the noise output of said noise generators being adjustable to accomplish simulated annealing.

12. The network of claim 11 wherein said noise generators comprise a thermal noise source which is connected to the input of a variable-gain amplifier, the gain of said amplifier being controlled by a ramp signal generator which during each annealing cycle starts at a high voltage and decreases to a low voltage, whereby the noise at the output of said amplifier starts out with a high amplitude and decreases toward zero amplitude.