Sleep refreshed memory for neural network

• US 4,926,064 A
• Filed: 07/26/1989
• Issued: 05/15/1990
• Est. Priority Date: 07/22/1988
• Status: Expired due to Fees

• Current Assignee: Syntonic Systems Incorporated 20790 NW Quail Hollow Dr Portland Or
• Original Assignee: Syntonic Systems Incorporated
• Inventors: Tapang, Carlos C.
• Primary Examiner(s): Hudspeth, David
• Application Number:
  US07/387,373
• Time in Patent Office: 293 Days
• Field of Search: 307/201, 364/513, 364/807
• US Class Current: 706/26
• CPC Class Codes: G06N 3/065 Analogue means

Abstract

A method and apparatus are disclosed for implementing a neural network having a sleep mode during which capacitively stored synaptic connectivity weights are refreshed. Each neuron outputs an analog activity level, represented in a preferred embodiment by the frequency of digital pulses. Feed-forward synaptic connection circuits couple the activity level outputs of first level neurons to inputs of second level neurons, and feed-back synaptic connection circuits couple outputs of second level neurons to inputs of first level neurons, the coupling being weighted according to connectivity weights stored on respective storage capacitors in each synaptic connection circuit. The network learns according to a learning algorithm under which the connections in both directions between a particular first level neuron and a particular second level neuron are strengthened to the extent of concurrence of high activity levels in both the first and second level neurons, and weakened to the extent of concurrence of a high activity level in the second level neuron and a low activity level in the first level neuron. The network is put to sleep by disconnecting all environmental inputs and providing a non-specific low activity level signal to each of the first level neurons. This causes the network to randomly traverse its state space with low intensity resonant firings, each state being visited with a probability responsive to the initial connectivity weights of the connections which abut the second level neuron representing such state. Refresh is accomplished since the learning algorithm remains active during sleep. Thus, the sleep refresh mechanism enhances the contrast in the connectivity terrain and strengthens connections that would otherwise wash out due to lack of visitation while the system is awake. A deep sleep mechanism is also provided for preventing runaway strengthening of favored states, and also to encourage Weber Law compliance.

Claims

1. A neural network for use with a learning algorithm, a sleep refresh algorithm, and a plurality of environmental input signals, each of said environmental input signals having an activity level, comprising:

a plurality of N neuronal outputs, each carrying a signal having an activity level, said plurality including M first level neuronal outputs and N-M second level neuronal outputs;

feed-forward response means for generating on a j'"'"'th one of said second level neuronal outputs, M<

j≦

N, a j'"'"'th signal having an activity level responsive to the temporal and spatial sum of the activity levels of said signals on all i'"'"'th ones of said first level neuronal outputs, i=1,. . . ,M, each as weighted by a respective connectivity weight w_ij ;

means for modifying at least one of said connectivity weights w_ij in accordance with said learning algorithm; and

sleep activated means for, during a sleep period, isolating said neuronal outputs from said environmental inputs and adjusting at least some of said connectivity weights w_ij according to said sleep refreshed algorithm.

2. A neutron circuit suitable for use in a neural network comprising:

a summing junction, a constant voltage node and a summing capacitor coupled between said summing junction and said constant voltage- node;

an inverting integration having an input and a voltage output;

an oscillator capacitor; and

an oscillator circuit having an output coupled to said input of said inverting integrator, and further having means for transferring charge from said summing capacitor to said oscillator capacitor when the voltage across said oscillator capacitor is less than said voltage output of said inverting integrator, and for discharging said oscillator capacitor when the voltage across said oscillator capacitor exceeds said voltage output of said inverting integrator.

3. A neural network for use with input stimuli, comprising:

a plurality of neuronal circuits, each of said neuronal circuits having an output responsive according to a respective connectivity value to the output of at least on other one of said neuronal circuits;

learning means for altering said connectivity values in response to said input stimuli according to a learning algorithm; and

refresh means for altering said connectivity values independently of said input stimuli according to a refresh algorithm.

4. A neural network according to claim 3, further comprising mode select means for selecting between a sleep mode and a waking mode, for enabling said learning means during said waking mode and disabling said learning means during said sleep mode, and for enabling said refresh means during said sleep mode and disabling said refresh means during said waking mode.

5. A neural network according to claim 3, wherein said plurality of neuronal circuits includes M first neuronal circuits and N-M second meuronal circuits, th- output of each of said first and second neuronal circuits carrying a signal having an activity level, the activity level of the output of each j'"'"'th one of said second neuronal circuits being responsive according to one of said respective connectivity values w_ij to the activity level of the output of each i'"'"'th one of said first neuronal circuits, the activity level of the output of each i'"'"'th one of said first neuronal circuits being responsive according to one of said respective connectivity values w_ji to the activity level of the output of each j'"'"'th one of said second neuronal circuits,

said neural network further comprising first means for stimulating the activity levels on th- outputs of said first neuronal circuits toward higher activity levels in response to high activity levels on respective ones of said input stimuli,

and wherein said learning means comprises first memory adjustment means of'"'"' increasing at least one of the connectivity values w_ij and w_ji in response to simultaneously high activity levels on the output of the i'"'"'th first neutronal circuit and on the output of the j'"'"'th second neuronal circuit, and for decreasing said at least one of the connectivity values w_ij and w_ji in response to a simultaneous high activity level on the output of the j'"'"'th second neuronal circuit and low activity level on the output of the i'"'"'th first neuronal circuit.

6. A neural network according to claim 5, wherein said refresh means comprises:

said first memory adjustment means;

means, active during a sleep mode, disabling said first means for stimulating; and

means, active during said sleep mode, for inducing high activity levels on the outputs of said first neuronal circuits in a pseudorandom sequence.

7. A neuronal network according to claim 6, wherein said means for inducing comprises second means for stimulating the activity levels on the outputs of all of said first neuronal circuits equally toward higher activity levels.

8. A neuronal network according to claim 6'"'"' wherein said means for inducing comprises second means for stimulating the activity levels of the outputs of all of said second neuronal circuits equally toward higher activity levels.

9. A neural network according to claim 6, wherein said pseudorandom sequence is such as to tend said connectivity values toward values for which high activity levels appear with equal probability on all committed ones of said second neuronal circuits.

10. A neuronal network according to claim 6, further comprising:

second memory adjustment means for decreasing the connectivity value w_ij in response to simultaneously high activity levels on the output of the i'"'"'th first neuronal circuit and on the output of the j'"'"'th second neuronal circuit;

favored neuron detection means, active during said sleep mode, for detecting a favored one of said second neuronal circuits; and

means, responsive to said favored neuron detection means, for disabling said first memory adjustment means and enabling said second memory adjustment means.

11. A neural network according to claim 10,

wherein said first memory adjustment means increases both of said connectivity values w_ij and w_ji in response to simultaneously high activity levels on the output of the i'"'"'th first neuronal circuit and on the output of the j'"'"'th second neuronal circuit, and decreases both said connectivity values w_ij and w_ji in response a simultaneously high activity level on the output of the j'"'"'th second neuronal circuit and low, activity level on the output of the i'"'"'th first neuronal circuit;

and wherein said second memory adjustment means further increases said commectivity value w_ji in response to the simultaneously high activity levels on the output of the i'"'"'th first neuronal circuit and on the output of the j'"'"'th second neuronal circuit.

12. A neural network for use with a learning algorithm, a sleep refresh algorithm, and a plurality of environmental input signals, each of said environmental input signals having an activity level, comprising:

a plurality of N neuronal outputs, each carrying a signal having an activity level, said plurality including N first level neuronal outputs and N -M second level neuronal outputs;

feed-forward response means for generating on a j'"'"'th one of said second level neuronal outputs, M<

j≦

N, a j'"'"'th signal having an activity level responsive to a temporal and spatial sum of the activity levels of said signals on all i'"'"'th ones of said first level neuronal outputs, i=1, . . . ,M, each as weighed by a respective connectivity weight w_ij ;

means for modifying at least one of said connectivity weights w_ij in accordance with said learning algorithm; and

sleep activated means for, during a sleep period, isolating said neuronal outputs from said environmental inputs and adjusting at least some of said connectivity weights w_ij according to said sleep refresh algorithm.

13. A neural network according to claim 12, wherein said feed-forward response means generates said j'"'"'th signal such that the activity level of said j'"'"'th signal is positively responsive to a time integration of the sum of the activity levels of said signals on all i'"'"'th ones of said first level neuronal outputs as weighted by said respective connectivity weights w_ij.

14. A neural network according to claim 12, wherein said feed-forward response means generates said j'"'"'th signal such that the activity level of said j'"'"'th signal is negatively responsive to a time integration of the activity level of said j'"'"'th signal.

15. A neural network according to claim 13, wherein said feed-forward response means generates said j'"'"'th signal such that the activity level of said j'"'"'th signal is further negatively responsive to a time integration of the activity level of said j'"'"'th signal.

16. A neural network according to claim 12, wherein said means for modifying at least one of said connectivity weights w_ij includes means for weakening said at least one of said connectivity weights w_ij in response to the presence of some predetermined system condition.

17. A neural network according to claim 16, wherein said predetermined system condition is the concurrence of a low activity level on the signal on the i'"'"'th first level neuronal output with a high activity level on said signal on said j'"'"'th second level neuronal output.

18. A neural network according to claim 12, further comprising means for storing each of said connectivity weights w_ij imperfectly, such that each of said connectivity weights w_ij decays over time.

19. A neural network according to claim 12, further comprising:

a plurality of F third level neuronal outputs, each carrying a signal having an activity level; and

second-to-third level feed-forward response means for generating on a k'"'"'th one of said third level neuronal outputs a k'"'"'th signal having an activity level responsive to the temporal and spatial sum of the activity levels of said signals on all j'"'"'th ones of said second levels neuronal outputs, each as weighted by a respective connectivity weight w_jk, j=M+1,. . . ,N and N<

k≦

N+P.

20. A neural network according to claim 12, wherein the activity level of said j'"'"'th signal is further responsive to the temporal and spatial sum of the activity levels of said signals on all other k,th ones of said second level neuronal outputs, each as weighted by a respective connectivity weight w_kj, k=M+1,. . . ,N, k≠

j, the responsiveness of the activity level of said j'"'"'th signal to the activity levels of said second level neuronal outputs having a sense opposite that of said j'"'"'th signal to the activity levels of said first level neuronal outputs.

21. A neural network according to claim 20, wherein said activity level of said j'"'"'th signal is positively responsive to the activity levels of said first level neuronal outputs and negatively responsive to the activity levels of said second level neuronal outputs.

22. A neural network according to claim 20, wherein said connectivity weights w_kj, k=M+1,. . . , N, k≠

j, are all equal and substantially constant with time.

23. A neural network according to claim 12, wherein said feed-forward response means comprises:

a plurality of M synaptic circuit means, each having an output, each i'"'"'th one of said synaptic circuit means i=1,. . . ,M, being for generating on said output of said i'"'"'th synaptic circuit means a signal having an activity level substantially equal to w_ij times the activity level of the signal on a respective i'"'"'th one of said first level euronal outputs; and

neuronal circuit means for generating on said j'"'"'th one of said second level neuronal outputs a signal having an activity level responsive to the temporal and spatial sum of the activity levels of said signals on said outputs of all of said synaptic circuit means.

24. A neural network according to claim 12, wherein said signals on each of said neuronal outputs have a respective frequency, and wherein the activity level of said signal on each given one of said neuronal outputs is represented by the frequency of said given one of said signals, a higher frequency corresponding to a higher activity level and a lower frequency corresponding to a lower activity level.

25. A neural network according to claim 23, wherein said signals on each of said neuronal outputs and on said outputs of each of said synaptic circuit means have a respective frequency, and wherein the activity level of said signal on each given one of said neuronal outputs and on said output of each given one of said synaptic circuit means is represented by the frequency of said given one of said signals, a high frequency corresponding to a higher activity level and a lower frequency corresponding to a lower activity level.

26. A neural network according to claim 25, wherein said signals on each of said neuronal outputs are represented by respective voltage waveforms and said signals on said outputs of each of said synaptic circuit means are represented by respective current waveforms.

27. A neural network according to claim 12, further comprising means for inducing said sleep period periodically.

28. A neural network according to claim 12, further comprising:

feed-back response means for generating on an i'"'"'th one of said first level neuronal outputs, 1≦

i≦

M, an i'"'"'th signal having an activity level responsive to the temporal and spatial sum of the activity levels of said signals on all j'"'"'th ones cf said second level neuronal outputs, j=M+1,. . . ,N, each as weighted by a respective connectivity weight w_ji, and of the activity level of at least one of said environmental input signals; and

means for modifying at least one of said connectivity weights w_ji, M<

j≦

N, in accordance with said learning algorithm,

wherein said sleep activated means is further for adjusting at least one of said connectivity weights w_ji, j=M+1,. . . ,N during said sleep period.

29. A neural network according to claim 19, further comprising third-to,-first level feed-back response means for generating ,on an i th one of said first level neuronal outputs, 1≦

i≦

M, an i'"'"'th signal having an activity level responsive to the temporal and spatial sum of the activity levels of said signals on all k'"'"'th ones of said third level neuronal outputs, k=N+1,. . . ,N+P, each as weighted by a respective connectivity weight w_ki, and of the activity level of at least one of said environmental inputs signals; and

means for modifying at least one of said connection weights w_ki, N<

k≦

N+P, in accordance with said learning algorithm,

wherein said sleep activated means is further for adjusting said connectivity weights w_ki, k=N+1,. . . ,N+P, during said sleep period.

30. A network according to claim 28,

wherein said feed-forward response means generates said j'"'"'th signal such that the activity level of said j'"'"'th signal is positively responsive to a time integration of the sum of the activity levels of said signals on all i'"'"'th ones of said first level neuronal outputs as weighted by said respective connectivity weights w_ij, is further negatively responsive, according to substantially constant respective inhibitory connectivity weights, to a time integration of the sum of the activity levels on all other ones of said second level neuronal outputs, and is further negatively responsive to a time integration of the activity level of said j'"'"'th signal;

wherein said feed-back response means generates said i'"'"'th signal such that the activity level of said i'"'"'th signal is positively responsive to a time integration of the sum of the activity levels of said signals on all j'"'"'th ones of said second level neuronal outputs as weighted by said respective connectivity weights w_ji, plus a time integration of the activity level of an i'"'"'th one of said environmental input signals, and is further negatively responsive to a time integration of the activity level of said i'"'"'th signal;

and wherein said sleep activated means is further for adjusting at least some of said connectivity weights w_ji, during said sleep period.

31. A network according to claim 30, further comprising a sleep neuronal output carrying a sleep signal having an activity level, wherein said feed-forward response means generates said j'"'"'th signal such that the activity level of said j'"'"'th signal is further positively responsive to the activity level of said sleep signal.

32. A network according to claim 31, wherein said sleep activated means is further for increasing the activity level of said sleep signal during said sleep period and decreasing the activity level of said sleep signal outside of said sleep period.

33. A network according to claim 30, further comprising a sleep neuronal output carrying a sleep signal having an activity level, wherein said feed-back response means generates said i'"'"'th signal such that the activity level of said i'"'"'th signal is further positively responsive to the activity level of said sleep signal.

34. A network according to claim 30, wherein said sleep activated means is further for generating secondary resonances during said sleep period, in which the activity levels of different ones of said second level output signals rise and fall according to a pseudorandom sequence, and wherein said sleep refresh algorithm revises said pseudorandom sequence such that the activity levels of all committee ones of said second level output signals rise and fall with greater equi-probability.

35. A network according to claim 30, wherein said learning algorithm includes increasing each of said connectivity weights w_ij by an amount commensurate with any concurrence of a high activity level of both the i'"'"'th and j'"'"'th neuronal output signals, and decreasing each of said connectivity weight w_ij by an amount commensurate with any concurrence of a low activity level on said i'"'"'th neuronal output signal and a high activity level on said j'"'"'th neuronal output signal.

36. A network according to claim 35, wherein said sleep activated means is further for generating secondary resonances during said sleep period, in which the activity levels of different ones of said second level output signals rise and fall according to a pseudorandom sequence, and wherein said sleep refresh algorithm revises said pseudorandom sequence such that the activity levels o&

all committed ones of said second level output signals rise and fall with greater equi-probability.

37. A network according to claim 35, wherein said sleep activated means is further for generating secondary resonances during said sleep period, in which the activity levels of different ones of said second level neuronal output signals rise and fall according to a pseudorandom sequence, and wherein said refresh algorithm includes strengthening at least some of said connectivity weights and weakening, in response to undesirably frequent secondary resonances involving a particular j'"'"'th neuronal output, each of said connectivity weights w_iJ by an amount commensurate with any concurrence of a high activity level on both said i'"'"'th and J'"'"'th neuronal output signals.

38. A neural network for use with a plurality of environmental input signals each having an activity level, comprising:

a plurality of first level neuronal circuits each having at least one input. each of said inputs of each of said first level neuronal circuits carrying an input signal having an activity level, each of said first level neuronal circuits further having an output and means for providing on said output of said first level neuronal circuit an output signal having an activity level at least partially responsive to a time integration of the sum of the activity levels of said input signals of said first level neuronal circuit;

a group of second level neuronal circuits each having at least one input, each at said inputs of each of said second level neuronal circuits carrying an input signal having an activity level, each of said second level neuronal circuits further having an output and means for providing on said output of said second level neuronal circuit an output signal having an activity level at least partially responsive to a time integration of the sum of the activity levels of said input signals of said second level neuronal circuit and at least partially counter-responsive to a time integration of the sum of the activity levels of the output signals of all other ones of said second level neuronal circuits in said group.

a plurality of feed forward connection circuits each having an input carrying an input signal having an activity level, storage means for storing a respective connectivity strength, an output and means for providing on said output of said feed-forward connection circuit an output signal having an activity level responsive to the activity level of said input signal of said feed-forward connection circuit as weighted by said respective connectivity strength, said input of each of said feed-forward connection circuits being coupled to receive the output signal of a respective one of said first level neuronal circuits and said output of each of said feed-forward connection circuits being coupled to provide said output signal of said feed-forward connection circuit to one of said inputs of a respective one of said second level neuronal circuits in said group, the output of each of said first level neuronal circuits being coupled to an input of all of said second level neuronal circuits in said group via respective ones of said feed-forward connection circuits;

a plurality of feed-back connection circuits each having an input carrying an input signal having an activity level, storage means for storing a respective connectivity strength, an output and means for providing on said output of said feed-back connection circuit an output signal having an activity level responsive to the activity level of said input signal of said feed-back connection circuit as weighted by said respective connectivity strength of said feed-back connection circuit, said input of each of said feed-back connection circuits being coupled to receive the output signal of a respective one of said second level neuronal circuits and said output of each of said feed-back connection circuit being coupled to provide said output signal of said feed-back connection circuit to one of said inputs of a respective one of said first level neuronal circuits, the output of each of said second level neuronal circuits in said group being coupled to an input of all of said first level neuronal circuits via respective ones of said feed-back connection circuits. said means for providing. in all of said first and second level neuronal circuits and said feed-forward and feed-back connection circuits, being such that the net feedback of activity levels around any loop is non-negative; and

learning means for adjusting said connectivity strengths stored in said storage means in said feed-forward and feed-back convection circuits in accordance with a learning algorithm.

39. A network according to claim 38, wherein one of said inputs to one of said first level neuronal circuits is coupled to receive a respective one of said environmental input signals.

40. A network according to claim 38, wherein said means, in each gives one of said second level neuronal circuits in said group, for providing an output signal having an activity level at least partially counter-responsive to a time integration of the sum of the activity levels of the output signals of all other ones of said second level neuronal circuits in said group comprises a plurality of inhibitory connection circuits each having an input carrying an input signal having an activity level, an output and means for providing on said output of said inhibitory connection circuit an output signal having an activity level counter-responsive to the activity level of said input signal of said inhibitory connection circuit, said outputs of all of said inhibitory connection circuits being coupled to respective inputs of said given one of said second level neuronal circuit and said input of each of said inhibitory connection circuits being coupled to receive said output signal of a respective one of said other second level neuronal circuits, the output of each of said second level neuronal circuits in said group being coupled to an input of all other ones of said second level neuronal circuits in said group via respective ones of said inhibitory connection circuits.

41. A network according to claim 38, wherein said means for providing, in each given one of said first and second level neuronal circuits, is further such that the time rate of change of the activity level of said output signal of said given one of said neuronal circuits is counter-responsive to the instantaneous activity level of said output signal of said given one of said neuronal circuits so as to dampen resonances due to said net non-negative feedback of activity levels around any loop which includes said given one of said neuronal circuits.

42. A network according to claim 41,

wherein said means for providing, in each given one of said first level neuronal circuits, is such that the activity level of said output signal of said given one of said first level neuronal circuits has a respective positive quiescent value when the sum of said activity levels of said inputs of said given one of said first level neuronal circuits is constant with time.

43. A network according to claim 40,

wherein said means for providing, in each given one of said first and second level neuronal circuits, is further such that the time rate of change of the activity level of said output signal of said given one of said neuronal circuits is counterresponsive to the instantaneous activity level of said output signal of said given one of said neuronal circuits so as to dampen resonances due to said net non-negative feedback of activity levels around any loop which includes said given one of said neuronal circuits,

wherein said means for providing, in each given one of said first bevel neuronal circuits, is such that the activity level of said output signal of said given one of said first level neuronal circuits has a respective positive quiescent value when the sum of said activity levels of said inputs of said given one of said first level neuronal circuits is constant with time,

and wherein said means for providing, in each given one of said second level neuronal circuits, is such that the activity level of said output signal of said given one of said second level neuronal circuits has a respective positive quiescent value when the sum of the activity levels of said input signals of said given one of said second level neuronal circuits, minus a weighted sum of the activity levels of the output signals of said all other ones of said second level neuronal circuits, is constant with time.

44. A network according to claim 38, wherein said learning means comprises:

means for increasing the connectivity strength stored in the storage means of each given one of said feed-forward connection circuits by an amount commensurate with any concurrence of a high activity level on the input signal of said given feed-forward connection circuit and a high activity level on the output signal of the second level neuronal circuit to which the output of said given feed-forward connection circuit is coupled, and for decreasing said connectivity strength stored in said storage means of said given feed-forward connection circuit by an amount commensurate with any concurrence of a low activity level on said input signal of said given feed-forward connection circuit and a high activity level on the output signal of the second level neuronal circuit to which the output of said given feed-forward connection circuit is coupled; and

means for increasing the connectivity strength stored in the storage means of each given one of said feed-back connection circuits by an amount commensurate with any concurrence of a high activity level on the input signal of said given feed-back connection circuit and a high activity level on the output signal of the first level neuronal circuit to which the output of said given feed-back connection circuit is coupled, and for decreasing said connectivity strength stored in said storage means of said given feed-back connection circuit by an amount commensurate with any concurrence of a high activity level on said input signal of said feed-forward connection circuit and a low activity level on the output signal of the first level neuronal circuit to which the output of said given feed-back connection circuit is coupled.

45. A network according to claim 38, operable selectably in an awake mode or in a sleep mode, wherein one of the inputs to each of said first level neuronal circuits is coupled to receive a respective one of said environmental input signals,

wherein said means for providing, in each given one of said first level neuronal circuits, is further such that the activity level of said output signal of said given one of said first level neuronal circuits is positively responsive to a time integration of the sum of all of the activity levels of the input signals of said given first level neuronal circuit, and is further such that the time rate of change of the activity level of said output signal of said given one of said first level neuronal circuits is negatively responsive to th- instantaneous value of said activity level of said output signal of said given one of said first level neuronal circuits,

wherein said means for providing, in each given one of said second level neuronal circuits, is further such that the activity level of said output signal of said given one of said second level neuronal circuits is positively responsive to a time integration of the sum of all of the activity levels of said input signals of said second level neuronal circuits is further such that the activity level of said output signal of said given one of said second level neuronal circuits is negatively responsive to a time integration of a weighted sum of the activity levels of said output signals of all other ones of said second level neuronal circuits, and is further such that the time rate of change of the activity level of said output signal of said given one of said second level neuronal circuits is negatively responsive to the instantaneous value of the activity level of said output signal of said given one of said second level neuronal circuits,

and wherein said learning means comprises:

means for increasing the connectivity strength stored in the stprage means of each given one of said feed-forward connection circuits by an amount commensurate with any concurrence of a high activity level on the input signal of said given feed-forward connection circuit and a high activity level on the output signal of the second level neuronal circuit to which the output of said given feed-forward connection circuit is coupled, and for decreasing said connectivity strength stored in said storage means of said given feed-forward connection circuit by an amount commensurate with any concurrence of a low activity level on said input signal of said given feed-forward connection circuit and a high activity level on the output signal of the second level neuronal circuit to which the output of said given feed-forward connection circuit is coupled; and

means for increasing the connectivity strength stored in the storage means of each given one of said feed-back connection circuits by an amount commensurate with any concurrence of a high activity level on the input signal of said given feed-back connection circuit and a high activity level on the output signal of the first level neuronal circuit to which the output of said given feed-back connection circuit is coupled, and for decreasing said connectivity strength stored in said storage means of said given feed-back connection circuit by an amount commensurate with any concurrence of a high activity level on said input signal of said feed-forward connection circuit and a low activity level on the output signal of the first level neuronal circuit to which the output of said given feed-back connection circuit is coupled.

said network further comprising background signal source means for, at least when said network is in said sleep mode, applying a substantially constant background signal to an input of all of said second level neuronal circuits.

46. A network according to claim 45, further comprising means for decoupling all of said input signals from said first level neuronal circuits while said network is operated in said sleep mode.

47. A network according to claim 46, further comprising:

first means operative during sleep for detecting a favored state condition in which the signals on the outputs of certain ones of said second level neuronal circuits reach an activity level disparately more often relative to the signal on the outputs of the others of said second level neuronal circuits; and

second means operative during sleep for reversing the sense by which said learning means adjusts the connectivity strengths stored in the storage means of each of said feed-forward connection circuits which abut any of said certain second level neuronal circuits.

48. A network according to claim 47, wherein said second means operates to reverse the sense by which said learning means adjusts the connectivity strengths stored in the storage means of all of said feed-forward connection circuits when said first means detects said favored state condition.

49. A network according to claim 38, operable selectably in an awake mode or in a sleep mode, wherein one of the inputs to each of said first level neuronal circuits is coupled to receive a respective one of said environmental input signals,

wherein said means for providing in each given one of said first level neuronal circuits, is further such that the activity level of said output signal of said given one of said first level neuronal circuits is positively responsive to a time integration of the sum of all of the activity levels of the input signals of said given first level neuronal circuit, and is further such that the time rate of change of the activity level of said output signal of said given one of said first level neuronal circuits is negatively responsive to the instantaneous value of said activity level of said output signal of said given one of said first level neuronal circuits,

wherein said means for providing, in each given one of said second level neuronal circuits, is further such that the activity level of said output signal of said given one of said second level neuronal circuit is positively responsive to a time integration of the sum of all of the activity levels of said input signals of said second level neuronal circuits, is further such that the activity level of said output signal of said given one of said second level neuronal circuits is negatively responsive to a time integration of a weighted sum of the activity levels of said output signals of all other ones of said second level neuronal circuits, and is further such that the time rate of change of the activity level of said output signal of said given one of said second level neuronal circuits is negatively responsive to the instantaneous value of the activity level of said output signal of said given one of said second level neuronal circuits,

and wherein said learning means comprises:

means for increasing the connectivity strength stored in the storage means of each given one of said feed-forward connection circuits by an amount commensurate with any concurrence of a high activity level on the input signal of said given feed-forward connection circuit and a high activity level on the output signal of the second level neuronal circuit to which the output of said given feed-forward connection circuit is coupled, and for decreasing said connectivity strength stored in said storage means of said given feed-forward connection circuit by an amount commensurate with any concurrence of a low activity level on said input signal of said given feed-forward connection circuit and a high activity level on the output signal of the second level neuronal circuit to which the output of said given feed-forward connection circuit is coupled; and

means for increasing the connectivity strength stored in the storage means of each given one of said feed-back connection circuits by an amount commensurate with any concurrence of a high activity level on the input signal of said given feed-back connection circuit and a high activity level on the output signal of the first level neuronal circuit to which the output of said given feed-back connection circuit is coupled, and for decreasing said connectivity strength stored in said storage means of said given feed-back connection circuit by an amount commensurate with any concurrence of a high activity level on said input signal of said feed-forward connection circuit and a low activity level on the output signal of the first level neuronal circuit to which the output of said given feed-back connection circuit is coupled,

said network further comprising background signal source means for, at least when said network is in said sleep mode, applying a substantially constant background signal to an input of all of said first level neuronal circuits.

50. A neuron circuit suitable for use in a neural network, Comprising:

a plurality of current input lines;

an output line; and

means for providing on said output line an output signal having a frequency positively responsive to a time integration of the sum of currents of said current input lines and negatively responsive to a time integration of said output signal.

51. A circuit according to claim 50 wherein said means for providing comprises:

a summing junction, coupled to receive all of said current inputs;

a constant voltage node;

a summing capacitor coupled between said summing junction and said constant voltage node;

regulator means having an input and an output, for providing of said output of said regulator means a voltage signal positively responsive to the frequency of the signal on said input of said regulator means; and

oscillator means having first and second inputs and an output, said first input of said oscillator means being coupled to said summing junction, said second input of said oscillator means being coupled to receive said output of said regulator means, and said input of said regulator means being coupled to receive said output of said oscillator means, said oscillator means being for providing on said output of said oscillator means a signal having a frequency positively responsive to the difference between the voltages on said first and second inputs of said oscillator means.

52. A synaptic connection circuit suitable for use in a neural network, comprising:

a pre-synaptic signal input and a current output;

a storage capacitor; and

means for providing on said output of said synaptic connection circuit a current signal the time average of which is positively responsive to the voltage across said storage capacitor times the frequency of the signal on said pre-synaptic signal input.

53. A circuit according to claim 52, wherein said means for providing comprises means for providing on said output of said synaptic connection circuit a current pulse signal having a frequency equal to the frequency of the signal on said pre-synaptic signal input and having a magnitude positively responsive to the voltage across said storage capacitor.

54. A circuit according to claim 52, further comprising:

a post-synaptic signal input; and

means for adjusting in a predetermined direction the voltage across said storage capacitor in response to the concurrence of a high frequency on said pre-synaptic signal input and a high frequency on said post synaptic signal input.

55. A circuit according to claim 54, wherein said predetermined direction is the increasing direction.

56. A circuit according to claim 54, further comprising a deep sleep signal input, wherein said means for adjusting comprises means for selecting between the increasing direction and the decreasing direction as said predetermined direction in response to said deep-sleep signal input.

57. A circuit according to claim 54, wherein the signal on said pre- and post-synaptic inputs each consists of a series of pulses at a respective varying frequency, and wherein said means for adjusting comprising means for adjusting the voltage across said storage capacitor in said predetermined direction whenever a pulse on said pre-synaptic input occurs at the same time as a pulse on said post-synaptic input.

58. A circuit according to claim 52, further comprising:

a post-synaptic signal input;

deep-sleep means having an input coupled to receive the signal on said post synaptic signal input and further having an output, for providing on said output a pulse signal having a frequency positively responsive to a time integration of the frequency of the signal on said post-synaptic signal input; and

means for increasing the voltage across said storage capacitor in response to the concurrence of a pulse on said pre-synaptic input, a pulse on said post-synaptic input and an inter-pulse space on said output of said deep-sleep means, and for decreasing the voltage across said storage capacitor in response to the concurrence of a pulse on said pre-synaptic input, a pulse on said post-synaptic input and a pulse on said output of said deep-sleep means.

59. A synaptic connection circuit suitable for use in a neural network, comprising:

a pre-synaptic signal input and a current output;

a storage capacitor;

a transconductance converter having an input and an output, said output of said transconductance converter being said current output of said synaptic connection circuit; and

switch means for coupling the voltage on said storage capacitor to said input of said transconductance converter on each pulse of said pre-synaptic signal input.

60. A circuit according to claim 59, further comprising:

a voltage reference node;

learning switch means for coupling said storage capacitor to said voltage reference node on each pulse on one of said pre- and post-synaptic inputs; and

means for generating a pulse on said voltage reference node on each pulse on the other of said pre- and post-synaptic inputs.

61. A circuit according to claim 60, wherein said means for generating a pulse on said voltage reference node comprises a pulse extender having an input coupled to receive pulses from said other of said pre- and post-synaptic inputs and having an output coupled to said voltage reference node.

62. A circuit according to claim 60, further comprising a deep-sleep signal input, wherein said means for generating a pulse on said voltage reference node comprises an inverter coupled between said voltage reference node and said other of said pre- and post-synaptic inputs, said inverter being operable on each pulse on said deep-sleep signal input.