Spiking neuron network adaptive control apparatus and methods

• US 9,367,798 B2
• Filed: 09/20/2012
• Issued: 06/14/2016
• Est. Priority Date: 09/20/2012
• Status: Active Grant

• Current Assignee: Brain Corporation
• Original Assignee: Brain Corporation
• Inventors: Coenen, Olivier, Sinyavskiy, Oleg
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): Bharadwaj, Kalpana
• Application Number:
  US13/623,842
• Publication Number:
  US 20140081895A1
• Time in Patent Office: 1,363 Days
• US Class Current: 1/1
• CPC Class Codes: G05B 13/027 using neural networks only
• CPC Class Codes: G06N 3/049 Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

Adaptive controller apparatus of a plant may be implemented. The controller may comprise an encoder block and a control block. The encoder may utilize basis function kernel expansion technique to encode an arbitrary combination of inputs into spike output. The controller may comprise spiking neuron network operable according to reinforcement learning process. The network may receive the encoder output via a plurality of plastic connections. The process may be configured to adaptively modify connection weights in order to maximize process performance, associated with a target outcome. The relevant features of the input may be identified and used for enabling the controlled plant to achieve the target outcome.

Claims

1. An adaptive controller apparatus, the apparatus comprising:

one or more computer-readable storage media configured to store an encoder block, the encoder block comprising a continuous-to-spiking expansion kernel, the expansion kernel comprising a plurality of spiking neurons, individual ones of the plurality of spiking neurons including a plurality of receptive fields associated therewith, the encoder block being configured to encode a continuous input signal into a spiking output signal using the expansion kernel; and

a spiking neuron network configured to receive the spiking output from the plurality of spiking neurons via one or more connections, the spiking neuron network being further configured to generate a control signal using a reinforcement learning process and an external signal;

wherein;

individual ones of the plurality of receptive fields are characterized by an input range associated with the continuous input signal;

individual ones of the plurality of spiking neurons are configured to generate one or more spikes based on at least a portion of the continuous input signal corresponding to respective individual ones of the plurality of receptive fields;

the spiking output signal comprises the one or more spikes; and

the spiking neuron network is characterized by a first plasticity mechanism modulated by the external signal.

2. The apparatus of claim 1, wherein:

individual ones of the one or more connections are characterized by a connection efficacy; and

the spiking neuron network is characterized by a second plasticity mechanism configured to adjust the connection efficacy in accordance with the external signal.

3. The apparatus of claim 2, wherein:

the adjustment of the connection efficacy comprises one or both of a connection potentiation or a connection depression;

the potentiation comprises an increase of the connection efficacy; and

the depression comprises a reduction of the connection efficacy.

4. The apparatus of claim 3, wherein:

the increase of the connection efficacy is characterized by a first time-dependent function having a time window associated therewith;

the one or more spikes are characterized by a pre-synaptic time;

the external signal comprises a reinforcement signal, the reinforcement signal comprising one or more spikes characterized by a reinforcement spike time;

the time window is selected based on at least one of the pre-synaptic time and the reinforcement spike time; and

integration of the time-dependent function over the time window generates a positive value.

5. The apparatus of claim 2, wherein:

the reinforcement learning process is configured to produce a target outcome; and

the external signal comprises a reinforcement indication configured based on whether a current outcome associated with the control signal is within a pre-determined range from the target outcome.

6. The apparatus of claim 2, wherein:

the reinforcement learning process is characterized by a performance measure comprising a distance between a target outcome and current outcome, associated with a current state;

a current value of the performance measure is based on the adjustment of the connection efficacy configured to cause a transition of the process from a prior state to the current state;

the reinforcement is positive when the current outcome is closer to the target outcome as characterized by a smaller value of the distance; and

the reinforcement is negative when the current outcome is farther from the target outcome as characterized by a greater value of the distance.

7. The apparatus of claim 1, wherein the continuous input signal includes one or more of an analog signal, a polyadic signal with arity greater than 2, an n-bit long discrete signal with n-bits greater than 2, or a real-valued signal.

8. The apparatus of claim 1, wherein the spike output signal comprises a bit stream.

9. The apparatus of claim 1, wherein:

the spiking network comprises one or more spiking neurons;

individual ones of the plurality of spikes are configured to cause individual ones of the one or more spiking neurons to generate one or more spiking responses; and

the control signal comprises a combination of individual ones of the one or more spiking responses.

10. The apparatus of claim 1, wherein:

individual ones of the plurality of receptive fields are effectuated using individual ones of a plurality of band-pass filters;

the continuous input signal is characterized by an input signal extent comprising a range of values of the input signal; and

individual ones of the plurality of band-pass filters are characterized by a pass band in the input signal extent.

11. The apparatus of claim 10, wherein individual ones of the plurality of band-pass filters comprise a gain and a temporal component, the temporal component being configured to modify one or both of (i) the gain with time or (ii) the pass band with time.

12. The apparatus of claim 11, wherein the temporal component is obtained using a time derivative of the input signal.

13. The apparatus of claim 12, wherein:

the external signal comprises a reinforcement spike; and

the temporal component is obtained using a time integration of the input signal over a time window, the time window preceding a time of the reinforcement spike.

14. The apparatus of claim 1, wherein:

the input signal comprises two or more dimensions, individual ones of the two or more dimensions being configured to describe an input property;

individual ones of the plurality of receptive fields are characterized by a dimensional space configured to match the two or more dimensions; and

at least the portion of the continuous input signal comprises two or more properties, the two or more properties including the input property.

15. The apparatus of claim 1, wherein the plurality of receptive fields comprise an overlapping set of receptive fields.

16. The apparatus of claim 1, wherein the plurality of receptive fields comprise a non-overlapping set of receptive fields so that individual ones of the plurality of receptive fields cover non-overlapping portions of the input signal extent.

17. The apparatus of claim 1, wherein:

the continuous input signal is characterized by an input extent comprising a range of values of the continuous input signal; and

individual ones of the plurality of receptive fields are characterized by a unimodal concave function of at least a portion of the range.

18. The apparatus of claim 1, wherein:

the unimodal concave function comprises a combination of two or more of (i) a Gaussian distribution, (ii) a sigmoid distribution, or (iii) a linear distribution.

19. A non-transitory computer-readable storage medium having instructions embodied thereon, the instructions being executable by one or more processors to perform a method of implementing reinforcement learning, the method comprising:

receiving an non-spiking input signal characterized by an input range of values;

transforming the input signal into one or more spiking signals using one or more filters;

operating one or more stochastic spiking neurons in accordance with a stochastic reinforcement learning process; and

combining the one or more spiking signals into an output using the one or more stochastic spiking neurons and a reinforcement signal.

20. An adaptive controller apparatus, the apparatus comprising:

one or more processors configured to execute computer program modules, the computer program modules being executable to cause the one or more processors to;

encode a continuous input signal into one or more spiking signals using a continuous-to-spiking expansion kernel, the expansion kernel comprising of a plurality of basis components; and

adapt one or more stochastic spiking neurons in accordance with a stochastic reinforcement learning process, the adapting of the one or more stochastic spiking neurons being configured to combine the one or more spiking signals into a control signal, the adaptation configured to maximize the performance;

wherein the expansion kernel is effectuated by a plurality of spiking neurons, individual ones of the plurality of spiking neurons having a plurality of receptive fields associated therewith, individual ones of the plurality of receptive fields configured to effectuate individual ones of the plurality of the basis components.

21. The adaptive controller apparatus of claim 20, wherein:

(i) the learning process is characterized by;

a prior state and a current state, the learning process being described by at least a state variable configured to transition the learning process from the prior state to the current state;

a current performance measure associated with the current state and a prior performance measure, the prior performance measure being associated with the prior state; and

a target outcome having a target state associated therewith;

(ii) the adaptation of the one or more stochastic spiking neurons is based on a reinforcement signal configured external to the apparatus so that it provides;

positive reinforcement when a distance measure between the current state and the target state is smaller compared to the distance measure between the prior state and the target state; and

negative reinforcement when the distance measure between the current state and the target state is greater compared to the distance measure between the prior state and the target state; and

(iii) performance maximization comprises a minimization of the distance measure.