Reconfigurable and customizable general-purpose circuits for neural networks

• US 8,856,055 B2
• Filed: 04/08/2011
• Issued: 10/07/2014
• Est. Priority Date: 04/08/2011
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Brezzo, Bernard V., Chang, Leland, Esser, Steven K., Friedman, Daniel J., Liu, Yong, Modha, Dharmendra S., Montoye, Robert K., Rajendran, Bipin, Seo, Jae-sun, Tierno, Jose A.
• Primary Examiner(s): Wong, Lut
• Application Number:
  US13/083,414
• Publication Number:
  US 20120259804A1
• Time in Patent Office: 1,278 Days
• Field of Search: 706/33
• US Class Current: 706/33
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/049 Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: G06N 3/082 modifying the architecture, e.g. adding, deleting or silencing nodes or connections

Abstract

A reconfigurable neural network circuit is provided. The reconfigurable neural network circuit comprises an electronic synapse array including multiple synapses interconnecting a plurality of digital electronic neurons. Each neuron comprises an integrator that integrates input spikes and generates a signal when the integrated inputs exceed a threshold. The circuit further comprises a control module for reconfiguring the synapse array. The control module comprises a global final state machine that controls timing for operation of the circuit, and a priority encoder that allows spiking neurons to sequentially access the synapse array.

Claims

1. A reconfigurable neural network circuit, comprising:

an electronic synapse array comprising multiple digital synapses interconnecting a plurality of digital electronic neurons, wherein each neuron comprises an integrator that integrates input spikes and generates a spike signal when the integrated input spikes exceed a threshold;

a first learning module and a second learning module for reconfiguring a pre-synaptic neuron and a post-synaptic neuron, respectively, in the synapse array, wherein each learning module is independently reconfigurable; and

a control module for reconfiguring the synapse array, the control module comprising a global finite state machine that controls timing for operation of the circuit, and a priority encoder that allows spiking neurons among said neurons, to sequentially access the synapse array.

2. The network circuit of claim 1, wherein the circuit provides one of spike-timing dependent plasticity (STDP), anti-STDP, Hebbian and anti-Hebbian learning rules on the synapse array.

3. The network circuit of claim 2, wherein the synapse array comprises a random access memory synapse array, each neuron comprises a reconfigurable digital complementary metal-oxide-semiconductor (CMOS) device and the circuit provides binary stochastic spike timing dependent plasticity using the random access memory synapse array.

4. The network circuit of claim 1, further comprising:

driver modules that receive digital inputs from said neurons and program the synapse array based on the digital inputs and learning rules; and

sense amplifiers that measure the state of each synapse and convert it to binary data.

5. The network circuit of claim 4, wherein:

each neuron comprises a learning module including digital counters that decay at a pre-specified rate during each timestep, and are reset to a pre-defined value when a neuron spiking event occurs.

6. The network circuit of claim 5, wherein:

the priority encoder allows spiking neurons among said neurons to sequentially access the synapse array, providing communication of synaptic weights and programming of said synapses.

7. The network circuit of claim 6, wherein:

each synapse interconnects an axon of a pre-synaptic neuron with a dendrite of a post-synaptic neuron;

each learning module is reconfigurable independent of the other; and

neuron parameters including spiking, integration, learning and communication for each neuron are reconfigurable using reconfiguration input controls.

8. The network circuit of claim 7, wherein:

during a timestep, multiple neuron spikes are sequentially handled in a read phase and synapse updates are sequentially handled in a write phase, utilizing cycles generated by a digital clock, wherein a timestep is divided into multiple digital clock cycles.

9. The network circuit or claim 8, wherein:

the start of a subsequent timestep is triggered using handshaking signals when neuron and synapse operations of a previous timestep are completed.

10. The network circuit of claim 9, wherein:

input operations, neuron and synapse operations, and output operations are pipelined.

11. The network circuit of claim 7, wherein the learning module further comprises a linear feedback shift registers for probabilistically updating said synapses according to learning rules based on a decay rate of said one or more digital counters.

12. The network circuit of claim 11, wherein:

each synapse comprises a binary synapse including a transposable 1-bit static random access memory cell; and

the linear feedback shift register generates a new random number during every phase for a programming synapse.

13. The network circuit of claim 12, wherein:

the learning module further comprises a comparator that compares the random number with a spike counter, to generate a digital signal for programming a connected synapse by a probabilistic update according to a learning rule specified in a decay rate of the counter.

14. The network circuit of claim 7, wherein in a learning phase, the circuit learns correlations in spatio-temperal patterns and classification of said patterns in the synapse array, and in a recall phase the reconfigurable neural network circuit predicts and completes incomplete patterns.

15. The network circuit of claim 7, wherein the circuit is reconfigurable for learning rules including spike-timing dependent plasticity (STDP), anti-STDP, Hebbian and anti-Hebbian.

16. A reconfigurable neural network circuit, comprising:

an electronic synapse array comprising multiple digital synapses interconnecting a plurality of digital electronic neurons, wherein each neuron comprises an integrator that integrates input spikes and generates a spike signal when the integrated input spikes exceed a threshold; and

a control module for reconfiguring the synapse array, the control module comprising a global finite state machine that controls timing for operation of the circuit, and a priority encoder that allows spiking neurons among said neurons, to sequentially access the synapse array;

wherein each synapse comprises a multi-bit synapse including multiple transposable 1-bit static random access memory cells.

17. The network circuit of claim 16, wherein each neuron comprises a reconfigurable digital complementary metal-oxide-semiconductor (CMOS) circuit device and the circuit provides multi-bit deterministic spike timing dependent plasticity using the synapse array.

18. The network circuit of claim 16, further comprising:

driver modules that receive digital inputs from said neurons and program the synapse array based on the digital inputs and learning rules; and

sense amplifiers that measure the state of each synapse and convert it to data.

19. The network circuit of claim 18, wherein:

each neuron comprises a learning module including digital counters that decay at a pre-specified rate during each timestep, and are reset to a pre-defined value when a neuron spiking event occurs.

20. The network circuit of claim 19, wherein:

the priority encoder allows spiking neurons among said neurons to sequentially access the synapse array, providing communication of synaptic weights and programming the synapses.

21. The network circuit of claim 20, wherein:

each synapse interconnects an axon of a pre-synaptic neuron with a dendrite of a post-synaptic neuron, the circuit further comprising a first learning module for an axonal, pre-synaptic neuron, and a second learning module for a dendritic, post-synaptic neuron, such that each learning module is reconfigurable independent of the other; and

neuron parameters including spiking, integration, learning and communication for each neuron are reconfigurable using reconfiguration input controls.

22. The network circuit of claim 21, wherein:

during a timestep, multiple neuron spikes are sequentially handled in a read phase and synapse updates are sequentially handled in a write phase, utilizing cycles generated by a digital clock, wherein a timestep is divided into multiple digital clock cycles.

23. The network circuit of claim 19, wherein each synapse comprises m transposable static random access memory cells that store a multi-bit value, representing a fine grain connection between every neuron connection.

24. The network circuit of claim 23, wherein:

in an update phase, the neuron reads an existing multi-bit synapse value from the synapse array, adds or subtracts the decay counter value to the value read from the synapse to generate a new multi-bit value, and updates the synapse with the new multi-bit value, such that the synapse update operation is performed in the update phase when a neuron spikes.

25. The network circuit of claim 24, wherein multiple read and write operations are performed in an update phase within a timestep, and the read and write operation are interleaved such that in every hardware cycle the circuit performs either a synapse read or write.

26. The network circuit of claim 21, wherein in a learning phase, the circuit learns correlations in spatio-temperal patterns and classification of said patterns in the synapse array, and in a recall phase the reconfigurable neural network circuit predicts and completes incomplete patterns.

27. The network circuit of claim 21, wherein the circuit is reconfigurable for learning rules including spike-timing dependent plasticity (STDP), anti-STDP, Hebbian and anti-Hebbian.