Efficient and scalable systems for calculating neural network connectivity in an event-driven way

• US 10,339,439 B2
• Filed: 10/01/2015
• Issued: 07/02/2019
• Est. Priority Date: 10/01/2014
• Status: Active Grant

• Current Assignee: Thalchemy Corporation
• Original Assignee: Thalchemy Corporation
• Inventors: Lipasti, Mikko H., Nere, Andrew, Hashmi, Atif, Wakerly, John F.
• Primary Examiner(s): Nilsson, Eric
• Application Number:
  US14/873,138
• Publication Number:
  US 20160098629A1
• Time in Patent Office: 1,370 Days
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

Systems and methods achieving scalable and efficient connectivity in neural algorithms by re-calculating network connectivity in an event-driven way are disclosed. The disclosed solution eliminates the storing of a massive amount of data relating to connectivity used in traditional methods. In one embodiment, a deterministic LFSR is used to quickly, efficiently, and cheaply re-calculate these connections on the fly. An alternative embodiment caches some or all of the LFSR seed values in memory to avoid sequencing the LFSR through all states needed to compute targets for a particular active neuron. Additionally, connections may be calculated in a way that generates neural networks with connections that are uniformly or normally (Gaussian) distributed.

Claims

1. A scalable system for recalculating, in an event-driven manner, property parameters including connectivity parameters of a neural network, the system comprising:

an input component that receives a time varying input signal;

a state machine comprising a pseudo-random number generator with a seed value, wherein the state machine is capable of recalculating property parameters of the neural network using the seed value, wherein the property parameters include connectivity among neurons of the neural network, and wherein the connectivity is deterministically generated by the pseudo-random number generator without reference to preexisting connectivity stored in memory, wherein the state machine is capable of generating a unique identifying number for each neuron in the neural network, and, wherein the state machine comprises a neuron identification counter with a first predefined initial value and a neuron connectivity counter with a second predefined initial value, wherein both the first predefined initial value and the second predefined initial value are utilized to update the state machine, and, wherein the pseudo-random number generator of the state machine comprises a Linear Feedback Shift Register (LSFR);

a storage component containing a plurality of predefined seed values from which a particular seed value is chosen for the pseudo-random number generator, wherein the number of seed values is less than the number of property parameters to be recalculated; and

an output component that generates output signals reflective of the calculated property parameters of the neural network and the input signal.

2. The system of claim 1, wherein the LFSR is configured to generate certain property parameters including connectivity.

3. The system of claim 1, wherein upon initialization, the state machine is configured to perform the following:

comparing the value of the neuron connectivity counter (Conn_Counter) to a predefined final value;

if Conn_Counter value is not equal to the predefined final value, causing the state machine to update, changing the Conn_Counter to a next value, and updating property parameters of the neuron identified by the neuron identification counter (Neuron_Index) in response to the input signal;

if Conn Counter value is equal to the predefined final value, changing Neuron_Index to a next value;

comparing Neuron_Index to a predefined total number of neurons in the neural network; and

if Neuron_Index value is not equal to the predefined total number of neurons in the neural network, resetting the Conn_Counter to the second predefined initial value and repeating the above steps for next neuron as identified by the Neuron_Index.

4. The system of claim 1, wherein the input component converts the received time varying input signal into a sequence of spikes.

5. The system of claim 1, wherein the state machine recalculates connectivity of a neuron currently being evaluated using a seed value corresponding to the neuron currently being evaluated only when the neuron currently being evaluated fires in response to the input signal.

6. The system of claim 5, wherein the state machine is configured to further perform the following:

maintaining a list of future firing neurons, wherein the recalculating at each time step is conducted only on neurons identified on the list;

for each target neuron of a neuron that fires at a current time step, comparing current membrane potential of that target neuron to a corresponding predefined firing threshold of that target neuron;

adding identity of a target neuron to the list of future firing neurons if the current membrane potential of that target neuron exceeds the corresponding predefined firing threshold; and

removing an identity of a target neuron from the list of future firing neurons if current membrane potential of that target neuron is below the corresponding predefined firing threshold.

7. The system of claim 1, wherein the storage component is configured to store seed values corresponding to the neurons in the neural network for the state machine and the state machine is configured to perform the following:

determining whether a neuron identified by the neuron identification counter (Neuron_Index) fires in response to the input signal;

if the neuron identified by the Neuron_Index fires, retrieving from the storage component a seed corresponding to the neuron identified by the Neuron_Index, initializing the neuron connectivity counter (Conn_Counter) to the second predefined initial value;

comparing Conn_Counter value to a predefined final value;

if Conn_Counter value is not equal to the predefined final value, causing the state machine to update to a next state, changing Conn_Counter to a next value, and repeating this step;

if Conn_Counter value is equal to the predefined final value, changing Neuron_Index to a next value;

comparing Neuron_Index to a predefined maximum number of neurons in the neural network; and

if Neuron_Index value is not equal to the predefined maximum number of neurons in the neural network, repeating the above steps for a next neuron as identified by Neuron_Index.

8. The system of claim 1, wherein the storage component includes a cache for storing seed values used by the state machine for recalculating certain property parameters.

9. The system of claim 8, wherein the state machine is configured to further perform the following:

calculating a seed value necessary for recalculating certain connectivity parameters upon determining that the seed value necessary for recalculating certain connectivity parameters is not stored in the cache; and

updating the cache to include the calculated seed value according to a predetermined cache rule.

10. The system of claim 1, wherein the state machine comprises:

a state skip unit with a number (N) of feedback networks, wherein each feedback network generates, in a number (M) of clock cycles, a state of the state machine after sequentially advancing a programmable number (P) of states from the state machine'"'"'s current state, and M<

P; and

a multiplexing circuit for updating the state machine by selecting one of the N feedback networks.

11. The system of claim 1, wherein the identifying number of a target neuron in the network is the sum of a predefined offset value and multiple random numbers generated by the state machine so as to center the normalized distribution of the target neurons.

12. The system of claim 1, wherein the state machine is also used to generate a connection type for each neuron of the network.

13. The system of claim 1, wherein the property parameters further include neural delays of each neuron in the network.

14. The system of claim 1, further comprising:

a plurality of processing elements, wherein each processing element has a state machine and is capable of calculating property parameters of a subset of neurons of the neural network.

15. The system of claim 1, wherein the state machine is configured to cache seed values of the state machine corresponding to N neurons last fired.

16. A computer-implemented method for recalculating network property parameters of a neural network including connectivity parameters in an event-driven manner, the method comprising:

initializing property parameters of the neural network;

receiving, at an evaluating neuron of the neural network, a neural input corresponding to a time varying input signal to the neural network;

recalculating, by a state machine comprising a pseudo-random number generator with a seed value, at least some of the property parameters of the evaluating neuron, wherein the connectivity parameters are deterministically generated after initialization by the pseudo-random number generator using the seed value without reference to preexisting connectivity stored in memory, wherein the state machine is capable of generating a unique identifying number for each neuron in the neural network, and, wherein the state machine comprises a neuron identification counter with a first predefined initial value and a neuron connectivity counter with a second predefined initial value, wherein both the first predefined initial value and the second predefined initial value are utilized to update the state machine, and wherein the pseudo-random number generator comprises a Linear Feedback Shift Register (LFSR);

determining whether the evaluating neuron is to generate a neural output to its target neurons in the neural network; and

if the evaluating neuron is determined to generate a neural output to its target neurons in the neural network, propagating the output of the evaluating neuron to its target neurons,

wherein a particular seed value is chosen from a plurality of predefined seed values for the pseudo-random number generator, the plurality of predefined seed values being contained in a storage component, and wherein the number of seed values is less than the number of property parameters to be recalculated.

17. The method of claim 16, wherein the property parameters of the neural network include on or more parameters of the group consisting of the maximum number of neurons in the neural network, seed values for pseudo-random number generation, neural axonal and dendritic delay values, positive connectivity strength values, negative connectivity strength values, neuron refractory period, decay rate of neural membrane potential, neuron membrane potential, and neuron membrane leakage parameter.

18. The method of claim 16, wherein the determining comprises:

calculating current membrane potential of the evaluating neuron based on the neural input and the connectivity parameters of the evaluating neuron;

comparing the calculated membrane potential to a firing threshold value of the evaluating neuron; and

reporting that the evaluating neuron is to generate an output if the calculated membrane potential exceeds the firing threshold value.

19. The method of claim 16, wherein the recalculating comprises:

retrieving a stored seed value corresponding to the evaluating neuron; and

calculating connectivity parameters of the evaluating neuron using the retrieved seed value.

20. The method of claim 16, wherein the recalculating comprises calculating connectivity of a neuron currently being evaluated only when the neuron currently being evaluated fires in response to the input signal.

21. The method of claim 16, further comprising retrieving a seed value of the state machine for recalculating the connectivity parameters of the evaluating neuron from a cache coupled to the state machine.

22. The method of claim 21, wherein the retrieving further comprises:

calculating the seed value of the state machine for recalculating the connectivity parameters of the evaluating neuron upon determining that the cache does not store the seed value; and

updating the cache to include the calculated seed value according to a predetermined cache rule.

23. The method of claim 16, further comprising:

maintaining a list of future firing neurons, wherein the updating at each time step is conducted only on neurons identified on the list;

for each target neuron of a neuron that fires at a current time step, comparing the current membrane potential of that target neuron to a corresponding predefined firing threshold of that target neuron;

adding an identity of a target neuron to the list of future firing neurons if the current membrane potential of that target neuron exceeds the corresponding predefined firing threshold; and

removing an identity of a target neuron from the list of future firing neurons if the current membrane potential of that target neuron is below the corresponding predefined firing threshold.

24. The method of claim 16, wherein the recalculating further comprises:

sequentially advancing a programmable number (P) of states beyond the current state of the state machine in a number (M) of clock cycles, wherein M<

P.

25. The method of claim 16, further comprising:

taking sum of the state machine results to form a uniform distribution; and

adding an offset to center the normalized distribution in calculating addresses of the target neurons.

26. The method of claim 16, wherein the recalculating further comprises using the state machine to generate a distinct connection type for each neuron of the network.

27. The method of claim 16, wherein the property parameters include neural delays of the neurons.

28. The method of claim 16, wherein the recalculating is carried out by a plurality of processing elements, wherein each processing element has a state machine and is capable of calculating property parameters of a subset of neurons of the neural network.