Neural network and method of neural network training

• US 9,390,373 B2
• Filed: 09/23/2015
• Issued: 07/12/2016
• Est. Priority Date: 03/06/2014
• Status: Active Grant

• Current Assignee: Progress, Inc.
• Original Assignee: Progress, Inc.
• Inventors: Pescianschi, Dmitri
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Zidanic, Michael
• Application Number:
  US14/862,337
• Publication Number:
  US 20160012330A1
• Time in Patent Office: 293 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent

Abstract

A neural network includes a plurality of inputs for receiving input signals, and synapses connected to the inputs and having corrective weights. The network additionally includes distributors. Each distributor is connected to one of the inputs for receiving the respective input signal and selects one or more corrective weights in correlation with the input value. The network also includes neurons. Each neuron has an output connected with at least one of the inputs via one synapse and generates a neuron sum by summing corrective weights selected from each synapse connected to the respective neuron. Furthermore, the network includes a weight correction calculator that receives a desired output signal, determines a deviation of the neuron sum from the desired output signal value, and modifies respective corrective weights using the determined deviation. Adding up the modified corrective weights to determine the neuron sum minimizes the subject deviation for training the neural network.

Claims

1. A neural network comprising:

a plurality of inputs of the neural network, each input configured to receive an input signal having an input value;

a plurality of synapses, wherein each synapse is connected to one of the plurality of inputs and includes a plurality of corrective weights, wherein each corrective weight is defined by a weight value;

a set of distributors, wherein each distributor is operatively connected to one of the plurality of inputs for receiving the respective input signal and is configured to select one or more corrective weights from the plurality of corrective weights in correlation with the input value;

a set of neurons, wherein each neuron has at least one output and is connected with at least one of the plurality of inputs via one of the plurality of synapses, and wherein each neuron is configured to add up the weight values of the corrective weights selected from each synapse connected to the respective neuron and thereby generate a neuron sum; and

a weight correction calculator configured to receive a desired output signal having a value, determine a deviation of the neuron sum from the desired output signal value, and modify respective corrective weight values using the determined deviation, such that adding up the modified corrective weight values to determine the neuron sum minimizes the deviation of the neuron sum from the desired output signal value to thereby train the neural network.

2. The neural network of claim 1, wherein:

the determination of the deviation of the neuron sum from the desired output signal includes division of the desired output signal value by the neuron sum to thereby generate a deviation coefficient; and

the modification of the respective corrective weight values includes multiplication of each corrective weight used to generate the neuron sum by the deviation coefficient.

3. The neural network of claim 1, wherein the deviation of the neuron sum from the desired output signal is a mathematical difference therebetween, and wherein the generation of the respective modified corrective weights includes apportionment of the mathematical difference to each corrective weight used to generate the neuron sum.

4. The neural network of claim 3, wherein the apportionment of the mathematical difference includes dividing the determined difference equally between each corrective weight used to generate the neuron sum.

5. The neural network of claim 3, wherein:

each distributor is additionally configured to assign a plurality of coefficients of impact to the respective plurality of corrective weights, such that each coefficient of impact is assigned to one of the plurality of corrective weights in a predetermined proportion to generate the respective neuron sum;

each neuron is configured to add up a product of the corrective weight and the assigned coefficient of impact for all the synapses connected thereto; and

the weight correction calculator is configured to apply a portion of the determined difference to each corrective weight used to generate the neuron sum according to the proportion established by the respective coefficient of impact.

6. The neural network of claim 5, wherein:

each respective plurality of coefficients of impact is defined by an impact distribution function;

the plurality of input values is received into a value range divided into intervals according to an interval distribution function, such that each input value is received within a respective interval, and each corrective weight corresponds to one of the intervals; and

each distributor uses the respective received input value to select the respective interval, and to assign the respective plurality of coefficients of impact to the corrective weight corresponding to the selected respective interval and to at least one corrective weight corresponding to an interval adjacent to the selected respective interval.

7. The neural network of claim 6, wherein each corrective weight is additionally defined by a set of indexes including:

an input index configured to identify the corrective weight corresponding to the input;

an interval index configured to specify the selected interval for the respective corrective weight; and

a neuron index configured to specify the corrective weight corresponding to the neuron.

8. The neural network of claim 7, wherein each corrective weight is further defined by an access index configured to tally a number of times the respective corrective weight is accessed by the input signal during training of the neural network.

9. A method of training a neural network, comprising:

receiving, via an input to the neural network, an input signal having an input value;

communicating the input signal to a distributor operatively connected to the input;

selecting, via the distributor, in correlation with the input value, one or more corrective weights from a plurality of corrective weights, wherein each corrective weight is defined by a weight value and is positioned on a synapse connected to the input;

adding up the weight values of the selected corrective weights, via a neuron connected with the input via the synapse and having at least one output, to generate a neuron sum;

receiving, via a weight correction calculator, a desired output signal having a value;

determining, via the weight correction calculator, a deviation of the neuron sum from the desired output signal value; and

modifying, via the weight correction calculator, respective corrective weight values using the determined deviation, such that adding up the modified corrective weight values to determine the neuron sum minimizes the deviation of the neuron sum from the desired output signal value to thereby train the neural network.

10. The method of claim 9, wherein:

said determining the deviation of the neuron sum from the desired output signal value includes dividing the desired output signal value by the neuron sum to thereby generate a deviation coefficient; and

said modifying the respective corrective weights includes multiplying each corrective weight used to generate the neuron sum by the deviation coefficient.

11. The method of claim 9, wherein said determining the deviation of the neuron sum from the desired output signal value includes determining a mathematical difference therebetween, and wherein said modifying of the respective corrective weights includes apportioning the mathematical difference to each corrective weight used to generate the neuron sum.

12. The method of claim 11, wherein said apportioning of the mathematical difference includes dividing the determined difference equally between each corrective weight used to generate the neuron sum.

13. The method of claim 9, further comprising:

assigning, via the distributor, a plurality of coefficients of impact to the plurality of corrective weights, and includes assigning each coefficient of impact to one of the plurality of corrective weights in a predetermined proportion to generate the neuron sum;

adding up, via the neuron, a product of the corrective weight and the assigned coefficient of impact for all the synapses connected thereto; and

applying, via the weight correction calculator, a portion of the determined difference to each corrective weight used to generate the neuron sum according to the proportion established by the respective coefficient of impact.

14. The method of claim 13, wherein the plurality of coefficients of impact is defined by an impact distribution function;

the method further comprising;

receiving the input value into a value range divided into intervals according to an interval distribution function, such that the input value is received within a respective interval, and each corrective weight corresponds to one of the intervals; and

using, via the distributor, the received input value to select the respective interval, and to assign the plurality of coefficients of impact to the corrective weight corresponding to the selected respective interval and to at least one corrective weight corresponding to an interval adjacent to the selected respective interval.

15. The method of claim 14, further comprising additionally defining each corrective weight by a set of indexes, wherein the set of indexes includes:

an input index configured to identify the corrective weight corresponding to the input;

an interval index configured to specify the selected interval for the respective corrective weight; and

a neuron index configured to specify the corrective weight corresponding to the neuron.

16. The method of claim 15, further comprising additionally defining each corrective weight by an access index configured to tally a number of times the respective corrective weight is accessed by the input signal during training of the neural network.