Neural network and method of neural network training

• US 10,423,694 B2
• Filed: 03/03/2017
• Issued: 09/24/2019
• Est. Priority Date: 03/06/2014
• Status: Active Grant

• Current Assignee: Progress, Inc.
• Original Assignee: Progress, Inc.
• Inventors: Pescianschi, Dmitri
• Primary Examiner(s): Cassity, Robert A
• Assistant Examiner(s): Zidanic, Michael
• Application Number:
  US15/449,614
• Publication Number:
  US 20170177998A1
• Time in Patent Office: Days
• Field of Search: None
• CPC Class Codes: G06F 17/16 Matrix or vector computation , e.g. matrix-matrix or matrix-vector multiplication, matrix factorization matrix transposition G06F7/78
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent

Abstract

A neural network includes inputs for receiving input signals, and synapses connected to the inputs and having corrective weights organized in an array. Training images are either received by the inputs as an array or codified as such during training of the network. The network also includes neurons, each having an output connected with at least one input via one synapse and generating a neuron sum array by summing corrective weights selected from each synapse connected to the respective neuron. Furthermore, the network includes a controller that receives desired images in an array, determines a deviation of the neuron sum array from the desired output value array, and generates a deviation array. The controller modifies the corrective weight array using the deviation array. Adding up the modified corrective weights to determine the neuron sum array reduces the subject deviation and generates a trained corrective weight array for concurrent network training.

Claims

1. A neural network comprising:

a plurality of inputs to the neural network configured to receive training images, wherein the training images are one of received as a training input value array and codified as the training input value array during training of the neural network;

a plurality of synapses, wherein each synapse is connected to one of the plurality of inputs and includes a plurality of corrective weights, wherein each corrective weight is defined by a weight value, and wherein the corrective weights of the plurality of synapses are organized in a corrective weight array;

a plurality of neurons, wherein each neuron has at least one output and is connected with at least one of the plurality of inputs via at least one of the plurality of synapses, and wherein each neuron is configured to add up the weight values of the corrective weights corresponding to each synapse connected to the respective neuron, such that the plurality of neurons generate a neuron sum array; and

a controller configured to;

receive desired images organized as a desired output value array;

determine a deviation of the neuron sum array from the desired output value array and generate a deviation array; and

modify the corrective weight array using the determined deviation array, such that adding up the modified corrective weight values to determine the neuron sum array reduces the deviation of the neuron sum array from the desired output value array to generate a trained corrective weight array and thereby facilitate concurrent training of the neural network.

2. The neural network of claim 1, wherein in a trained neural network:

the plurality of inputs to the neural network are configured to receive input images, wherein the input images are one of received as an input value array and codified as the input value array during recognition of the images by the neural network;

each synapse includes a plurality of trained corrective weights of the trained corrective weight array; and

each neuron is configured to add up the weight values of the trained corrective weights corresponding to each synapse connected to the respective neuron, such that the plurality of neurons generate a recognized images array, thereby providing recognition of the input images.

3. The neural network of claim 2, further comprising a set of distributors, wherein the set of distributors is configured to codify each of the training images and input images as the respective training input value array and input value array, and wherein the set of distributors is operatively connected to the plurality of inputs for receiving the respective training images and input images.

4. The neural network of claim 1, wherein the controller is additionally programmed with an array of target deviation of the neuron sum array from the desired output value array, and wherein the controller is additionally configured to complete training of the neural network when the deviation of the neuron sum array from the desired output value array is within an acceptable range of the array of target deviation.

5. The neural network of claim 2, wherein the training input value array, input value array, corrective weight array, neuron sum array, desired output value array, deviation array, trained corrective weight array, recognized image array, and target deviation array is organized, respectively, as a training input value matrix, input value matrix, corrective weight matrix, neuron sum matrix, desired output value matrix, deviation matrix, trained corrective weight matrix, recognized image matrix, and target deviation matrix.

6. The neural network of claim 5, further comprising a plurality of data processors, wherein the controller is additionally configured to partition at least one of the respective input value, training input value, corrective weight, neuron sum, and desired output value matrices into respective sub-matrices and communicate a plurality of the resultant sub-matrices to the plurality of data processors for separate parallel mathematical operations therewith to thereby facilitate concurrent data processing and an increase in speed of one of image recognition of the input value matrix and training of the neural network.

7. The neural network of claim 5, wherein the controller modifies the corrective weight matrix by applying an algebraic matrix operation to the training input value matrix and the corrective weight matrix to thereby train the neural network.

8. The neural network of claim 7, wherein the mathematical matrix operation includes a determination of a mathematical product of the training input value and corrective weight matrices to thereby form a current training epoch weight matrix.

9. The neural network of claim 8, wherein the controller is additionally configured to:

subtract the neuron sum matrix from the desired output value matrix to generate a matrix of deviation of neuron sums; and

divide the matrix of deviation of neuron sums by the number of inputs connected to the respective neuron to generate a matrix of deviation per neuron input.

10. The neural network of claim 9, wherein the controller is additionally configured to:

determine a number of times each corrective weight was used during one training epoch of the neural network;

form an averaged deviation matrix for the one training epoch using the determined number of times each corrective weight was used during the one training epoch; and

add the averaged deviation matrix for the one training epoch to the corrective weight matrix to thereby generate the trained corrective weight matrix and complete the one training epoch.

11. A method of operating a neural network, comprising:

receiving training images via a plurality of inputs to the neural network, wherein the training images are one of received as a training input value array and codified as the training input value array during training of the neural network;

organizing corrective weights of a plurality of synapses in a corrective weight array, wherein each synapse is connected to one of the plurality of inputs and includes a plurality of corrective weights, and wherein each corrective weight is defined by a weight value;

generating a neuron sum array via a plurality of neurons, wherein each neuron has at least one output and is connected with at least one of the plurality of inputs via one of the plurality of synapses, and wherein each neuron is configured to add up the weight values of the corrective weights corresponding to each synapse connected to the respective neuron;

receiving, via a controller, desired images organized as a desired output value array;

determining, via the controller, a deviation of the neuron sum array from the desired output value array and generate a deviation array; and

modifying, via the controller, the corrective weight array using the determined deviation array, such that adding up the modified corrective weight values to determine the neuron sum array reduces the deviation of the neuron sum array from the desired output value array to generate a trained corrective weight array and thereby facilitate concurrent training of the neural network.

12. The method of claim 11, wherein in a trained neural network:

receiving input images via the plurality of inputs to the neural network, wherein the input images are one of received as an input value array and codified as the input value array during recognition of the images by the neural network;

attributing to each synapse a plurality of trained corrective weights of the trained corrective weight array, wherein each trained corrective weight is defined by a weight value; and

adding up the weight values of the trained corrective weights corresponding to each synapse connected to the respective neuron, such that the plurality of neurons generate a recognized images array, thereby providing recognition of the input images.

13. The method of claim 12, further comprising codifying, via a set of distributors, each of the training images and input images as the respective training input value array and input value array, wherein the set of distributors is operatively connected to the plurality of inputs for receiving the respective training images and input images.

14. The method of claim 11, wherein the controller is additionally programmed with an array of target deviation of the neuron sum array from the desired output value array, the method further comprising, completing, via the controller, training of the neural network when the deviation of the neuron sum array from the desired output value array is within an acceptable range of the array of target deviation.

15. The method of claim 12, further comprising organizing the training input value array, input value array, corrective weight array, neuron sum array, desired output value array, deviation array, trained corrective weight array, recognized image array, and target deviation array, respectively, as a training input value matrix, input value matrix, corrective weight matrix, neuron sum matrix, desired output value matrix, deviation matrix, trained corrective weight matrix, recognized image matrix, and target deviation matrix.

16. The method of claim 15, wherein the neural network additionally includes a plurality of data processors, the method further comprising partitioning, via the controller, at least one of the respective input value, training input value, corrective weight, neuron sum, and desired output value matrices into respective sub-matrices and communicating a plurality of the resultant sub-matrices to the plurality of data processors for separate parallel mathematical operations therewith to thereby facilitate concurrent data processing and an increase in speed of one of image recognition of the input value matrix and training of the neural network.

17. The method of claim 15, further comprising modifying, via the controller, the corrective weight matrix by applying an algebraic matrix operation to the training input value matrix and the corrective weight matrix to thereby train the neural network.

18. The method of claim 17, wherein applying the mathematical matrix operation includes determining a mathematical product of the training input value and corrective weight matrices to thereby form a current training epoch weight matrix.

19. The method of claim 18, further comprising:

subtracting, via the controller, the neuron sum matrix from the desired output value matrix to generate a matrix of deviation of neuron sums; and

dividing, via the controller, the matrix of deviation of neuron sums by the number of inputs connected to the respective neuron to generate a matrix of deviation per neuron input.

20. The method of claim 19, further comprising:

determining, via the controller, a number of times each corrective weight was used during one training epoch of the neural network;

forming, via the controller, an averaged deviation matrix for the one training epoch using the determined number of times each corrective weight was used during the one training epoch; and

adding, via the controller, the averaged deviation matrix for the one training epoch to the corrective weight matrix to thereby generate the trained corrective weight matrix and complete the one training epoch.

21. A non-transitory computer-readable storage device for operating an artificial neural network, the storage device encoded with instructions executable to:

receive training images via a plurality of inputs to the neural network, wherein the training images are one of received as a training input value array and codified as the training input value array during training of the neural network;

organize corrective weights of a plurality of synapses in a corrective weight array, wherein each synapse is connected to one of the plurality of inputs and includes a plurality of corrective weights, and wherein each corrective weight is defined by a weight value;

generate a neuron sum array via a plurality of neurons, wherein each neuron has at least one output and is connected with at least one of the plurality of inputs via one of the plurality of synapses, and wherein each neuron is configured to add up the weight values of the corrective weights corresponding to each synapse connected to the respective neuron;

receive desired images organized as a desired output value array;

determine a deviation of the neuron sum array from the desired output value array and generate a deviation array; and

modify the corrective weight array using the determined deviation array, such that adding up the modified corrective weight values to determine the neuron sum array reduces the deviation of the neuron sum array from the desired output value array to generate a trained corrective weight array and thereby facilitate concurrent training of the neural network.

22. The storage device of claim 21, further encoded with instructions executable to:

receive input images via the plurality of inputs to the neural network, wherein the input images are one of received as an input value array and codified as the input value array during recognition of the images by the neural network;

attribute to each synapse a plurality of trained corrective weights of the trained corrective weight array, wherein each trained corrective weight is defined by a weight value; and

add up the weight values of the trained corrective weights corresponding to each synapse connected to the respective neuron, such that the plurality of neurons generate a recognized images array, thereby providing recognition of the input images.

23. An apparatus for operating an artificial neural network, comprising:

a means for receiving training images via a plurality of inputs to the neural network, wherein the training images are one of received as a training input value array and codified as the training input value array during training of the neural network;

a means for organizing corrective weights of a plurality of synapses in a corrective weight array, wherein each synapse is connected to one of the plurality of inputs and includes a plurality of corrective weights, and wherein each corrective weight is defined by a weight value;

a means for generating a neuron sum array via a plurality of neurons, wherein each neuron has at least one output and is connected with at least one of the plurality of inputs via one of the plurality of synapses, and wherein each neuron is configured to add up the weight values of the corrective weights corresponding to each synapse connected to the respective neuron;

a means for receiving desired images organized as a desired output value array;

a means for determining a deviation of the neuron sum array from the desired output value array and generate a deviation array; and

a means for modifying the corrective weight array using the determined deviation array, such that adding up the modified corrective weight values to determine the neuron sum array reduces the deviation of the neuron sum array from the desired output value array to generate a trained corrective weight array and thereby facilitate concurrent training of the neural network.

24. The apparatus of claim 23, wherein in a trained neural network:

a means for receiving input images via the plurality of inputs to the neural network, wherein the input images are one of received as an input value array and codified as the input value array during recognition of the images by the neural network;

a means for attributing to each synapse a plurality of trained corrective weights of the trained corrective weight array, wherein each trained corrective weight is defined by a weight value; and

a means for adding up the weight values of the trained corrective weights corresponding to each synapse connected to the respective neuron, such that the plurality of neurons generate a recognized images array, thereby providing recognition of the input images.