ARCHITECTURE, SYSTEM AND METHOD FOR ARTIFICIAL NEURAL NETWORK IMPLEMENTATION

• US 20120166374A1
• Filed: 12/28/2011
• Published: 06/28/2012
• Est. Priority Date: 12/08/2006
• Status: Active Grant

• Current Assignee: ZAMA Innovations LLC
• Original Assignee: Antony Savich, Medhat Moussa
• Inventors: Moussa, Medhat, Savich, Antony, Areibi, Shawki
• Granted Patent: US 8,468,109 B2
• Time in Patent Office: Days
• US Class Current: 706/16
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent

Abstract

Systems and methods for a scalable artificial neural network, wherein the architecture includes: an input layer; at least one hidden layer; an output layer; and a parallelization subsystem configured to provide a variable degree of parallelization to the artificial neural network by providing scalability to neurons and layers. In a particular case, the systems and methods may include a back-propagation subsystem that is configured to scalably adjust weights in the artificial neural network in accordance with the variable degree of parallelization. Systems and methods are also provided for selecting an appropriate degree of parallelization based on factors such as hardware resources and performance requirements.

Claims

1. A system comprising:

an input port for receiving an input vector;

a scalable artificial neural network, wherein the input vector is fed forward through the scalable artificial neural network to provide an output vector and wherein the input vector is subject to synchronization within the scalable artificial neural network based on a predetermined degree of parallelization; and

an output port for outputting the output vector.

2. A system according to claim 1, wherein the scalable artificial neural;

network comprises;

an input layer;

at least one hidden layer;

an output layer; and

a scalable neuron synchronization module configured to synchronize operations performed on the input vector during one or more clock cycles.

3. A system according to claim 2, wherein each layer comprises one or more artificial neural processing elements and the scalable neuron synchronization module comprises:

a temporal synchronization module configured to synchronize operation among neurons and between neurons within the layer over one or more clock cycles; and

a spatial synchronization module configured to synchronize parallel processing on an input vector among neurons within the layer.

4. A system according to claim 3, where the one or more artificial neural processing elements comprises:

an input port for receiving an input;

an output port;

a parallel processing module that is configured to provide a predetermined degree of parallel operation on the input received from the input port; and

a scalable synchronization module configured to synchronize operations performed on the input based on the predetermined degree of parallel operation.

5. A system according to claim 2, where each layer comprises:

an input port configured to receive the input vector;

an output port;

a plurality of artificial neural processing elements between the input port and the output port;

a temporal synchronization module block configured to synchronize operation among the artificial neural processing elements and between artificial neural processing elements over one or more clock cycles; and

a spatial synchronization module configured to synchronize parallel processing on the input vector among the artificial neural processing elements.

6. A method for designing a hardware configuration of an artificial neural network, the method comprising:

receiving information relating to hardware resources available for at least one hardware device;

receiving a desired network topology;

determining a plurality of degrees of parallelism for the desired network topology;

for each degree of parallelism of the plurality of degrees of parallelism estimating at least one of;

a hardware resource estimate to implement the network topology with the degree of parallelism; and

a performance estimate for the network topology with the degree of parallelism;

selecting a degree of parallelism based on the hardware resources available and at least one of the hardware resource estimates and the performance estimates; and

generating a hardware configuration based on the degree of parallelism.

7. A method according to claim 6, wherein estimating the hardware resource estimate comprises:

determining a number of weights based on the network topology;

determining a measure of the hardware resources required to provide the determined number of weights based on the degree of parallelism; and

assigning the determined measure of the hardware resources required as the hardware resource estimate.

8. A method according to claim 6, wherein the estimating a performance estimate comprises:

determining a number of weights based on the network topology;

determining a measure of the hardware processing speed available;

determining a number of updates that can be performed on the number of weights in a predetermined time based on the processing speed and the degree of parallelism; and

assigning the determined number of updates as the performance estimate.

9. A method according to claim 6, wherein the selecting a degree of parallelism based on the hardware resources available and at least one of the hardware resource estimates and the performance estimates comprises:

determining the maximum hardware resources available from among the hardware resources available;

determining a hardware resource estimate from among the hardware resource estimates that is closest to but less than or equal to the maximum hardware resources available; and

determining the degree of parallelism associated with the determined hardware estimate.

10. A method according to claim 9, wherein determining a hardware resource estimate from among the hardware resource estimates that is closest to but less than or equal to the maximum hardware resources available further comprises determining the hardware resource estimate from among the hardware resource estimates that maximizes performance.

11. A method according to claim 6, wherein the method further comprises receiving information related to an application performance requirement and wherein the selecting a degree of parallelism based on the hardware resources available and at least one of the hardware resource estimates and the performance estimates comprises:

determining a performance estimate from among the performance estimates that is equal to or greater than the application performance requirement; and

determining the degree of parallelism associated with the determined performance estimate.

12. A method according to claim 6, further comprising receiving an arithmetic representation and wherein the estimating at least one of a hardware resource estimate and a performance estimate is based on the received arithmetic representation.

13. A method according to claim 6, wherein the generating a hardware configuration based on the degree of parallelism comprises generating a hardware configuration comprising:

an input layer;

at least one hidden layer;

an output layer;

a back-propagation subsystem configured to send error data back through the network to adjust weights associated with the output layer and the at least one hidden layer; and

a parallelization system configured to provide the determined degree of parallelization to each of the input layer, at least one hidden layer, output layer and back-propagation system.

14. A method according to claim 13, further comprising configuring hardware device based on the hardware configuration.

15. A computer readable non-transitory storage medium, the storage medium comprising instructions to execute the method of claim 6.

16. A method for training a scalable artificial neural network involving multi-layer perceptrons and error back propagation, the method comprising:

feed-forwarding an input vector through the scalable network;

wherein the input vector is subject to synchronization to provide a synchronized output vector; and

back-propagating an error gradient vector through the scalable network, wherein the error gradient vector is calculated using the synchronized output vector and a target vector, which has been subject to synchronization, such that the error gradient vector is provided in a synchronized format based on the degree of parallelization.

17. A method for operating a scalable artificial neural network involving multi-layer perceptions, the method comprising feed-forwarding an input vector through the scalable network;

wherein the input vector is subject to synchronization within the scalable network to provide a synchronized output vector.

18. A computer readable non-transitory storage medium, the storage medium comprising instructions to execute the method of claim 17.