Architecture, system and method for artificial neural network implementation

• US 8,468,109 B2
• Filed: 12/28/2011
• Issued: 06/18/2013
• Est. Priority Date: 12/08/2006
• Status: Active Grant

• Current Assignee: ZAMA Innovations LLC
• Original Assignee: Antony Savich, Medhat Moussa
• Inventors: Moussa, Medhat, Savich, Antony, Areibi, Shawki
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): Hill, Stanley K
• Application Number:
  US13/338,869
• Publication Number:
  US 20120166374A1
• Time in Patent Office: 538 Days
• Field of Search: 706 26- 43
• US Class Current: 706/31
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent

Abstract

Systems and methods for a scalable artificial neural network, wherein the architecture includes: an input layer; at least one hidden layer; an output layer; and a parallelization subsystem configured to provide a variable degree of parallelization to the artificial neural network by providing scalability to neurons and layers. In a particular case, the systems and methods may include a back-propagation subsystem that is configured to scalably adjust weights in the artificial neural network in accordance with the variable degree of parallelization. Systems and methods are also provided for selecting an appropriate degree of parallelization based on factors such as hardware resources and performance requirements.

Claims

1. A method for designing a hardware configuration of an artificial neural network, the method comprising:

receiving information relating to hardware resources available for at least one hardware device;

receiving a desired network topology;

determining a plurality of degrees of parallelism for the desired network topology;

for each degree of parallelism of the plurality of degrees of parallelism estimating at least one of;

a hardware resource estimate to implement the network topology with the degree of parallelism; and

a performance estimate for the network topology with the degree of parallelism;

selecting a degree of parallelism based on the hardware resources available and at least one of the hardware resource estimates and the performance estimates; and

generating a hardware configuration based on the degree of parallelism simultaneously across a plurality of levels of hardware parallelism.

2. A method according to claim 1, wherein estimating the hardware resource estimate comprises:

determining a number of weights based on the network topology;

determining a measure of the hardware resources required to provide the determined number of weights based on the degree of parallelism; and

assigning the determined measure of the hardware resources required as the hardware resource estimate.

3. A method according to claim 1, wherein the estimating a performance estimate comprises:

determining a number of weights based on the network topology;

determining a measure of the hardware processing speed available;

determining a number of updates that can be performed on the number of weights in a predetermined time based on the processing speed and the degree of parallelism; and

assigning the determined number of updates as the performance estimate.

4. A method according to claim 1, wherein the selecting a degree of parallelism based on the hardware resources available and at least one of the hardware resource estimates and the performance estimates comprises:

determining the maximum hardware resources available from among the hardware resources available;

determining a hardware resource estimate from among the hardware resource estimates that is closest to but less than or equal to the maximum hardware resources available; and

determining the degree of parallelism associated with the determined hardware estimate.

5. A method according to claim 4, wherein determining a hardware resource estimate from among the hardware resource estimates that is closest to but less than or equal to the maximum hardware resources available further comprises determining the hardware resource estimate from among the hardware resource estimates that maximizes performance.

6. A method according to claim 1, wherein the method further comprises receiving information related to an application performance requirement and wherein the selecting a degree of parallelism based on the hardware resources available and at least one of the hardware resource estimates and the performance estimates comprises:

determining a performance estimate from among the performance estimates that is equal to or greater than the application performance requirement; and

determining the degree of parallelism associated with the determined performance estimate.

7. A computer readable non-transitory storage medium, the storage medium comprising instructions to execute the method of claim 1.

9. A method according to claim 1, wherein the generating a hardware configuration based on the degree of parallelism comprises generating a hardware configuration comprising:

an input layer;

at least one hidden layer;

an output layer;

a back-propagation subsystem configured to send error data back through the network to adjust weights associated with the output layer and the at least one hidden layer;

and

a parallelization system configured to provide the determined degree of parallelization to each of the input layer, at least one hidden layer, output layer and back- propagation system.

10. A method according to claim 9, further comprising configuring a hardware device based on the hardware configuration.

8. A method for designing a hardware configuration of an artificial neural network, the method comprising:

receiving information relating to hardware resources available for at least one hardware device;

receiving a desired network topology;

determining a plurality of degrees of parallelism for the desired network topology;

for each degree of parallelism of the plurality of degrees of parallelism estimating at least one of;

a hardware resource estimate to implement the network topology with the degree of parallelism; and

a performance estimate for the network topology with the degree of parallelism;

selecting a degree of parallelism based on the hardware resources available and at least one of the hardware resource estimates and the performance estimates; and

generating a hardware configuration based on the degree of parallelism; and

receiving an arithmetic representation and wherein the estimating at least one of a hardware resource estimate and a performance estimate is based on the received arithmetic representation.