Scalable flow virtual learning neurocomputer

• US 5,329,611 A
• Filed: 06/21/1993
• Issued: 07/12/1994
• Est. Priority Date: 05/22/1990
• Status: Expired due to Fees

• Current Assignee: AST Research, Inc. (Samsung Electronics Co. Ltd.)
• Original Assignee: International Business Machines Corporation
• Inventors: Pechanek, Gerald G., Vassiliadis, Stamatis, Delgado-Frias, Jose G.
• Primary Examiner(s): MacDonald, Allen R.
• Application Number:
  US08/080,989
• Time in Patent Office: 386 Days
• Field of Search: 395/27, 395/11, 395/800
• US Class Current: 706/42
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: G06N 3/10 Interfaces, programming languages or software development kits, e.g. for simulating neural networks

Abstract

A scalable flow virtual learning neurocomputer system and appratus with a scalable hybrid control flow/data flow employing a group partitioning algorithm, and a scalable virtual learning architecture, synapse processor architecture mapping, inner square folding and array separation, with capability of back propagation for virtual learning. The group partitioning algorithm creates a common building block of synapse processors containing their own external memory. The processor groups are used to create a general purpose virtual learning machine which maintains complete connectivity with high performance. The synapse processor group allows a system to be scalable in virtual size and direct execution capabilities. Internal to the processor group, the synapse processors are designed as a hybrid control flow/data flow architecture with external memory access and reduced synchronization problems.

Claims

1. A computer system comprising a virtual neural synapse processor apparatus including:

an N physical neuron structure, wherein N is an integer greater than or equal to one, partitioned into groups including

means for synapse processing including means for storing instructions and data, means for receiving instructions and data, means for executing instructions, and means for interfacing with external data storage for weights and neuron output values for V neurons, where V is greater than N and V is a number of neurons to be emulated (virtual neurons) on the N neuron structure,

means for executing group instructions, and communicating adder trees, said N neuron structure further including

neuron activation means for controlling a start of a neuron activation function,

means for interconnecting said groups into the N neuron structure, and

means for communicating instructions, data, and outputs of the neuron activation means back to input means for synapse processing through the communicating adder trees.

2. The system according to claim 1 wherein the apparatus includes N² means for synapse processing, each associated with a connection weight in a N neuron network which is placed in the form of a N by N connection weight matrix folded along a diagonal and which includes diagonal cells and general cells.

3. The system according to claim 2 wherein the diagonal cells each include a single means for synapse processing and are associated with diagonal connection weights of the folded N by N connection weight matrix, and wherein the general cells each include a top means for synapse processing and a bottom means for synapse processing, said general cells associated with symmetric connection weights of the folded N by N connection weight matrix.

4. The system according to claim 3 wherein the diagonal cells'"'"' means for synapse processing include means for supplying results to and receiving instructions and data from an attached communicating adder tree of the N neuron structure.

5. The system according to claim 3 wherein the top and bottom means for synapse processing each include means for supplying a result to and receiving instructions and data from an attached communicating adder tree of the N neuron structure.

6. The system according to claim 2 wherein the N neuron structure is partitioned into G groups, where G is an integer greater than one, and wherein each said group includes diagonal cells and general cells and each said diagonal cell is attached to a communicating adder tree and each said general cell is attached to a top communicating adder tree and a bottom communicating adder tree.

7. The system according to claim 6 wherein each said group further includes means for interfacing with means for external storage, said means for interfacing including H read and write weight value data paths, with H an even power of two, H read only neuron output value data paths, H write only neuron output value data paths, means for addressing group weight values, and means for separate addressing of the write only neuron output value data paths and the read only neuron output value data paths.

8. The system according to claim 7 further including a common weight value reading and writing means for simultaneously reading H weight values from the means for external storage and for simultaneously writing H weight values to the means for external storage.

9. The system according to claim 7 wherein the apparatus further includes a programmable group execution means responsive to group commands, received from a selected communicating adder tree, for controlling the means for interfacing with means for external storage.

10. The system according to claim 1 wherein the means for storing instructions and data includes one bit which controls whether the data may be overwritten.

11. The system according to claim 1 wherein the means for neuron activation includes a non-linear sigmoid generator for providing a neuron value ("Y") output and a sigmoid derivative generator for providing a Y (1-Y) value output.

12. The system according to claim 1 wherein the communicating adder trees contain Log₂ N 2 to 1 communicating adder stages.

13. The system according to claim 12 wherein each stage in the communicating adder trees contain 2 to 1 communicating adders comprised of a 2 to 1 adder, an adder bypass path for communicating values in a reverse direction than that used for an addition, and means for switching between the addition and a reverse communication path.

14. A neural synapse processor apparatus comprising a neural network emulation structure including:

partitioned groups comprised of at least one synapse processing unit including means for storing instructions and data, means for receiving instructions and data, means for executing instructions, means for interfacing with external storage units to communicate weights and neuron output values, means for executing group instructions, and communicating adder trees;

neuron activation units;

means for interconnecting the groups into a N physical neuron structure supporting the emulation of V neurons (virtual neurons), where N and V are positive integers and V is greater than or equal to N; and

means for communicating instructions, data, and outputs of the neuron activation units back to the synapse processing units through the communicating adder trees;

the synapse processing units each associated with a connection weight in a neutral network, the neural network in the form of a N by N matrix folded along a diagonal, including diagonal cells and general cells, each of the diagonal cells including one of the synapse processing units and each of the general cells including two of the synapse processing units.

15. The apparatus according to claim 14 wherein the neural network is separated into one or more square matrices each including 2H of said synapse processing units made up of general cells and multiple triangular matrices each including H of said synapse processing units comprising said diagonal and general cells, where H is a positive integer.

16. The apparatus according to claim 15 wherein each of the square matrices is folded along its diagonal by folding communicating adder trees communicating with said synapse processing units.

17. The apparatus according to claim 16 wherein the folded square matrices become two separate triangular arrays by rearranging the weights and neuron output values in each synapse processing unit, and wherein each of the triangular arrays includes H synapse processing units and wherein each of the triangular arrays represents a processor group.

18. The apparatus according to claim 14 wherein the partitioned groups are attached to external summation and switching mechanisms, said summation and switching mechanisms including:

means for external summation of N partial neuron input summations generated from the groups'"'"' communicating adder trees;

means for reverse communication of neuron activation unit output values to the corresponding synapse processing unit'"'"'s storage, said synapse processing units including corresponding connection weight storage;

means for reverse communication of neuron error signal values, wherein each of the error signal values is communicated to one of the corresponding synapse processing unit'"'"'s storage;

means for external summation of partial weighted error summations of products generated from the groups'"'"' communicating adder trees.

19. The apparatus according to claim 18 wherein the external summation and switching mechanisms are attached to iterative adders each including an accumulating storage unit.

20. The apparatus according to claim 19 wherein the iterative adders are incorporated into a final summation stage of an external summation tree.

21. The apparatus according to claim 19 wherein the iterative adders each connect to an additional summation stage wherein an external input value is summed with an output of one of the iterative adders and said additional summation stage provides results to the neuron activation units.

22. The apparatus according to claim 19, wherein the apparatus further comprises a programmable processor controller which interfaces to a host computer system and to external summation and switching mechanisms, the processor controller including;

means for controlling the external summation and switching mechanisms, the communicating adder trees, and the iterative adders;

means for initializing the synapse processing units;

means for issuing synapse processing unit instructions and data;

means for calculating algorithmic data;

means for starting and for stopping a neural network emulation; and

means for sequence control of multi-layer networks,

the means for stopping a neural network emulation including means for stopping by convergence testing to a desired result or by completing a host computer system specified number of emulation cycles.

23. The apparatus according to claim 14, wherein the external storage units include one weight array, a present neuron output value array, and a new neuron output value array.

24. The apparatus according to claim 23, further comprising means for storing newly created neuron output values into the new neuron output value array while using neuron output values read from the present neuron output value array in neural calculations, and means for writing the newly created neuron output values in the present neuron output value array from which neuron output values are read.

25. The apparatus according to claim 14, wherein the N neuron structure is scaled to a larger V by adding more external storage locations in each external storage unit that interfaces with the synapse processing units.

26. The apparatus according to claim 14 wherein the weights and neuron output values are stored in a bit-serial format in the external storage units.

27. The apparatus according to claim 14, wherein the two synapse processing units in each of the general cells comprise a top synapse processing unit and a bottom synapse processing unit, the apparatus further comprising:

data path storage units; and

means for directing instructions;

wherein during one mode of operation the means for directing instructions passes instructions received from a top communicating adder tree to a top synapse processing unit'"'"'s instruction storage unit and instructions received from a bottom communicating adder tree to a bottom synapse processing unit'"'"'s instruction storage unit and data received from the top communicating adder tree to a top synapse processing unit'"'"'s specified data storage unit and data received from the bottom communicating adder tree to a bottom synapse processing unit'"'"'specified data storage unit.

28. The apparatus according to claim 27, wherein, during a second mode of operation, the means for directing instructions passes instructions received from the top communicating adder tree to the bottom synapse processing unit'"'"'s instruction storage unit and instructions received from the bottom communicating adder tree to the top synapse processing unit'"'"'s instruction storage unit and data received from the top communicating adder tree to the bottom synapse processing unit'"'"'s specified data storage unit and data received from the bottom communicating adder tree to the top synapse processing unit'"'"'s specified data storage unit.

29. A neural synapse processor apparatus comprising:

N² synapse processing units;

N communicating adder trees;

a programmable processor controller; and

N neuron activation units for a N physical neuron structure partitioned into G groups of H synapse processing units per group, each said group interfacing with means for external storage for providing storage capabilities for V virtual neurons, where V is greater than N, with N external summation switching mechanisms, where G, H, V, and N are positive integers.

30. The apparatus according to claim 29, wherein each communicating adder tree is attached to N synapse processing units at leaf nodes of the tree and to one neuron activation unit, and wherein the apparatus further comprises:

means for communicating data between the means for external storage and the synapse processing units;

means for controlling a destination of data received from the means for external storage in each synapse processing unit; and

means for controlling the external summation and switching mechanisms.

31. The apparatus according to claim 30 further comprising means for inputting weights and data values to each of the synapse processing units, the means for inputting weights and data values including the processor controller which has access to a weight and data value storage unit in each synapse processing unit and in external storage.

32. The apparatus according to claim 30 further comprising means for communicating tagged instructions, data, and neuron output values back to the synapse processing units through tag matching units and the communicating adder trees.

33. The apparatus according to claim 30, further comprising an instruction decoding mechanism, for controlling a destination of instructions, data, and neuron output values received from the attached communicating adder trees, in each of the synapse processing units.

34. The apparatus according to claim 30, further comprising means for executing received synapse processor instructions in each synapse processing unit responsive to a synapse processor instruction set.

35. The apparatus according to claim 30, further comprising means for executing received group instructions in each of the G groups by separate group logic controlling external storage address formation and responsive to a synapse processor instruction set.

36. The apparatus according to claim 30, further comprising instruction decoding means for operand selection and destination path control, thereby allowing results to stay locally in each synapse processing unit, to be sent to external storage, or to be sent to the attached communicating adder tree.

37. The apparatus according to claim 30, further comprising:

means for signalling completion of operations whose results remain in each synapse processing unit to the programmable processor controller; and

means for signalling an unsuccessful synapse processor operation.

38. The apparatus according to claim 30, further comprising means for summing values received from the synapse processing units via the attached communicating adder trees acting in a function execution mode and via the external summation and switching mechanisms acting in a summation mode.

39. The apparatus according to claim 30, further comprising means for inputting external input values to each neuron activation unit, the external input value provided by the programmable processor controller and applied to a final summation stage located at an output of an iterative adder.

40. The apparatus according to claim 30, further comprising means for generating new instructions, data, and neuron output values via the neuron activation units and the programmable processor controller which interfaces to an attached host computer and to the N external summation and switching mechanisms.