Error back-propagation method and neural network system
First Claim
1. A neural network device for receiving a signal representing an ambiguous external stimulus, and performing artificial recognition, learning and updating by reiteratively comparing the signal to a signal representing an unambiguous stimulus comprising:
- a) feedforward resolving means comprising an input for receiving the comparative and external stimulus signals and a first plurality of processors (1, . . . , N) arranged in a layered architecture of K successive layers, for determining neuron output states Vjk for each successive layer, each processor comprising;
i) first data input means connected to the input of the resolving means for receiving input data;
ii) data output means for supplying output data;
each processor Nk having its first data input means coupled to the data output means of a preceding processing layer NK-1,iii) a read/write coefficient memory for storing a group of values of synaptic coefficients for weighting input data received at the first data input means;
iv) calculating means coupled to the coefficient memory and the first data input means for weighting the input data by the synaptic coefficients and for linearly combining the weighted input data for generating the output data;
v) second data input means for providing coefficient matrix update data;
vi) multiplier means havingA) first multiplier input means coupled to the first data input means for receiving the input data,B) second multiplier input means coupled to the second data input means, andC) multiplier output means for supplying updates, the multiplier means multiplying in parallel the input data by the input received at the second multiplier input;
vii) memory control means coupled between the coefficient memory and the multiplier means for generating respective sums by adding respective ones of the updates to respective ones of the values of the synaptic coefficients and also including means for replacing the values in the coefficient memory by the sums;
b) a central processing device having a main input coupled to the output of the last processor (N) for, upon detection of a discrepancy between desired output and device generated output data, supplying at the main output of the central processing device an error signal representative of the discrepancy;
c) an error back-propagation means comprising a second plurality of further processors (1, . . . , N-1) arranged in a layered architecture of successively descending layers, each successive further processor comprising;
i) first error input means connected to the central processing device output for receiving the error signal;
ii) error output means for supplying output error data to the next successive processor;
the first further processor having its first error input means coupled to the main output of the central processing device and to a next further processor, each next further processor having its first error input means coupled to the error output of a preceding further processor, each further processor also comprising;
iii) a further coefficient memory for storing a transpose matrix of synaptic coefficients for weighting component input error data received at the first error input means;
iv) further calculating means coupled to the further coefficient memory and the first error input means for weighting the input error components by the synaptic coefficients of the transpose matrix and for linearly combining the weighted input error components for generating the output error data;
v) second error input means for receiving transpose matrix coefficient updating;
vi) further multiplier means includingA) first multiplier input means coupled to the first error input means for receiving the input error components,B) second multiplier input means coupled to the second error input means, andC) multiplier output means for supplying further transpose matrix updates,the further multiplier means multiplying in parallel the input error components by input received at the second multiplier input means;
vii) further memory control means coupled between the further coefficient memory and the further multiplier means for generating respective further sums by adding respective ones of the further updates to respective ones of the values of the synaptic coefficients and replacing the values in the further coefficient memory by the further sums;
at least one of the second data input means in the resolving means being coupled to receive a value supplied at one of the error output means in the backpropagation means; and
at least one of the second error input means in the backpropagation means being coupled to receive a value produced at one of the data output means in the resolving means.
0 Assignments
0 Petitions
Accused Products
Abstract
Method and apparatus of error back-propagation for use in a neural network system. A first group (11) of processing devices (131, 132, 133) performs the resolving steps and a second group (12) of analogous processing devices (134, 135) performs the training steps while backpropagating errors calculated in a central processing device (10). The synaptic coefficient matrix Cij of the first group and the transposed matrix Tji of the second group are simultaneously updated. This updating of the synaptic coefficients can be performed by means of multipliers (341 to 34N) and adders (371 to 37N).
28 Citations
21 Claims
-
1. A neural network device for receiving a signal representing an ambiguous external stimulus, and performing artificial recognition, learning and updating by reiteratively comparing the signal to a signal representing an unambiguous stimulus comprising:
-
a) feedforward resolving means comprising an input for receiving the comparative and external stimulus signals and a first plurality of processors (1, . . . , N) arranged in a layered architecture of K successive layers, for determining neuron output states Vjk for each successive layer, each processor comprising; i) first data input means connected to the input of the resolving means for receiving input data; ii) data output means for supplying output data;
each processor Nk having its first data input means coupled to the data output means of a preceding processing layer NK-1,iii) a read/write coefficient memory for storing a group of values of synaptic coefficients for weighting input data received at the first data input means; iv) calculating means coupled to the coefficient memory and the first data input means for weighting the input data by the synaptic coefficients and for linearly combining the weighted input data for generating the output data; v) second data input means for providing coefficient matrix update data; vi) multiplier means having A) first multiplier input means coupled to the first data input means for receiving the input data, B) second multiplier input means coupled to the second data input means, and C) multiplier output means for supplying updates, the multiplier means multiplying in parallel the input data by the input received at the second multiplier input; vii) memory control means coupled between the coefficient memory and the multiplier means for generating respective sums by adding respective ones of the updates to respective ones of the values of the synaptic coefficients and also including means for replacing the values in the coefficient memory by the sums; b) a central processing device having a main input coupled to the output of the last processor (N) for, upon detection of a discrepancy between desired output and device generated output data, supplying at the main output of the central processing device an error signal representative of the discrepancy; c) an error back-propagation means comprising a second plurality of further processors (1, . . . , N-1) arranged in a layered architecture of successively descending layers, each successive further processor comprising; i) first error input means connected to the central processing device output for receiving the error signal; ii) error output means for supplying output error data to the next successive processor; the first further processor having its first error input means coupled to the main output of the central processing device and to a next further processor, each next further processor having its first error input means coupled to the error output of a preceding further processor, each further processor also comprising; iii) a further coefficient memory for storing a transpose matrix of synaptic coefficients for weighting component input error data received at the first error input means; iv) further calculating means coupled to the further coefficient memory and the first error input means for weighting the input error components by the synaptic coefficients of the transpose matrix and for linearly combining the weighted input error components for generating the output error data; v) second error input means for receiving transpose matrix coefficient updating; vi) further multiplier means including A) first multiplier input means coupled to the first error input means for receiving the input error components, B) second multiplier input means coupled to the second error input means, and C) multiplier output means for supplying further transpose matrix updates, the further multiplier means multiplying in parallel the input error components by input received at the second multiplier input means; vii) further memory control means coupled between the further coefficient memory and the further multiplier means for generating respective further sums by adding respective ones of the further updates to respective ones of the values of the synaptic coefficients and replacing the values in the further coefficient memory by the further sums;
at least one of the second data input means in the resolving means being coupled to receive a value supplied at one of the error output means in the backpropagation means; and
at least one of the second error input means in the backpropagation means being coupled to receive a value produced at one of the data output means in the resolving means. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16)
-
-
17. A processor for reiteratively receiving, processing and outputting data related to signal components representing an ambiguous external stimulus and for performing artificial recognition, learning and updating by repetitively comparing the signal components no signal components representing an unambiguous stimulus comprising:
-
i) first data input means for receiving input data; ii) data output means for supplying output data; iii) a read/write coefficient memory for storing a group of values of synaptic coefficients for weighting input data received at the first data input; iv) calculation means for weighting the input data by the synaptic matrix coefficients and for linearly combining the weighted input data components for generating the output data; v) second data input means for receiving matrix coefficient update data; vi) multiplier means including A) first multiplier input means coupled to the first data input means for receiving the input data components, B) second multiplier input means coupled to the second data input means, and C) multiplier output means for supplying updates, the multiplier means multiplying in parallel the input data components by data received at the second multiplier input; and vii) memory control means coupled between the coefficient memory and the multiplier means for generating respective sums by adding respective ones of the updates to respective ones of the values of the synaptic coefficients and replacing the values in the coefficient memory by the sums. - View Dependent Claims (18, 19, 20, 21)
-
Specification