Discrete weight neural network

• US 4,918,618 A
• Filed: 04/11/1988
• Issued: 04/17/1990
• Est. Priority Date: 04/11/1988
• Status: Expired due to Fees

• Current Assignee: Analog Intelligence Corporation
• Original Assignee: Analog Intelligence Corporation
• Inventors: Tomlinson, Max S. Jr.
• Primary Examiner(s): MacDonald, Allen
• Application Number:
  US07/180,236
• Time in Patent Office: 736 Days
• Field of Search: 364/513, 364/200, 364/276.6, 364/807, 382/14, 382/15
• US Class Current: 706/25
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

A Neural Network using interconnecting weights each with two values, one of which is selected for use, can be taught to map a set of input vectors to a set of output vectors.

A set of input vectors is applied to the network and in response, a set of output vectors is produced by the network. The error is the difference between desired outputs and actual outputs.

The network is trained in the following manner. A set of input vectors is presented to the network, each vector being propogated forward through the network to produce an output vector. A set of error vectors is then presented to the network and propagated backwards. Each Tensor Weight Element includes a selective change means which accumulates particular information about the error.

After all the input vectors are presented, an update phase is initiated. During the update phase, in accordance with the results of the derived algorithm, the selective change means selects the other weight value if selecting the other weight value will decrease the total error. Only one such change is made per set.

After the update phase, if a selected value was changed, the entire process is repeated. When no values are switched, the network has adapted as well as it can, and the training is completed.

Claims

1. A Discrete Weight Neural Network system for the mapping of a received Input Vector to a produced Output Vector, where the mapping function is adaptively learned by the application of a received Error Vector and a received update signal;

said Discrete Weight Neural Network comprising;

(1) an Input Layer comprising one or more Input Layer Processing Elements;

(2) a Tensor Weight Layer comprising one or more Tensor Weight Elements;

(3) a Hidden Layer comprising one or more Hidden Layer Processing Elements;

(4) a Matrix Weight Layer comprising one or more Matrix Weight Elements;

(5) an Output Layer comprising one or more Output Layer Processing Elements;

(6) each Input Layer Processing Element comprising;

(a) an input data line means for receiving as input data value an element of the Input Vector

(b) an output data line means for outputing a data output value; and

(c) means for transferring the input data from the input data line means to the output data line means;

(7) each Hidden Layer Processing Element comprising;

(a) one or more weighted data input line means for receiving weighted data input values;

(b) one or more weighted error input line means for receiving weighted error input values;

(c) a data output line means for transmitting a data output value;

(d) a net data feedback output line means for transmitting a net data feedback output value;

(e) an error output line means for transmitting an error output value;

(f) means for combining weighted data input values, for producing said data output value, and for producing said net data feedback output value; and

(g) means for combining weighted error input values and for producing said error output value;

(8) each Output Layer Processing Element comprising;

(a) one or more weighted data input line means for receiving weighted data input values;

(b) a data output line means for transmitting a data output value as an element of the Output Vector;

(c) summation means connected to said weighted data input lines for receiving the weighted data input value and for producing the data output value as a sum thereof;

(d) an input error line means for receiving an element of the Error Vector;

(e) an output error line means for transmitting an error output value; and

(f) means for transferring error information from said input error line means to said output error line means;

(9) each Matrix Weight Element comprising;

(a) a data input line means connected to the data output line means of one of said one or more Hidden Layer Processing Elements for receiving the data output value from that Hidden Layer Processing Element as a data input value;

(b) a weighted data output line means connected to one of the weighted data input line means of one of said one or more Output Layer Processing Elements for transmitting a weighted data output value to the Output Layer Processing Element as one of its weighted data input values;

(c) a error input line means connected to the error output line means of one of said one or more Output Layer Processing Elements for receiving the error output value from that Output Layer Processing Element as an error input value;

(d) a weighted error output line means connected to one of the weighted error input line means of one of said one or more Hidden Layer Processing Element(s) for transmitting a weighted error output value to that Hidden Layer Processing Element as one of its weighted error input values; and

(e) a weighting means including a weight value for transferring the data input value in proportion to the weight value as the weighted data output value to the weighted data output line means and for transferring the error input value in proportion to the weight value as the weighted data output value to the weighted error output line means; and

(10) each Tensor Weight Element comprising;

(a) a data input line means connected to a data output line means of one of said one or more Input Layer Processing Elements for receiving the data output value from that Input Layer Processing Element as a data input value;

(b) a weighted data output line means connected to one of the weighted data input line means of one of said one or more Hidden Layer Processing Elements for transmitting a weighted data output value to that Hidden Layer Processing Element as one of its weighted data input values;

(c) an error input line means connected to the error output line means of one of said one or more Hidden Layer Processing Elements for receiving the error output value from that Hidden Layer Processing Element as an error input value;

(d) a net data feedback input line means connected to the net data feedback output line means of one of said one or more Hidden Layer Processing Elements for receiving the net data feedback output value from that Hidden Layer Processing Element as a net data feedback input value;

(e) an update signal line means for receiving the update signal;

(f) selective weighting means comprising;

(i) selection means for producing a selected weight value, said selected weight value being either a low weight value or a high weight value dependent on said selector value; and

(ii) combining means for producing the weighted data output value from the data input value and the selected weight value; and

(g) selection change means for receiving the low weight value, the high weight value, the selector value, the error input value, the net data feedback input value, and the update signal and for determining therefrom if switching the selector value and thereby causing the non-selected weight value to become the selected weight value would decrease the total error, and if it would, for producing a switch signal.

2. A Tensor Weight Element for use in a neural network;

the neural network for receiving;

an input vector comprised of one or more input vector element values;

an error vector comprised of one or more error vector element values, each element value dependent on a corresponding output vector element value; and

an update signal;

the neural network for producing;

an output vector comprised of one or more output vector element values;

there being an associated total error value dependent on the magnitude of the error vector;

the neural network comprising;

an input processing element means for receiving an input vector element value, and for producing a data input value dependent on the input vector element value;

an output processing element means for receiving a weighted output data value and an error vector element value, and for producing an output vector element value dependent on the weighted output data value, a net data feedback value dependent on the weighted output data value, and an error feedback value dependent on the error vector element value;

said Tensor Weight Element for connection between the input processing element means and the output processing element means;

for receiving the input data value from the input processing element means, and the net data feedback value and the error feedback value from the output processing element means and for transmitting the weighted output data value to the output processing element means;

said Tensor Weight Element comprising;

(1) data input means for receiving the data input value from the input processing element means;

(2) weighted data output means for transmitting the weighted data output value to the output processing element;

(3) selective weighting means comprising;

(a) selector value means for storing and modifying a selector value;

(b) selector means for producing a selected weight value;

said selected weight value being either a low weight value or a high weight value dependent on the selector value; and

(c) combining means for combining the selected weight value with the data input value and, in response thereto, for producing the weighted data output value; and

(4) selective change means for determining from the net data feedback value, the error feedback value, the data input value, the low weight value, the high weight value and the selector value if switching the selector value, and thereby making the non-selected weight value the selected weight value will decrease the total error value of the neural network.

3. A tensor weight element for use in a neural network;

said tensor weight element comprising;

(1) data input value line for receiving a data input value;

(2) weighted data output line;

(3) selective weighting means comprising;

(a) selector value means for storing a selector value;

(b) selection means for producing a selected weight value;

said selected weight value being either a low weight value or a high weight value, dependent on the selector value;

(c) combining means for combining the selected weight value from said selection means with the data input value from said data input line to produce a weighted data output value on said weighted data output line; and

(d) selective change means for determining if changing the selector value, and thereby changing the selected weight, will decrease the error of the neural network.

4. A Tensor Weight Element for use in a neural network of the type having:

an input layer for receiving an input vector of one or more input vector elements, the input layer having one or more input processing elements, each for receiving an element of the input vector and in response thereto for producing a data value;

an output layer for producing an output vector of one or more output vector elements and for receiving an error vector of one or more error vector elements, the error vector being a function of the difference between the output vector and a target vector, the output layer having one or more output processing elements, each for receiving one or more weighted data values and in response thereto for producing an element of the output vector and a net data feedback value, and each for receiving a element of the error vector and in response thereto for producing an error output value;

said Tensor Weight Element for connecting an input processing element with an output processing element and comprising;

(1) data input value line for receiving the data value from an input processing element as an input data value;

(2) average error line means for receiving the error values from one or more output processing elements and for producing an average error value;

(3) net data feedback line means for receiving the net data feedback value from an output processing element;

(4) weighted data output line for outputing a weighted data value;

(5) selective weighting means comprising;

(a) selector value means for storing a selector value;

(b) selection means for producing a selected weight value;

said selected weight value being either a low weight value or a high weight value, dependent on the selector value;

(c) combining means for combining the selected weight value from said selection means with the data input value from said data input line to produce a weighted data output value on said weighted data output line; and

(6) selective change means for receiving;

the net data feedback value from said net data feedback line;

the average error value from said average error line means;

the data input value from said data input line; and

the low weight value, the high weight value, and the selector value from said selective weighting means; and

for determining therefrom if changing the selector value, and thereby changing the selected weight, will decrease the error of the neural network.