Learning process system for use with a neural network structure data processing apparatus

• US 5,333,239 A
• Filed: 01/11/1993
• Issued: 07/26/1994
• Est. Priority Date: 09/12/1988
• Status: Expired due to Term

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: Masuoka, Ryusuke, Watanabe, Nobuo, Kawamura, Akira, Kimoto, Takashi, Asakawa, Kazuo
• Primary Examiner(s): MacDonald, Allen R.
• Assistant Examiner(s): Downs, Robert W.
• Application Number:
  US08/003,856
• Time in Patent Office: 561 Days
• Field of Search: 395/24, 395/23
• US Class Current: 706/25
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent

Abstract

A learning process system is provided for a neural network. The neural network is a layered network comprising an input layer, an intermediate layer and an output layer formed of basic units. In the basic units, a plurality of inputs is multiplied by a weight signal and the products are accumulated, thereby supplying the sum of products. An output signal is obtained using a threshold value function in response to the sum of products. An error signal is generated by an error circuit in response to a difference between the output signal obtained from the output layer and a teacher signal. A weight updating signal is determined in a weight learning circuit by obtaining a weight value in which the sum of the error values falls within an allowable range. Thus, the learning is performed in the layered neural network through use of a back propagation method. Through such learning in the layered neural network, an updating quantity to be obtained in the present weight updating cycle is determined in response to a once delayed weight updating quantity signal in a previous weight updating cycle and a twice delayed weight updating quantity obtained at a twice-previous weight updating cycle prior to the previous weight updating cycle.

Claims

1. A learning process system for use with a neural network comprising an input layer including a first plurality of basic units, each including a basic unit input each connected to receive an input signal and a basic unit output each connected to receive a weight signal to be multiplied by the input signal received by the basic unit input, connected to provide a sum-of-products and connected to convert the sum-of-products by a threshold value function to provide a final result signal at a basic unit output, at least one intermediate layer including a second plurality of basic units, each including a basic unit input each connected to receive another input signal and a basic unit output each connected to receive the weight signal to be multiplied by the another input signal received by the basic unit input, connected to provide another sum-of-products and connected to convert the another sum-of-products by another threshold value function to provide another final result signal at the basic unit output, an output layer including at least one basic unit, each including a basic unit input each connected to receive a further input signal and a basic unit output each connected to receive the weight signal to be multiplied by the further input signal received by the basic unit input, connected to provide a further sum-of-products and connected to convert the further sum-of-products by a further threshold value function to provide a further final result signal at the basic unit output, and internal connections from the basic unit outputs of the input layer to the basic unit inputs of the at least one intermediate layer and from the basic unit outputs of the at least one intermediate layer to the basic unit inputs of the output layer, said learning process system comprising:

output signal providing means for, when a plurality of predetermined input signals are supplied to the input layer basic unit inputs, receiving first output signals from the basic unit output of each at least one basic unit of the output layer and for providing second output signals identical to the first output signals,

error value means for receiving the second output signals and a teacher signal and for generating error signals designating an inconsistency between the second output signal and the teacher signal, and

weight learning means for calculating and storing a weight updating signal in respective updating cycles and for generating the weight signal so that a sum of the error signals becomes within a predetermined allowable range by sequentially updating in updating cycles the weight signal in accordance with the weight updating signal based on the sum of the error signals determined at a first updating cycle, based on a once dalyed weight updating signal determined at a second updating cycle prior to the first updating cycle, and based on a twice delayed weight updating signal determined at a third updating cycle prior to the second updating cycle.

2. The learning process system according to claim 1, wherein said weight learning means comprises:

a register group for temporarily storing a first parameter -ε

, a second parameter α

, and a third parameter -β

,

memory means for storing the weight signal, the present weight updating signal, the once dalyed weight updating signal, and the twice delayed weight updating signal,

multiplying means for executing a first multiplication of -ε

by a partial differentiation of the error signals, a second multiplication of α

by the once delayed weight updating signal, and a third multiplication of -β

by the twice delayed weight updating signal,

first adding means for adding results of the first, second and third multiplications to produce a result,

second adding means for adding the weight signal to the result of said first adding means to determine a new weight and storing the new weight signal in the memory means.

3. The learning process system according to claim 2, wherein said memory means comprises means for receiving a common weight address selection signal and a shift signal, for sequentially shifting the weight updating signal to provide the once delayed weight updating signal and the twice delayed weight updating quantity.

4. The learning process system according to claim 2, further comprising means for providing a weight selection signal comprising a layer number, a basic unit number and a connection number as an address signal to said memory means.

5. The learning process system according to claim 4, wherein said weight learning means includes means for determining the weight signal in response to the error signal based on the following differential equation, and ##EQU24## wherein a differential approximate of the differential equation is expressed with regard to weight updating signal (Δ

W(t)) as follows, ##EQU25## where Δ

W(t) is the weight updating signal,

E is the error signal,

Δ

W(t-1) is the once delayed weight updating signal,

Δ

W(t-2) is the twice delayed weight updating signal, and

ε

, α and

β

are predetermined coefficients.

6. The learning process system according to claim 1, wherein each basic unit comprises part of an analog neuron chip.

7. The learning process system according to claim 1, wherein said input, intermediate and output layers are controlled by an analog time division signal.

9. The learning process system according to claim 6,

wherein said learning process system further comprises:

a weight data memory,

a weight updating quantity memory,

a previous weight updating quantity memory, and

a twice-previous weight updating quantity memory,

wherein said main control circuit means includes means for outputting a clear signal to said weight updating quantity memory, said previous weight updating quantity memory, and said twice-previous weight updating quantity memory for clearing said weight updating quantity memory, said previous weight updating quantity memory and said twice-previous weight updating quantity memory, outputting a shift signal for copying the weight updating signal from said weight updating quantity memory to said previous weight updating quantity memory and copying the once delayed weight updating signal from the previous weight updating quantity memory to said twice previous weight updating quantity memory, a weight selecting signal for an address for said weight data memory, said weight updating quantity memory, said previous weight updating quantity memory and said twice-previous weight updating quantity memory, a weight reading signal for controlling reading of said weight data memory, said weight updating quantity memory, said previous weight updating quantity memory and said twice-previous weight updating quantity memory, an initial value setting signal for setting an initial value in the weight data memory, learning parameter signals -ε

, α and

-β

for said weight learning means and a weight updating signal for synchronizing the weight learning means.

10. The learning process system according to claim 9, wherein said weight updating signal comprises a synchronization pulse signal and said weight learning means includes:

means for multiplying the learning parameter signals with the error rate signal, the contents of the previous weight updating quantity memory, and the contents of the twice-previous weight updating quantity memory to produce products in a first time period;

first adding means for adding the products and for outputting the weight updating signal in a second time period

second adding means for updating the weight signal in a third time period; and

means for storing the weight data into said weight data memory in a fourth time period.

11. The learning process system according to claim 6, wherein said weight learning means in the network comprises:

serial read and write analog neuron chips connected to store the weight signal and the weight updating signal, the once delayed weight updating signal and the twice delayed weight updating signal at an address.

8. A learning process system for use with a network comprising a layered network including an input layer, at least one intermediate layer and an output layer, each layer being responsive to a weight signal,

input signal maintaining means for maintaining an input signal to be input to the layered network, the input signal being selected based on a pattern selection signal,

teacher signal maintaining means for maintaining a teacher signal,

error means for calculating an error signal between an output of said layered network and the teacher signal when said layered network receives the input signal and for outputting an error rate signal representing a rate of change of the error signal,

weight learning means for receiving the error rate signal, for calculating and storing weight updating signal in respective updating cycles and for generating the weight updating signal in response to the error rate signal, a once delayed weight updating signal and a twice delayed weight updating signal, and

main control circuit means for providing the weight signal to the layered network, for performing an initial setting of said weight learning means, before said weight learning means determines the updating quantities, and for applying the pattern selection signal to said input signal maintaining means to perform control of said error means.

12. A learning process system for use in a network, comprising:

a single layer including a plurality of basic units, each including a basic unit input each connected to receive an input signal and a basic unit output each connected to receive a weight signal to be multiplied by the input signal received by the basic unit input, connected to provide a sum-of-products and connected to convert the sum-of-products by a threshold value function to provide an output signal at a basic unit output;

a feedback path for selectively feeding back output signals at basic unit outputs as inputs to basic unit inputs to sequentially operate as an input layer, an intermediate layer and an output layer;

output signal providing means for receiving a first output signal from the basic unit output of at least one basic unit of said single layer when operating as the output layer and providing a second output signal based on the first output signal by supplying a plurality of predetermined input signals to the basic unit input of at least one basic unit of said single layer when operating as the input layer;

error value means for obtaining the second output signal from said output signal providing means and a teacher signal and for calculating an error signal designating an inconsistency between the second output signal and the teacher signal; and

weight learning means for calculating and storing a weight updating signal in respective updating cycles and for producing the weight signal so that a sum of the error signals becomes within a predetermined allowable range by sequentially updating in updating cycles the weight signal in accordance with the weight updating signal based on the sum of the error signals determined at a first one of the updating cycles, based on a once delayed weight updating signal determined at a second updating cycle prior to the first updating cycle and based on a twice delayed weight updating signal determined at a third updating cycle prior to the second updating cycle.

13. A method of learning in a layered neural network, comprising the steps of:

(a) determining in the layered neural network an output signal in response to a weight signal from a weight learning circuit;

(b) generating an error signal in an error circuit between a teacher signal and the output signal of the layered neural network; and

(c) generating the weight signal in a weight learning circuit in response to a previous weight signal and a weight updating signal, the weight updating signal being generated in response to the error signal from the error circuit, a once delayed weight updating signal and a twice delayed weight updating signal.

14. A method according to claim 13, wherein said step (c) comprises the substeps of:

(c1) obtaining a rate of change of the error signal in response to the error signal;

(c2) generating the weight updating signal in the weight learning circuit in response to the rate of change of the error signal, the once delayed weight updating signal, and the twice delayed weight updating signal;

(c3) delaying the once delayed weight updating signal; and

(c4) delaying the weight updating signal.

15. A method according to claim 13, wherein said step (c) comprises the substep of:

(c1) determining a rate of change of the error signal.

16. A method according to claim 15, wherein said step (c) further comprises the substeps of:

(c2) summing in the weight updating circuit the rate of change of the error signal corresponding to each of a plurality of output signals to produce a summed differential error signal; and

(c3) determining in the weight updating circuit at least one weight updating signal in response to the summed differential error signal, the once delayed weight updating signal and the twice delayed weight updating signal.

17. A method according to claim 13, further comprising the steps of:

(d) determining if the error signal exceeds a predetermined threshold; and

(e) repeating steps (a) through (d) when said step (d) determines the error signal exceeds the predetermined threshold.

18. A learning process system for use with a layered neural network comprising an input layer, at least one intermediate layer connected to respond to the input layer and a weight signal, and an output layer connected to respond to at least one of the at least one intermediate layer, said learning process system comprising:

a teacher signal providing circuit;

an error circuit, operatively connected to said output layer and said teacher signal providing circuit and connected to output an error signal; and

a weight learning circuit operatively connected to said error circuit and the layered neural network to provide the weight signal to the layered neural network including means for generating the weight signal in response to a previous weight signal and a weight updating signal, the weight updating signal being generated in response to the error value signal, a once delayed weight updating signal and a twice delayed weight updating signal.

19. An apparatus according to claim 18, wherein said weight learning circuit includes means for generating the weight updating signal in response to the error value signal, a once delayed weight updating signal and a twice delayed weight updating signal.

20. An apparatus according to claim 19, wherein said weight learning circuit includes means for providing the weight updating signal as ##EQU26## where Δ

W(t) is the weight updating signal,

E is the error signal,

Δ

W(-1) is the once delayed weight updating signal,

Δ

W(t-2) is the twice delayed weight updating signal, and

ε

, α and

β

are predetermined coefficients.

21. A learning process system, comprising:

a neural network capable of generating an output signal in response to an input signal and a weight signal;

a teacher signal circuit capable of generating a teacher signal;

an error circuit, operatively connected to receive the teacher signal from said teacher signal providing circuit and the output signal from said neural network, to provide an error signal; and

a weight learning circuit, operatively connected to receive the error signal from said error circuit and to provide the weight signal to said neural network, the weight signal being generated in response to a previously generated weight signal and a weight updating signal, the weight updating signal being generated in response to the error signal, a once delayed weight updating signal and a twice delayed weight updating signal.

22. A learning system according to claim 21, wherein said weight learning circuit generates the weight updating signal in response to a rate of change of the error signal, a once delayed weight updating signal and a twice delayed weight updating signal.

23. A learning process system comprising:

a layered neural network having weight values input by a weight value input signal and updated during weight value updating cycles, said layered neural network receiving a first input signal and producing a first output signal responsive to the first input signal; and

weight learning means for receiving the first output signal, for determining updates to said weight values for a current weight value updating cycle responsive to said weight values for two weight value updating cycles immediately prior to the current weight value updating cycle, and for producing said weight value input signal.