Learning method for neural network having discrete interconnection strengths

• US 5,644,681 A
• Filed: 12/23/1994
• Issued: 07/01/1997
• Est. Priority Date: 12/18/1989
• Status: Expired due to Fees

• Current Assignee: Mitsubishi Electronics America Inc (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Denki Kabushiki Kaisha (Mitsubishi Electric Corporation)
• Inventors: Takahashi, Masanobu, Kojima, Keisuke, Kyuma, Kazuo, Tai, Shuichi, Shinnishi, Toshio
• Primary Examiner(s): Downs, Robert W.
• Application Number:
  US08/363,597
• Time in Patent Office: 921 Days
• Field of Search: 395/23, 395/24, 395/25
• US Class Current: 706/25
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

A learning method for a neural network, in which at least a portion of the interconnection strength between neurons takes discrete values, includes the steps of updating an imaginary interconnection strength taking continuous values by using the quantity of update of the interconnection strength which has been calculated by using the discrete interconnection strength, and discretizing the imaginary interconnection strength so as to provide a novel interconnection strength.

Claims

1. A learning method for a neuron computer having a plurality of neurons and interconnections among the neurons, wherein each interconnection has a weight, wherein at least one weight is taken from a discrete set of possible weights, the learning method comprising the steps of:

(a) applying an input to the neuron computer;

(b) obtaining an actual output from the neuron computer;

(c) for each interconnection of the neuron computer having a weight taken from the discrete set,

(i) calculating an update quantity by comparing the actual output to a desired output associated with the applied input to obtain an error and by calculating the update quantity as a function of the error and outputs of neurons of the neuron computer,

(ii) updating an imaginary interconnection strength taken from a range of values, wherein each possible weight in the discrete set corresponds to more than one value in the range of values and each value in the range of values corresponds to only one possible weight in the discrete set, using the calculated update quantity,

(iii) discretizing the updated imaginary interconnection strength by converting the updated imaginary interconnection strength to the corresponding weight in the discrete set; and

(iv) setting the weights of the neuron computer to the corresponding discretized updated imaginary interconnection strengths; and

(d) repeating all steps for a plurality of inputs, until errors in the actual output with respect to the desired output of the neuron computer are below a predetermined value.

2. The learning method of claim 1, wherein the step of applying an input comprises the steps of:

(a) inputting a set of educator signals, each educator signal comprising an input and a corresponding desired output;

(b) selecting an educator signal from said set of educator signals; and

(c) applying the input of the selected educator signal to the electrical neural network.

3. The learning method of claim 1, wherein the step of calculating the update quantity comprises the step of squaring the calculated error.

4. The teaming method of claim 1, wherein the neurons of the neuron computer form an input layer, an intermediate layer and an output layer and wherein the step of calculating the update quantity includes the step of calculating ##EQU3## for all i and j, where i and j are integers, wherein Δ

W^IH ij represents the update quantity for the interconnection between an i-th neuron in the intermediate layer and a j-th neuron in the input layer, Δ

W^HO ij represents the update quantity for the interconnection between an i-th neuron in the output layer and a j-th neuron in the intermediate layer, W^HO ki represents the weight of the interconnection between a k-th neuron in the output layer and an i-th neuron in the intermediate layer, wherein k is an integer, V^O m, V^H m, and V^I m represent an m-th (m=i,j or k) output of the output, intermediate and input layers respectively, α

represents a learning rate, n₀ represents the number of neurons in the output layer, and V^O(m) represents the desired output associated with the input applied to the neuron computer.

5. A learning method for an optical neural network, the optical neural network having a plurality of neurons and interconnections among the plurality of neurons, wherein each interconnection has a strength, wherein at least one strength is taken from a discrete range of values, the learning method comprising the steps of:

applying an input to the optical neural network;

obtaining an actual output from the optical neural network;

calculating a quantity of update for the strength of each interconnection having a strength taken from a discrete range of values by comparing the actual output to a desired output to obtain an error and by calculating the quantity of update as a function of the error and outputs of the plurality of neurons of the optical neural network;

updating an imaginary interconnection strength taken from a second range of values for each interconnection having a strength taken from the discrete range of values, wherein each value in the discrete range of values corresponds to more than one value in the second range of values, and wherein each value in the second range of values corresponds to only one value in the discrete range of values, by using the calculated quantity of update;

discretizing each imaginary interconnection strength by converting the imaginary interconnection strength to the corresponding value in the discrete range of values;

setting each strength taken from a discrete range of values in the optical neural network to the corresponding discretized imaginary strength; and

repeating all steps for a plurality of inputs, until the actual output of the optical neural network converges.

6. The learning method of claim 5, wherein the step of comparing the actual output to a desired output includes the step of squaring a calculated difference between the actual output and the desired output.

7. The learning method of claim 5, wherein neurons of the optical neural network form an input layer, an intermediate layer and an output layer, and wherein the step of calculating the quantity, of update includes the step of calculating ##EQU4## for all i and j, wherein i and j are integers, wherein, Δ

W^IH ij represents the quantity of update for the interconnection between an i-th neuron in the intermediate layer and aj-th neuron in the input layer, Δ

W^HO ij represents the quantity of update for the interconnection between an i-th neuron in the output layer and aj-th neuron in the intermediate layer, W^HO ki represents the strength of the interconnection between a k-th neuron in the output layer and an i-th neuron in the intermediate layer, wherein k is an integer, V^O m, V^H m, and V^I m represent an m-th (m=i,j or k) output of the output, intermediate and input layers respectively, α

represents a learning rate, n_O represents the number of neurons in the output layer, and V^O(m) represents the desired output associated with the input applied to the optical neural network.

8. A learning method for an optical neural network, the optical neural network comprising neurons and interconnections among the neurons wherein each interconnection has an imaginary strength taken from a first range of values and an actual strength taken from a discrete range of values, wherein each value in the discrete range of values corresponds to more than one value in the first range of values and wherein each value in the first range of values corresponds to only one value in the discrete range of values, the learning method comprising the steps of:

(a) inputting a set of educator signals, each educator signal comprising an input and a corresponding desired output;

(b) selecting an educator signal from the provided set;

(c) discretizing the imaginary interconnection strength of each interconnection by converting the imaginary interconnection strength to the corresponding value in the discrete range of values to obtain the actual strength of the interconnection and setting the actual strength of the interconnection to the obtained actual strength;

(d) applying an input, corresponding to the selected educator signal to the optical neural network;

(e) obtaining an actual output from the optical neural network;

(f) calculating the error between the actual output of the optical neural network and the corresponding desired output of the selected educator signal;

(g) calculating an update value for each interconnection on the basis of the calculated error and outputs of the neurons of the optical neural network;

(h) adding the calculated update values to the imaginary interconnection strengths; and

(i) repeating steps (b) through (i) until the actual output of the optical neural network converges.

9. The learning method as set forth in claim 8, wherein the step of calculating the error includes the step of squaring the calculated error.

10. The learning method, as set forth in claim 8, wherein neurons of the optical neural network form an input layer, an intermediate layer and an output layer, and wherein the step of calculating the update includes the step of calculating ##EQU5## for all i and j wherein i and j are integers, Δ

W_ij^IH represents the update value for the interconnection between an i-th neuron in the intermediate layer and aj-th neuron in the input layer,

Δ

W_ij^HO represents the update value for the interconnection between an i-th neuron in the output layer and aj-th neuron in the intermediate layer,

W^HO ki represents the actual strength of the interconnection between a k-th neuron in the output layer and an i-th neuron in the intermediate layer, wherein k is an integer,.

^V.sbsp.O m, ^V.sbsp.H m, ^V.sbsp.I m represent an m-th (m=i,j or k) output of the output, intermediate and input layers respectively,

α

represents the learning rate,

n_O represents the number of neurons in the output layer, and V^O(m) represents the desired output corresponding to input applied to the optical neural network.

11. A learning method for an optical neural network, the optical neural network having a plurality of neurons and interconnections among the plurality of neurons, wherein at least one of the interconnections has a discrete weight taken from a discrete set of possible weights, the learning method comprising the steps of:

providing, for each interconnection having a discrete weight, an imaginary interconnection strength taken from a range of values, wherein each possible weight in the discrete set corresponds to more than one value in the range of values and each value in the range of values corresponds to only one possible weight in the discrete set;

discretizing the imaginary interconnection strengths by converting each imaginary interconnection strength to the corresponding weight in the discrete set;

setting the discrete weights of the optical neural network to the corresponding discretized imaginary interconnection strengths;

applying an input to the optical neural network;

obtaining an actual output from the optical neural network;

for each interconnection of the optical neural network having a discrete weight, calculating a quantity of update by calculating an error between the actual output and a desired output for the applied input and by calculating the quantity of update as a function of the error and outputs of neurons of the optical neural network, updating the corresponding imaginary interconnection strength by using the calculated quantity of update, and discretizing the updated imaginary interconnection strength;

repeating the steps of setting, applying, obtaining, calculating, update and discretizing until the actual output of the optical neural network converges.

12. The learning method of claim 11, wherein the step of comparing the actual output to the desired output includes the step of squaring a calculated difference between the actual output and the desired output.

13. The learning method of claim 11, wherein neurons of the optical neural network from an input layer an intermediate layer and a hidden layer, and wherein the step of calculating the quantity of update includes the step of calculating ##EQU6## for all i and j wherein i and j are integers, W^IH ij represents the quantity of update for the interconnection between an i-th neuron in the intermediate layer and a j-th neuron in the input layer, W^HO ij represents the quantity of update for the interconnection between an i-th neuron in the output layer and a j-th neuron in the intermediate layer, W^HO ki represents the discrete weight of the interconnection between a k-th neuron in the output layer and an I-th neuron in the intermediate layer, wherein k is an integer, V^O m, V^H m, and V^I m represent an m-th (m=i,j or k) output of the output, intermediate and input layers respectively, α

represents a learning rate, n_o represents the number of neurons in the output layer, and V^O(m) represents the desired output corresponding to the input applied to the optical neural network.

14. A learning method for an optical neural network having a plurality of neurons and interconnections among the neurons, wherein each interconnection has a weight, wherein at least one weight is taken from a discrete set of possible weights, the learning method comprising the steps of:

a. applying an input to the optical neural network;

b. obtaining an actual output from the optical neural network;

c. for each interconnection of the optical neural network having a weight taken from the discrete set,

(i) calculating an update quantity by calculating an error between the actual output and a desired output associated with the applied input and by calculating the update quantity as a function of the error and outputs of neurons of the optical neural network,

(ii) updating an imaginary interconnection strength taken from a range of values using the calculated update quantity, wherein a value in the range of values has only one corresponding weight in the discrete set, and a weight in the discrete set has more than one corresponding value in the range of values;

(iii) discretizing the updated imaginary interconnection strength by determining the weight in the discrete set corresponding to the updated imaginary interconnection strength; and

(iv) setting the weight of the interconnection to the discretized updated imaginary interconnection strength; and

d. repeating all steps for a plurality of inputs until errors in the actual output with respect to the desired output of the optical neural network are below a predetermined value.

15. The learning method of claim 14, wherein the step of applying an input comprises the steps of:

(a) inputting a set of educator signals, each educator signal comprising an input and a corresponding desired output;

(b) selecting an educator signal from said set of educator signals and

(c) applying the input of the selected educator signal to the optical neural network.

16. The learning method of claim 14, wherein the step of calculating the update quantity comprises the step of squaring the calculated error.

17. The learning method of claim 14, wherein neurons of the optical neural network form an input layer, an intermediate layer and an output layer and wherein the step of calculating the update quantity comprises the step of calculating ##EQU7## for all i and j, where i and j are integers, wherein Δ

W^IH ij represents the update quantity for the interconnection between an i-th neuron in the intermediate layer and a j-th neuron in the input layer, Δ

W^HO ij represents the update quantity for the interconnection between an i-th neuron in the output layer and a j-th neuron in the intermediate layer, W^HO ki represents the weight of the interconnection between a k-th neuron in the output layer and an i-th neuron in the intermediate layer, wherein k is an integer, V^O m, V^H m, and V^I m represent an m-th (m=i,j or k) output of said output intermediate and input layers respectively, α

represents a learning rate, n_O represents the number of neurons in the output layer, and V^O(m) represents the desired output associated with the input applied to the optical neural network.

18. The method of claim 14 wherein the optical neural network is an optoelectronic neural network.

19. The method of claim 18 wherein the step of setting the weights of the interconnections in the optical neural network comprises the step of setting weights defined by a spatial light modulator.

20. The method of claim 14 wherein the optical neural network is an optical neural computer.

21. The method of claim 20 wherein the step of setting the weights of the interconnections in the optical neural network comprises the step of setting weights defined by a spatial light modulator.

22. The method of claim 14 wherein the step of setting the weights of the interconnections in the optical neural network comprises the step of setting weights defined by a spatial light modulator.

23. A learning method for a neuron computer having a plurality of neurons and interconnections among the neurons, wherein each interconnection has a weight, wherein at least one weight is taken from a discrete set of possible weights, the learning method comprising the steps of:

(a) applying an input to the neuron computer;

(b) obtaining an actual output from the neuron computer;

(c) calculating an error between the actual output and a desired output associated with the applied input;

(d) propagating the calculated error backwards through the neuron computer to obtain an update quantity for each interconnection;

(e) for each interconnection having a weight taken from the discrete set,

(i) updating an imaginary interconnection strength taken from a range of values, wherein each possible weight in the discrete set corresponds to more than one value in the range of values and each value in the range of values corresponds to only one possible weight in the discrete set, using the calculated update quantity for the interconnection,

(ii) discretizing the updated imaginary interconnection strength by converting the updated imaginary interconnection strength to the corresponding weight in the discrete set, and

(iii) setting the weights of the neuron computer to the corresponding discretized updated imaginary interconnection strengths; and

(f) repeating all steps for a plurality of inputs, until errors in the actual output with respect to the desired output of the neuron computer are below a predetermined value.

24. A learning method for a neuron computer having a plurality of neurons and interconnections among the neurons, wherein each interconnection has a weight, wherein at least one weight is taken from a discrete set of possible weights, the learning method comprising the steps of:

(a) applying an input to the neuron computer;

(b) obtaining an actual output from the neuron computer;

(c) for each interconnection of the neuron computer having a weight taken from the discrete set,

(i) calculating an update quantity by comparing the actual output to a desired output associated with the applied input to obtain an error and by calculating the update quantity as a function of the error and weights of interconnections of the neuron computer,

(ii) updating an imaginary interconnection strength taken from a range of values, wherein each possible weight in the discrete set corresponds to more than one value in the range of values and each value in the range of values corresponds to only one possible weight in the discrete set, using the calculated update quantity,

(iii) discretizing the updated imaginary interconnection strength by converting the updated imaginary interconnection strength to the corresponding weight in the discrete set, and

(iv) setting the weights of the neuron computer to the corresponding discretized updated imaginary interconnection strengths; and

(d) repeating all steps for a plurality of inputs, until errors in the actual output with respect to the desired output of the neuron computer are below a predetermined value.

25. The learning method of claim 24 wherein the step of calculating the update quantity further includes using outputs of the neurons of the neuron computer.