Method of assigning initial values of connection parameters to a multilayered neural network

• US 5,490,236 A
• Filed: 08/08/1994
• Issued: 02/06/1996
• Est. Priority Date: 05/22/1989
• Status: Expired due to Fees

• Current Assignee: Toru Niki
• Original Assignee: Toru Niki
• Inventors: Niki, Toru
• Primary Examiner(s): Downs, Robert W.
• Application Number:
  US08/286,914
• Time in Patent Office: 547 Days
• Field of Search: 395/22, 395/23, 395/21
• US Class Current: 706/25
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

A generator in a back propagation type neural network having an input layer, an output layer and an intermediate layer coupled the input and output layers forms initial values for connection parameters. A first generator produces an initial value W10 of a weight coefficient of each connection parameter of the intermediate layer from in-class covariant data SW and inter-class co-variant data SB over data inputted to the input layer. The produced values are set into respective synapses of the intermediate layer as connection parameters.

Claims

1. An apparatus for classifying input data representing a physical quantity into predetermined k classes, each class designating a set including a plurality of data which have similar characteristics, said apparatus comprising

a neural network comprising an input layer, an output layer, and an intermediate layer coupled between the input layer and the output layer, and

determining means for determining connection parameters between the layers of said neural network by using training data prior to classifying the input data, the connection parameters being represented at least by a weight coefficient,

wherein the input layer includes means for inputting data;

the intermediate layer includes means for receiving data from the input layer and processing the received data by means of the connection parameters;

the output layer includes means for receiving data processed by the intermediate layer, processing the received data by means of the connection parameters, and outputting signals indicating one of the predetermined k classes; and

the determining means comprises;

a first generating means for generating initial value data W₁₀ of the weight coefficient of each connection parameter of the intermediate layer from in-class dispersion data S_W and inter-class dispersion data S_B of the training data over all classes into which the training data has been partitioned, said in-class dispersion data S_W representing a dispersion degree of the training data within a class and said inter-class dispersion data S_B representing a dispersion degree of the training data over classes;

setting means for setting the generated initial value data into the respective synapses of the intermediate layer, as connection parameters; and

correction means for correcting the connection parameters by using a back-propagation learning method.

2. The apparatus according to claim 1, wherein said first generating means calculates the matrix data of said inclass covariant S_W and the matrix data of interclass covariant S_B when input data is represented as a vector, calculates the data of a matrix representing a mapping A such that the inter-class covariant matrix data S_B becomes maximum while keeping the in-class covariant matrix data S_W constant and calculates said initial value data W₁₀ by calculating the transposed matrix of the calculated matrix representing the mapping A.

3. A method for classifying input data representing a physical quantity into predetermined k classes, each class designating a set including a plurality of data which have a similar characteristic, by using a back propagation type neural network having an input layer, an output layer and an intermediate layer coupled between the input and output layers, each layer comprising a network of synapse elements and connections between the synapse elements, each connection being represented by a connection parameter which includes at least a weight coefficient, the method comprising the steps of:

(a) partitioning a plurality of training data x into k classes;

(b) generating data representing an in-class dispersion matrix S_W and data representing an inter-class dispersion matrix S_B of the training data over all classes into which the training data has been partitioned, each class designating a set in which a plurality of training data which have similar characteristics are included, said in-class dispersion data S_W representing a dispersion degree of the training data within a class and said inter-class dispersion data S_B representing a dispersion degree of the training data over classes on the basis of the plurality of input data;

(c) determining initial values W₁₀ of the weight coefficients of the synapses of the intermediate layer on the basis of the data representing said inter-class dispersion matrix S_W and said in-class dispersion matrix S_B ; and

(d) determining final values of said weight coefficients of the synapses of the intermediate layer beginning with the initial values W₁₀ according to back propagation type learning method in which training is input via said input layer, and setting connection parameters including the determined final weight coefficients into the respective synapses of the intermediate layer;

(e) inputting input data to the input layer;

(f) processing the input data through the intermediate layer having the determined weight coefficients; and

(g) further processing the data processed through the intermediate layer by means of the output layer, and outputting from the output layer signals indicating one of the predetermined k classes to which the input data belongs.

4. The learning method according to claim 3, wherein said step (c) further comprises the steps of:

(c1) forming a mapping A, such that the inter-class covariant matrix S_B becomes maximum while keeping said in-class covariant matrix S_W constant; and

(c2) generating said initial value W₁₀ by calculating a transposed matrix of a matrix representing the mapping A.

5. An apparatus for classifying input data representing a physical quantity into predetermined k classes, each class designating a set including a plurality of data which have similar characteristics, said apparatus comprising:

a back propagation type neural network including an input layer, an output layer, and an intermediate layer coupled between the input layer and the output layer; and

determining means for determining connection parameters between the layers of said neural network by using training data prior to classifying the input data, each connection parameter being represented by a weight coefficient and a bias θ

,

wherein the neural network input layer includes means for inputting data;

the neural network intermediate layer includes means for receiving data from the input layer and processing the received data by means of the connection parameters;

the neural network output layer includes means for receiving data processed by the intermediate layer, processing the received data by means of the connection parameters and outputting signals indicating one of the predetermined k classes; and

the determining means comprises;

generating means for generating data representing an initial value θ

₁₀ of the bias of each connection parameter of the intermediate layer from a total average x of input data x to be input to synapses of the input layer and a weight coefficient W₁₀ of each connection parameter of the intermediate layer; and

setting means for setting the generated initial values θ

₁₀ and W₁₀ into the respective synapses of the intermediate layer, as beginning connection parameters for the back propagation type neural network.

6. The apparatus according to claim 5, wherein said generating means generates a vector representing the bias θ

₁₀ of the intermediate layer on the basis of θ

₁₀ =W₁₀ x_tot wherein x_tot denotes a total average vector of the input data x.

7. The apparatus according to claim 5, wherein said network accepts input data x as vector input, further includes a third generating means for forming the initial value W₁₀ of the weight coefficient of the intermediate layer from the in-class covariant matrix S_W and the inter-class covariant matrix S_B of the vector x, and said second generating means generates the bias θ

₁₀ of said intermediate layer by using the initial value of the weight coefficient of the intermediate layer formed by the third generating means.

8. A method for classifying input data representing a physical quantity into predetermined k classes, each class designating a set including a plurality of data which have a similar characteristic., by using a back propagation type neural network having an input layer, an output layer and an intermediate layer coupled between the input and output layers, each layer comprising a network of synapse elements and connections between the synapse elements, each connection being represented by a connection parameter which includes at least a weight coefficient and a bias θ

, the method comprising the steps of;

(a) accepting plural input data x as vectors;

(b) generating an average vector x_tot of all vectors x by performing a statistical processing on input data;

(c) forming an initial value θ

₁₀ of the intermediate layer from the average vector x_tot and the weight coefficient W₁₀ of the intermediate layer;

(d) determining said connection parameters beginning with the initial value θ

₁₀ according to a back propagation method;

(e) inputting input data to the input layer;

(f) processing the input data through the intermediate layer having the determined weight coefficients; and

(g) further processing the data processed through the intermediate layer by means of the output layer, and outputting from the output layer signals indicating one of the predetermined k classes to which the input data belongs.

9. The method according to claim 8, wherein said step (c) comprises generating a vector representing the bias θ

₁₀ of the intermediate layer on the basis of θ

₁₀ =W₁₀ x_tot wherein a total average vector of the input data x is denoted by x_tot.

10. The method according to claim 8, wherein said step (a) accepts the input data x as a vector, further including a step for generating an initial value W₁₀ of the weight coefficient of the intermediate layer from an in-class covariant matrix S_W and an inter-class covariant matrix S_B of vector x, and a step of generating the initial value θ

₁₀ of the bias of the intermediate layer from the initial value W₁₀ and the average vector x_tot.

11. An apparatus for classifying input data representing a physical quantity into predetermined k classes, each class designating a set including a plurality of data which have similar characteristics, said apparatus comprising:

a back propagation type neural network including an input layer, an output layer and an intermediate layer coupled between the input layer and the output layer; and

determining means for determining connection parameters between the layers of said neural network by using training data prior to classifying the input data, the connection parameters being represented by at least a weight coefficient,

wherein, the neural network input layer includes means for inputting data;

the neural network intermediate layer includes means for receiving data from the input layer and processing the received data by means of the connection parameters;

the neural network output layer includes means for receiving data processed by the intermediate layer, processing the received data by means of the connection parameters and outputting signals indicating one of the predetermined k classes; and

the determining means comprises generating means for generating data representing a weight coefficient matrix W₂₀ of the output layer from data representing an average vector x.sup.(k) of each class of input data of the input layer, an initial value W₁₀ of a weight coefficient matrix of the intermediate layer, and a bias θ

₁₀ representing an initial value of a bias of the intermediate layer as beginning values for the back propagation type neural network.

12. The apparatus according to claim 11, wherein said generating means comprises means for generating an average vector x.sup.(k) of each class k when the input data is input as a vector, class by class;

means for generating a weight coefficient vector W₂₀.sup.(k) of each class in accordance with W₂₀.sup.(k) =f(W₁₀ x.sup.(k) -θ

₁₀), where f is a transfer function; and

means for generating the weight coefficient matrix W₂₀ of the output layer by combining the weight coefficient vector W₂₀.sup.(k) of each class.

13. The apparatus according to claim 11, wherein said generating means further includes:

means for generating an in-class covariant matrix S_w and an inter-class covariant matrix S_B from said input vector x;

means for generating a mapping A such that the inter-class covariant matrix S_B becomes maximum while keeping the in-class covariant matrix constant;

means for generating said initial value W₁₀ by forming a transposed matrix of a matrix representing this mapping A, and

means for generating the bias θ

₁₀ of the intermediate layer from the total average value x_tot of the input data x and the weight coefficient W₁₀ of the intermediate layer.

14. A method for classifying input data representing a physical quantity into predetermined k classes, each class designating a set including a plurality of data which have a similar characteristic, by using a back propagation type neural network having an input layer, an output layer and an intermediate layer coupled between the input and output layers, each layer comprising a network of synapse elements and connections between the synapse elements, each connection being represented by a connection parameter which includes at least a weight coefficient and a bias θ

, the method comprising the steps of;

(a) partitioning input data x to classes as vectors;

(b) generating an average vector x.sup.(k) for each class of input data of the input layer, an initial value W₁₀ of a weight coefficient of the intermediate layer, and a vector θ

₁₀ representing an initial value of a bias of the intermediate layer by performing a statistical process on the input data;

(c) generating a weight coefficient matrix W₂₀ of the output layer from the average vector x.sup.(k) the initial value W₁₀ of a weight coefficient matrix of the intermediate layer, and the initial value vector θ

₁₀ of the bias of the intermediate layer;

(d) determining the connection parameters beginning with these initial values according to a back propagation method;

(e) inputting input data to the input layer;

(f) processing the input data through the intermediate layer having the determined weight coefficients; and

(g) further processing the data processed through the intermediate layer by means of the output layer, and outputting from the output layer signals indicating one of the predetermined k classes to which the input data belongs.

15. The method according to claim 14, wherein said step (c) further comprises the steps of

(c1) forming an average vector x.sup.(k) for each class when input data is input class k by class k as a vector

(c2) forming the weight coefficient vector W₂₀.sup.(k) of each class in accordance with

16. The method according to claim 14, wherein said step (b) further comprises the steps of:

(b1) forming an in-class covariant matrix S_W and an inter-class covariant matrix S_B from said input vector x;

(b2) forming a mapping A such that the inter-class covariant matrix S_B becomes maximum while keeping the in-class covariant matrix S_W constant and generating said initial value W₁₀ by calculating a transposed matrix of a matrix representing this mapping A; and

(b3) forming the bias θ

₁₀ of the intermediate layer from the total average value x_tot of input data x and the weight coefficient W₁₀ of the intermediate layer.

17. An apparatus for classifying input data representing a physical quantity into predetermined k classes, each class designating a set including a plurality of data which have similar characteristics, said apparatus comprising:

a back propagation type neural network including an input layer, an output layer, and an intermediate layer coupled between the input layer and the output layer; and

determining means for determining connection parameters between the layers of said neural network by using training data prior to classifying the input data, the connection parameters being represented at least by a bias,

wherein, the neural network input layer includes means for inputting data;

the neural network intermediate layer includes means for receiving data from the input layer and processing the received data by means of the connection parameters;

the neural network output layer includes means for receiving data processed by the intermediate layer, processing the received data by means of the connection parameters and outputting signals indicating one of the predetermined k classes; and

the determining means comprises means for generating a vector θ

₂₀ representing an initial value of a bias of the output layer from an initial value W₂₀ of a weight coefficient matrix of the output layer as a beginning value for the back propagation type neural network.

18. The apparatus according to claim 17, wherein said generating means comprises:

means for forming the average vector x.sup.(k) of each class when input data is input class k by class k as a vector;

means for calculating the weight coefficient vector W₂₀.sup.(k) of each class in accordance with W₂₀.sup.(k) =f(W₁₀ x.sup.(k) -θ

₁₀), from the average vector x.sup.(k) and said initial values W₁₀ and θ

₁₀ where f is a transfer function;

means for calculating the bias vector θ

₂₀.sup.(k) of each class in accordance with θ

₂₀.sup.(k) =α

∥

W₂₀.sup.(k) ∥

from said weight coefficient vector W₂₀.sup.(k) of each class and a predetermined constant α

; and

means for combining the bias vector θ

₂₀ (k) of each class to generate the bias matrix θ

₂₀ of the output layer.

19. The apparatus according to claim 17, wherein said generating means further comprises:

means for forming an in-class covariant matrix S_B and an inter-class covariant matrix from an input vector x;

means for forming a mapping A such that the inter-class covariant matrix S_B becomes maximum while keeping the in-class covariant matrix S_B constant; and

means for generating said initial value W₁₀ by calculating the transposed matrix of the mapping A;

means for calculating a bias θ

₁₀ of the intermediate layer from a total average value x_tot of input data x and the weight coefficient w₁₀ of the intermediate layer, and

means for forming the weight coefficient matrix W₂₀ of the output layer from the average vector x.sup.(k) of each class of input data of the input layer, an initial value W₁₀ of a weight coefficient matrix of the intermediate layer, and a vector θ

₁₀ representing the initial value of a bias of the intermediate layer.

20. A method for classifying input data representing a physical quantity into predetermined k classes, each class designating a set including a plurality of data which have a similar characteristic, by using a back propagation type neural network having an input layer, an output layer and an intermediate layer coupled between the input and output layers, each layer comprising a network of synapse elements and connections between the synapse elements, each connection being represented by a connection parameter which includes at least a weight coefficient and a bias θ

, the method comprising the steps of;

(a) partitioning input data x for classes as vectors;

(b) generating an initial value W₂₀ of a weight coefficient of the output layer by performing a statistical process on the input data;

(c) generating an initial value θ

₂₀ of a vector representing a bias of the output layer from the initial value W20 of the weight coefficient of the output layer;

(d) determining the connection parameters beginning with these initial values according to a back propagation method;

(e) inputting input data to the input layer;

(f) processing the input data through the intermediate layer having the determined weight coefficients; and

(g) further processing the data processed through the intermediate layer by means of the output layer, and outputting from the output layer signals indicating one of the predetermined k classes to which the input data belongs.

21. The method according to claim 20, wherein said step (c) further comprises the steps of:

(c1) forming an average vector x.sup.(k) of each class when input data is input class k by class k as a vector

(c2) forming a weight coefficient vector W₂₀.sup.(k) of each class in accordance with W20=f(W₁₀ x.sup.(k) -θ

₁₀) from the average vector x.sup.(k) and said initial values W₁₀ and θ

₁₀, where f is a transfer function;

(c3) forming a bias vector θ

₂₀ of each class in accordance with θ

₂₀.sup.(k) =α

∥

W₂₀.sup.(k) ∥

from said weight coefficient vector W₂₀ (k) of each class and a predetermined constant α

; and

(c4) combining the bias vector θ

₂₀ (k) of each class to calculate an initial value θ

₂₀ of a bias matrix of the output layer.

22. The method according to claim 20, wherein said step (b) further comprises the steps of:

(b1) forming an in-class covariant matrix S_W and an inter-class covariant matrix S_B from said input vector x;

(b2) forming a mapping A such that the inter-class covariant matrix covariant matrix SB^becomes maximum while keeping the in-class covariant matrix S_W constant;

(b3) forming an initial value W₁₀ by calculating a transposition matrix of a matrix representing the mapping A;

(b4) forming the bias θ

₁₀ of the intermediate layer from a total average value x_tot of input data x and the weight coefficient W₁₀ of the intermediate layer; and

(b5) forming the weight coefficient matrix W₂₀ of the output layer from the average vector x.sup.(k) of each of input data of the input layer, the initial value W₁₀ of a weight coefficient matrix of the intermediate layer, and the vector θ

₁₀ representing the initial value of a bias of the intermediate layer.