Multi-layer network and learning method therefor

• US 5,212,767 A
• Filed: 12/10/1990
• Issued: 05/18/1993
• Est. Priority Date: 12/15/1989
• Status: Expired due to Fees

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Matsushima, Hitoshi, Higashino, Junichi
• Primary Examiner(s): MacDonald, Allen R.
• Application Number:
  US07/625,166
• Time in Patent Office: 890 Days
• Field of Search: 364/513, 364/807, 382/15, 307/201, 395/24, 395/26, 395/27, 395/800, 395/11
• US Class Current: 706/25
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: G06N 3/10 Interfaces, programming languages or software development kits, e.g. for simulating neural networks

Abstract

A multi-layer neural network comprising an input layer, a hidden layer and an output layer and a learning method for such a network are disclosed. A processor belonging to the hidden layer stores both the factors of multiplication or weights of link for a successive layer nearer to the input layer and the factors of multiplication or weights of link for a preceding layer nearer to the output layer. Namely, the weight for a certain connection is doubly stored in processors which are at opposite ends of that connection. Upon forward calculation, the access to the weights for the successive layer among the weights stored in the processors of the hidden layer can be made by the processors independently from each other. Similarly, upon backward calculation, the access to weights for the preceding layer can be made by the processors independently from each other.

Claims

1. A multi-layer network comprising:

an input layer including input means for inputting a plurality of data;

a hidden layer connected to the input means of said input layer and including a plurality of data processors, each of which having means for storing factors of multiplication from input layer to hidden layer corresponding to connections with the input means of said input layer and having means for calculating based on said factors of multiplication from input layer to hidden layer a sum of products on a plurality of data delivered from the input means of said input layer; and

an output layer connected to the plurality of data processors of said hidden layer and including a plurality of data processors, each of which having means for storing factors of multiplication from hidden layer to output layer corresponding to connections with the plurality of data processors of said hidden layer and having means for calculating based on said forces of multiplication a sum of products on a plurality of data delivered from the plurality of data processors of said hidden layer,

said each data processor of said hidden layer further having means for storing factors of multiplication from output layer to hidden layer corresponding to connections with the plurality of data processors of said output layer, and said calculating means performing calculation of a sum of products based on said factors of multiplication from output layer to hidden layer on a plurality of data delivered from the plurality of data processors of said output layer.

2. A multi-layer network according to claim 1, wherein each one of the plurality of data processors of at least one of said hidden layer and said output layer is provided in correspondence to one of said nodes, said nodes being units which constitute the multi-layer network.

3. A multi-layer network according to claim 1, wherein each one of the plurality of data processors of at least one of said hidden layer and said output layer is provided in correspondence to a set of plural ones of said nodes, said nodes being units which constitute the multi-layer network.

4. A multi-layer network according to claim 1, wherein a data processing means belonging to said hidden layer across a content of said storing means with the contents being divided among a plurality of data processing means.

5. A multi-layer network according to claim 1, wherein said storing means stores a part of the factors of multiplication stored in the storing means which are respectively included in each one of the plurality of data processors belonging to successive layers.

6. A multi-layer network according to claim 1, wherein

said hidden layer comprises at least two layers including a first hidden layer located closer to said input layer and a second hidden layer located closer to said output layer,

said first hidden layer includes a plurality of data processors, and said second hidden layer includes a plurality of data processors,

each of the data processors of said first hidden layer has means for storing factors of multiplication from input layer to first hidden layer corresponding to connections with the input means of said input layer, means for storing factors of multiplication from second hidden layer to first hidden layer corresponding to connections with the plurality of data processors of said second hidden layer, and means for calculating a sum of products on a plurality of data delivered from the input means of said input layer based on said factors of multiplication from input layer to first hidden layer, or on a plurality of data delivered from the plurality of data processors of said second hidden layer based on said factors of multiplication from second hidden layer to first hidden layer, and

each of the data processors of said second hidden layer has means for storing factors of multiplication from first hidden layer to second hidden layer corresponding to connections with the plurality of data processors of said first hidden layer, means for storing factors of multiplication from output layer to second hidden layer corresponding to connections with the plurality of data processors of said output layer, and means for calculating a sum of products on a plurality of data delivered from said plurality of data processors of said first hidden layer based on said factors of multiplication from first hidden layer to second hidden layer, or on a plurality of data delivered from said plurality of data processor of said output layer based on said factors of multiplication from output layer to second hidden layer.

7. A multi-layer network according to claim 1, wherein

said hidden layer comprises at least three layers including a first hidden layer located closer to said input layer, a second hidden layer located closer to said output layer, and a third hidden layer other than said first and second hidden layers,

said first hidden layer includes a plurality of data processors, said second hidden layer includes a plurality of data processors, and said third hidden layer includes a plurality of data processors,

each of the data processors of said first hidden layer has means for storing factors of multiplication from input layer to first hidden layer corresponding to connections with the input means of said input layer, means for storing factors of multiplication from third hidden layer to first hidden layer corresponding to connections with the plurality of data processors of said third hidden layer, and means for calculating a sum of products on a plurality of data delivered from the input means of said input layer based on said factors of multiplication from input layer to first hidden layer, or on a plurality of data delivered from the plurality of data processors of said third hidden layer based on said factors of multiplication from third hidden layer to first hidden layer,

each of the data processors of said second hidden layer has means for storing factors of multiplication from third hidden layer to second hidden layer corresponding to connections with the plurality of data processors of said third hidden layer, means for storing factors of multiplication from output layer to second hidden layer corresponding to connections with the plurality of data processors of said output layer, and means for calculating a sum of products on a plurality of data delivered from the plurality of data processors of said third hidden layer based on said factors of multiplication from third hidden layer to second hidden layer, or on a plurality of data delivered from the plurality of data processors of said output layer based on said factors of multiplication from output layer to second hidden layer, and

each of the data processors of said third hidden layer has means for storing factors of multiplication from first hidden layer to third hidden layer corresponding to connections with the plurality of data processors of said first hidden layer, means for storing factors of multiplication from second hidden layer to third hidden layer corresponding to connections with the plurality of data processors of said second hidden layer, and means for calculating a sum of products on the plurality of data delivered from the plurality of data processors of said first hidden layer based on said factors of multiplication from first hidden layer to third hidden layer, or on a plurality of data delivered from the plurality of data processors of said second hidden layer based on said factors of multiplication from second hidden layer to third hidden layer.

8. A multi-layer network according to claim 1, wherein each of the data processors of said hidden layer includes means for storing factors of multiplication corresponding to connections of another of the data processors with the input means of said input layer, and means for storing factors of multiplication corresponding to connections of said another of the data processors with the plurality of data processors of said output layer.

9. A multi-layer network according to claim 1, further comprising

connecting means for connecting among said input layer, said hidden layer, and said output layer, and exchanging means for changing over connections by said connecting means.

10. A multi-layer network according to claim 9, wherein said connecting means includes a plurality of buses.

11. A learning method performed by using a multi-layer network, said multi-layer network comprising:

an input layer including an input means for inputting a plurality of data;

a hidden layer connected to the input means o said input layer and including a plurality of data processors, each of which having means for storing factors of multiplication from input layer to hidden layer corresponding to connections with the input means of the input layer, means for storing factors of multiplication from output layer to hidden layer corresponding to connections with data processors of an output layer, and means for calculating a sum of products on a plurality of data delivered from the input means of said input layer based on said factors of multiplication from input layer to hidden layer, or on a plurality of data delivered from the data processors of said output layer based on said factors of multiplication from output layer to hidden layer; and

said output layer connected to the plurality of data processors of said hidden layer and including the plurality of data processors, each of which having means for storing factors of multiplication from hidden layer to output layer corresponding to connections with the plurality of data processors of said hidden layer, and means for calculating based on said factors of multiplication from hidden layer to output layer a sum of products on a plurality of data delivered from the plurality of data processors of said hidden layer;

said learning method comprising the steps of;

setting an initial value of each of said factors of multiplication from input layer to hidden layer and said factors of multiplication from output layer to hidden layer in said plurality of data processors in said hidden layer, and said factors of multiplication from hidden layer to output layer in said plurality of data processors in said output layer,

inputting a plurality of data to the input means of said input layer and calculating a sum of products in each of the plurality of data processors of said hidden layer and in each of the plurality of data processor of said output layer,

adjusting said factors of multiplication from hidden layer to output layer, said factors of multiplication from output layer to hidden layer, and said factors of multiplication from input layer to hidden layer on the basis of differences between a plurality of data obtained from the plurality of data processors of said output layer as a result of said calculation of the sum of products and supervised data representing desired output data,

repeating the adjustment of said factors of multiplication from hidden layer to output layer, said factors of multiplication from output layer to hidden layer, and said factors of multiplication from input layer to hidden layer until the plurality of data obtained from the plurality of data processors of said output layer are approximately equal to the supervised data,

adjusting the factors of multiplication from output layer to hidden layer in the plurality of data processors of said hidden layer and said factors of multiplication from hidden layer to output layer in the plurality of data processors of said output layer, so that the first mentioned factors of multiplication and the second mentioned factors of multiplication assume identical values.

12. A multi-layer network comprising:

an input layer including input means for inputting a plurality of data;

a first hidden layer connected to the input means of said input layer and including a plurality of data processors, each of which having means for storing factors of multiplication from input layer to first hidden layer corresponding to connections with the input means of said input layer, and means for calculating based on said factors of multiplication from input layer to first hidden layer a sum of products on a plurality of data delivered from the input means of said input layer;

a second hidden layer connected to the plurality of data processors of said first hidden layer, or to a plurality of data processors of a hidden layer other than said first hidden layer, said second hidden layer including a plurality of data processors, each of which having means for storing factors of multiplication from first hidden layer to second hidden layer corresponding to connections with the plurality of data processors of said first hidden layer, or with the plurality of data processors of the hidden layer other than said first hidden layer, and means for calculating a sum of products on a plurality of data delivered from the plurality of processors of said first hidden layer based on said factors of multiplication from first hidden layer to second hidden layer, or on a plurality of data delivered from the plurality of data processors of said another hidden layer based on said factors of multiplication form another hidden layer to second hidden layer; and

an output layer connected to the plurality of data processors of said second hidden layer and including a plurality of data processors, each of which having means for storing factors of multiplication from second hidden layer to output layer corresponding to connections with the plurality of data processors of said second hidden layer, and means for calculating based on said factors of multiplication from second hidden layer to output layer a sum of products on a plurality of data delivered from the plurality of data processors of said second hidden layer, wherein said each of the plurality of data processors of said first hidden layer further comprises means for storing factors of multiplication from second hidden layer to first hidden layer corresponding to connections with the plurality of data processors of said second hidden layer, or storing factors of multiplication from another hidden layer to first hidden layer corresponding to connections with the plurality of data processors of said another hidden layer, and said calculating means performs calculation of a sum of products on a plurality of data delivered from the plurality of data processors of said second hidden layer based on said factors of multiplication from second hidden layer to first hidden layer, or from the plurality of data processors of said another hidden layer based on said factors of multiplication from another hidden layer to first hidden layer, and

said each of the plurality of data processors of said second hidden layer further comprises means for storing factors of multiplication from output layer to second hidden layer corresponding to connections with the plurality of data processors of said output layer, and said calculating means performs calculation, based on said stored factors of multiplication from output layer to second hidden layer, of a sum of products on a plurality of data delivered from the plurality of data processors of said output layer.