Recurrent neural network with variable size intermediate layer

• US 5,129,039 A
• Filed: 07/10/1991
• Issued: 07/07/1992
• Est. Priority Date: 09/17/1988
• Status: Expired due to Term

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Hiraiwa, Atsunobu
• Primary Examiner(s): MacDonald, Allen R.
• Application Number:
  US07/727,871
• Time in Patent Office: 363 Days
• Field of Search: 364/513, 395/22, 395/23, 395/24
• US Class Current: 706/25
• CPC Class Codes: G06N 3/082 modifying the architecture, e.g. adding, deleting or silencing nodes or connections
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent

Abstract

The present invention is concerned with a signal processing system having a learning function pursuant to the back-propagation learning rule by the neural network, in which the learning rate is dynamically changed as a function of input values to effect high-speed stable learning. The signal processing system of the present invention is so arranged that, by executing signal processing for the input signals by the recurrent network formed by units each corresponding to a neuron, the features of the sequential time series pattern such as voice signals fluctuating on the time axis can be extracted through learning the coupling state of the recurrent network. The present invention is also concerned with a learning processing system adapted to cause the signal processing section formed by a neural network to undergo signal processing pursuant to the back-propagation learning rule, wherein the local minimum state in the course of the learning processing may be avoided by learning the coefficient of coupling strength while simultaneously increasing the number of the unit of the intermediate layer.

Claims

1. A learning processing system comprising

a signal processing section including at least an input layer, an intermediate layer and an output layer, each of said layers being made up of a plurality of signal processing units, and

a learning processing section for repeatedly and sequentially computing, from said output layer towards said input layer, a coefficient W_ji of coupling strength between said output layer and said intermediate layer and said intermediate layer and said input layer of said signal processing section on the basis of error data δ

_ji between an output value of said output layer for input signal patterns entered into said input layer and a predetermined output value denoted as a teacher signal, thereby performing learning processing of said coefficient of coupling strength, wherein control means are provided in said learning processing section for increasing the number of said signal processing units of said intermediate layer, and wherein said learning processing section performs learning processing of said coefficient of coupling strength as said learning processing section causes the number of said signal processing units of said intermediate layer to be increased.

2. The learning processing system according to claim 1 wherein said signal processing section is a neural network of multi-layer structure including said input layer, said intermediate layer and said output layer, each of said layers being constituted by said plurality of signal processing units for performing signal processing corresponding to that of a neuron.

3. A signal processing system comprising

a signal processing section including at least an input layer, an intermediate layer and an output layer, each of said layers being made up of a plurality of signal processing units, and

a learning processing section for repeatedly and sequentially computing, from said output layer towards said input layer, a coefficient of coupling strength between said output layer and said intermediate layer and said intermediate layer and said input layer of said signal processing section on the basis of error data between an output value of said output layer for input signal patterns entered into said input layer and a predetermined output value denoted as a teacher signal, thereby performing learning processing of said coefficient of coupling strength, wherein

each of said signal processing units of said intermediate layer and of said output layer is provided with delay means and wherein said signal processing section is arranged as a recurrent network including loop and feedback circuitry, said loop circuitry being arranged so that an output of said intermediate layer is its own input after passing through said delay means and an output of said output layer is its own input after passing through said delay means, and said feedback circuitry is arranged so that an output of said output layer is fed to an input of said intermediate layer.

4. The signal processing system according to claim 3 wherein said signal processing section is a neural network of a multi-layer structure including said input layer, said intermediate layer and said output layer, each of said layers being constituted by said plurality of signal processing units each performing signal processing corresponding to that of a neuron.

5. The signal processing system according to claim 3 wherein control means are provided in said learning processing section for increasing the number of said signal processing units of said intermediate layer, and wherein said learning processing section performs said learning processing of said coefficient of coupling strength as said learning processing section causes the number of said signal processing units of said intermediate layer to be increased.