Method of and system for controlling learning in neural network

• US 5,313,559 A
• Filed: 02/10/1992
• Issued: 05/17/1994
• Est. Priority Date: 02/15/1991
• Status: Expired due to Fees

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Abe, Masahiro, Ogata, Hisao, Higashino, Junichi, Sakou, Hiroshi
• Primary Examiner(s): Fleming, Michael R.
• Assistant Examiner(s): Downs, Robert W.
• Application Number:
  US07/833,127
• Time in Patent Office: 827 Days
• Field of Search: 395/23, 395/76
• US Class Current: 706/25
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

A learning control method reduces overall learning time by displaying data related to an appropriate determination of learning protraction and a proper restoring method. Prior to initiating the learning, the user is inquired about the current problem and a problem data set representing items associated with the problem is obtained. Evaluation data indicating a state of learning obtained during the learning on the current problem is sequentially stored and displayed. When there is a high possibility of learning protraction during the learning, a message informing the user is displayed. When the learning is stopped by the user in this case, the problem data set and evaluation data set are stored. Then, a list of restoring methods is displayed and a particular restoring method is selected by the user once the learning is stopped. The learning is restarted on the current problem in accordance with the selected restoring method.

Claims

1. A method of managing learning in a neural network, comprising the steps of:

providing reference data representing criteria of protraction of the learning;

performing learning on a current problem by use of learning pattern data for learning cycles in accordance with a predetermined learning method while generating and updating evaluation data representing a learning state of the neural network for each learning cycle;

comparing the evaluation data with the reference data for each learning cycle;

judging in accordance with the compared result and predetermined judging conditions whether or not there exists a possibility of protraction of the learning;

displaying, when it is judged that there exists the possibility of the learning protraction, a list of learning methods for a user;

selectively displaying past evaluation data for each of past problems which is analogous to the current problem, a past learning data set for each of the past problems, and a learning method for each of the past problems employed when it was judged that there existed a possibility of learning protraction in the learning of each of the past problems;

selecting a new learning method from the list by a user based on the displayed past evaluation data, past learning data set and learning method, for each of the past problems, respectively; and

re-initiating the learning in accordance with the new learning method selected by a user.

2. A method according to claim 1, further comprising the steps of:

prior to the initiation of the learning, inputting the current problem, a learning method and a problem data set representing items associated with the problem; and

determining a structure of a neural network and initial weights for neurons of the neural network based on the problem and the problem data set and initializing the neural network by giving the neurons the weights to perform the learning.

3. A method according to claim 2, wherein said problem data set includes data representing a kind of the learning pattern data and characteristics thereof, and including:

data representing dimensions of input signal pattern data and teacher signal pattern data of the learning pattern data,

data representing the number of patterns in the learning pattern data,

data representing the number of patterns of the input signal pattern data paired with patterns of the teacher signal pattern data,

data representing a mean value of values of elements in each dimension of the input signal pattern data and the teacher signal pattern data,

data representing a standard deviation for the values of the elements in the dimension of the input signal pattern data and the teacher signal pattern data,

data representing that the learning pattern data is either binary data or continuous data, and

date representing whether or not the learning pattern data includes noise.

4. A method according to claim 1, wherein said comparing step further includes the step of displaying the evaluation data.

5. A method according to claim 1, wherein said evaluation data includes:

data representing an error of an output from the neural network,

data representing a means value of norms of steepest descent gradient vector for the respective weights of synapses in the neural network,

data representing a mean value of norms of weight vectors for each neuron, and

data representing a mean value of derivative values of a Sigmoid function over (number of patterns of the learning pattern data)×

(number of neurons).

6. A method according to claim 1, wherein the predetermined judging conditions include:

a condition that an output error from the neural network is greater than a first reference value;

a condition that a mean value of norms of steepest descent gradient vectors for respective weights of synapses in the neural network is smaller than a second reference value;

a condition that a mean value of norms of weight vectors for each neuron is greater than a third reference value; and

a mean value of derivative values of a Sigmoid function of each neuron over (number of patterns of learning pattern data)×

(number of neurons) being greater than a fourth reference value.

7. A method according to claim 6, wherein said judging step includes the step of judging that there exists the possibility of the learning protraction when at least one of the predetermined judging conditions is satisfied.

8. A method according to claim 1, wherein said step of displaying a list of learning methods further comprises the steps of:

stopping the learning for selection by the user of whether the learning is to be continued or stopped, when it is judged that there exists the possibility of the learning protraction;

notifying the user that there is the possibility of the learning protraction when it is judged that there is the possibility of the learning protraction; and

displaying the list of learning methods for selection of the new learning method by the user when the stopping of the learning is selected by the user.

9. A method according to claim 1, wherein the list of learning methods includes at least one of the methods of:

initializing the weights to re-initiate the learning;

reducing the norms of the weight vectors;

reducing the norms of the weight vectors and adding a small amount of noise to the weights;

reducing the norm of a portion of the weight vectors;

reducing the norm of a portion of the weight vectors and adding a small amount of noise to the weights;

fixing each of the norms for the weight vectors to one value for each layer and repeatedly performing the learning for a predetermined number of times; and

increasing the number of neurons of an intermediate layer or varying learning coefficients or momentum coefficients.

10. A method according to claim 1, further comprising the step of storing the new learning method and the evaluation data if the new learning method is selected by the user.

11. A method according to claim 10, further comprising the step of storing, in addition to the new learning method and the evaluation data, a particular identifier of the evaluation data when the re-initialed learning is completed.

12. A system for controlling learning in a neural network, comprising:

first storing means for storing a learning pattern data set;

second storing means for storing a reference data set;

third storing means for storing an evaluation data set;

fourth storing means for storing a list of restoring methods;

display means for displaying data inputted thereto;

learning means for reading out a current learning pattern data of the learning pattern data set for a current problem from said first storing means and performing learning of the current problem in the neural network according to the read out learning pattern data;

protraction determining means for reading a current reference data of the reference data set from said second storing means, storing, as an element of the evaluation data set in said third storing means, current evaluation data indicative of a learning state of a neural network and obtained during the learning, and determining whether or not there exists a possibility of protraction of the learning, in accordance with the read out current reference data, the obtained evaluation data and a predetermined judging condition set;

selection auxiliary means for respectively reading the current evaluation data and the list of restoring methods from said third and fourth storing means when it is determined that there exists the possibility of the learning protraction and outputting the read out current evaluation data and the read out list of restoring methods to said display means for selection of a particular restoring method by a user;

learning control means for controlling said learning means to perform the learning of the current problem in accordance with the particular restoring method designated by the user, when it is determined that there exists the possibility of the learning protraction; and

fifth storing means for storing a case data set indicating past cases, the case data set including a particular restoring method adopted for a past learning that encountered a problem similar to the current problem, and a past evaluation data set employed in the past learning, wherein said selection auxiliary means further comprises means for storing in said fifth storing means, as new case data of the case data set, the selected restoring method and the evaluation data set.

13. A method of controlling learning on a current problem in a neural network, comprising the steps of:

initiating the learning of the current problem in accordance with learning pattern data;

judging whether or not there exists a possibility of protraction of the learning, in accordance with current evaluation data obtained during the learning, a reference data set, and a predetermined judging condition set, each current evaluation data representing a state of the learning in the neural network;

displaying a selection auxiliary data and a list of restoring methods for the user to select the particular storing method when it is judged that there exists the possibility of the learning protraction;

re-initiating the learning in accordance with the particular restoring method selected by the user;

wherein the step of displaying includes displaying past evaluation data for each of past problems which is analogous to the current problem, a past learning data set for each of the past problems, and a new learning method for each of the past problems employed when it was judged that there existed a possibility of the learning protraction in learning each of the past problems, respectively.

14. A method according to claim 13, further comprising the step of adding as new case data the particular restoring method and the obtained evaluation data set to the selection auxiliary data.

15. A method according to claim 14, further comprising the step of adding a particular identifier indicating success of the learning to the new case data.