Optimized artificial neural networks

• US 6,601,053 B1
• Filed: 05/25/2000
• Issued: 07/29/2003
• Est. Priority Date: 05/19/1989
• Status: Expired due to Fees

• Current Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Original Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Inventors: Caruana, Richard A., Schaffer, J. David, Eshelman, Larry J.
• Primary Examiner(s): STARKS, WILBERT L
• Application Number:
  US09/578,428
• Time in Patent Office: 1,160 Days
• Field of Search: 706/13, 706/26, 706/15, 706/20, 706/4
• US Class Current: 706/26
• CPC Class Codes: G06N 3/086 using evolutionary algorithms, e.g. genetic algorithms or genetic programming

Abstract

Neural network architectures are represented by symbol strings. An initial population of networks is trained and evaluated. The strings representing the fittest networks are modified according to a genetic algorithm and the process is repeated until an optimized network is produced.

Claims

1. A method of producing optimized artificial neural networks, comprising:

providing a plurality of different neural network input-output sets including a plurality of training sets and an evaluation set;

providing a population of multiple symbol strings, the symbols of the strings representing parameters of neural network architecture and parameters of a learning method;

providing artificial neural networks for respective symbol strings of the population, based on the neural network architecture parameters;

training the artificial neural networks using the learning method depending on the training sets and learning method parameters;

evaluating the trained neural networks using the evaluation set to generate evaluations of the trained neural networks;

generating a further population of multiple symbol strings from the population of symbol strings using a genetic algorithm depending on the evaluations;

continuing the training, evaluation, and generating until a stopping criteria is met; and

selecting one of the symbol strings for producing the optimized artificial neural network depending on the evaluations.

2. The method of claim 1 in which:

the generating includes assigning a fitness value to each of the plurality of symbol strings depending on the evaluations; and

the genetic algorithm generates symbol strings depending on the fitness values.

3. The method of claim 1, wherein the neural networks include a layer of input neurons, a layer of output neurons, one or more hidden layers of neurons between the input layer and the output layer, a plurality of signal transmission paths interconnecting the neurons, the neurons multiplying signals received from the transmission paths by a weight for each path, the weight being adjusted during the training.

4. The method of claim 3, in which the neural network architectural parameters include:

the number of layers of hidden neurons; and

the number of neurons within each hidden layer of the network.

5. The method of claim 3, in which the network is a feed forward, wherein signals between neurons are only propagated in a direction toward the output layer;

the learning method includes back propagation; and

the learning method parameters include;

learning rate, momentum, and initial path weight information.

6. The method of claim 5 wherein the training includes applying the inputs of a training set to a neural network;

comparing the resulting output from the neural network to the outputs of the training set; and

updating link weights within the neural network depending on results of the comparing.

7. The method of claim 1 in which the genetic algorithm selects less than all the symbol strings of the population for use in generating the further population, the selection depending on results of the evaluating.

8. The method of claim 1 wherein the genetic algorithm copies some of the symbols from one symbol string to another symbol string to generate the further population.

9. The method of claim 1 in which the genetic algorithm randomly changes values of symbols in the symbol strings of the population to generate the further population.

10. A neural network produced by the method of claim 1.

11. The network of claim 10 in which:

the generating includes assigning a fitness value to each of the plurality of symbol strings depending on the evaluations; and

the genetic algorithm generates symbol strings depending on the fitness values;

the neural networks include a layer of input neurons, a layer of output neurons, one or more hidden layers of neurons between the input layer and the output layer, a plurality of signal transmission paths interconnecting the neurons, the neurons multiplying signals received from the transmission paths by a weight for each path, the weight being adjusted during the training;

the neural network architectural parameters include;

the number of layers of hidden neurons; and

the number of neurons within each hidden layer of the network;

the network is a feed foreword, wherein signals between neurons are only propagated in a direction toward the output layer;

the learning method includes back propagation; and

the learning method parameters include, learning rate, momentum, and initial path weight information;

the genetic algorithm selects less than all the symbol strings of the population for use in generating the further population, the selection depending on results of the evaluating;

the genetic algorithm copies some of the symbols from one symbol string to another symbol string to generate the further population;

the genetic algorithm randomly changes values of symbols in the symbol strings of the population to generate the further population; and

the training includes applying the inputs of a training set to a neural network;

comparing the resulting output from the neural network to the outputs of the training set; and

updating link weights within the neural network depending on results of the comparing.

12. The method of claim 1 in which the symbols of the symbol strings include information representing parameters of a supervised learning method.

13. The method of claim 1 in which the symbols of the symbol strings represent parameters of a back propagation type learning method.

14. The method of claim 13 in which learning parameters include information for determining a learning rate.

15. The method of claim 13 in which learning parameters include information for determining a learning momentum.

16. The method of claim 13 in which learning parameters include information for determining initial path weights.

17. The method of claim 1 in which:

the generating includes assigning a fitness value to each of the plurality of symbol strings depending on the evaluations; and

the genetic algorithm generates symbol strings depending on the fitness values;

the neural networks include a layer of input neurons, a layer of output neurons, one or more hidden layers of neurons between the input layer and the output layer, a plurality of signal transmission paths interconnecting the neurons, the neurons multiplying signals received from the transmission paths by a weight for each path, the weight being adjusted during the training;

the neural network architectural parameters include;

the number of layers of hidden neurons; and

the number of neurons within each hidden layer of the network;

the network is a feed foreword, wherein signals between neurons are only propagated in a direction toward the output layer;

the learning method includes back propagation; and

the learning method parameters include, learning rate, momentum, and initial path weight information;

the genetic algorithm selects less than all the symbol strings of the population for use in generating the further population, the selection depending on results of the evaluating;

the genetic algorithm copies some of the symbols from one symbol string to another symbol string to generate the further population;

the genetic algorithm randomly changes values of symbols in the symbol strings of the population to generate the further population; and

the training includes applying the inputs of a training set to a neural network;

comparing the resulting output from the neural network to the outputs of the training set; and

updating link weights within the neural network depending on results of the comparing.

18. A system for producing optimized artificial neural networks, comprising:

means for providing a plurality of different neural network input-output sets including a plurality of training sets and an evaluation set;

means for providing an initial population of multiple symbol strings, the symbols of the strings representing parameters of neural network architecture and parameters of a learning method;

means for providing artificial neural networks for respective symbol strings of the population, based on the neural network architecture parameters;

training the artificial neural networks using the learning method depending on the training sets and learning method parameters;

means for evaluating the trained neural networks using the evaluation set to generate evaluations of the trained neural networks;

means for generating a further population of multiple symbol strings from the population of symbol strings using a genetic algorithm depending on the evaluations; and

means for selecting one of the symbol strings for producing the optimized artificial neural network depending on the evaluations.

19. The system of claim 18 in which:

the generating includes assigning a fitness value to each of the plurality of symbol strings depending on the evaluations; and

the genetic algorithm generates symbol strings depending on the fitness values;

the neural networks include a layer of input neurons, a layer of output neurons, one or more hidden layers of neurons between the input layer and the output layer, a plurality of signal transmission paths interconnecting the neurons, the neurons multiplying signals received from the transmission paths by a weight for each path, the weight being adjusted during the training;

the neural network architectural parameters include;

the number of layers of hidden neurons; and

the number of neurons within each hidden layer of the network;

the network is a feed foreword, wherein signals between neurons are only propagated in a direction toward the output layer;

the learning method includes back propagation; and

the learning method parameters include, learning rate, momentum, and initial path weight information;

the genetic algorithm selects less than all the symbol strings of the population for use in generating the further population, the selection depending on results of the evaluating;

the genetic algorithm copies some of the symbols from one symbol string to another symbol string to generate the further population;

the genetic algorithm randomly changes values of symbols in the symbol strings of the population to generate the further population; and

the training includes applying the inputs of a training set to a neural network;

comparing the resulting output from the neural network to the outputs of the training set; and

updating link weights within the neural network depending on results of the comparing.

20. A method for selecting an architecture of an artificial neural network, the method comprising the steps of

a) choosing a population of symbol strings, each symbol string representing a respective neural network architecture including a plurality of neurons, which population has at least two members;

b) for each symbol string,

i) providing a set of neural network inputs including a training set and an evaluation set, which evaluation set is not identical to the training set;

ii) training the respective neural network using the training set of inputs;

iii) evaluating the trained neural network using the evaluation set of inputs;

c) modifying the population according to a true genetic algorithm, based on results of the evaluating step, and repeating b), until a stopping criterion is reached;

d) choosing a symbol string representing the selected architecture from the population after the stopping criterion is reached;

further comprising the step of selecting the fittest, from among a plurality of networks which have undergone a training and evaluation step, according to a defined criterion;

wherein the step of modifying is carried out on the symbol strings representing the selected fittest networks;

wherein the symbol string includes a representation of the number of layers of hidden neurons, and the number of neurons within each layer of the network; and

wherein the training step is carried out by back propagation and the symbol string further represents back propagation parameters of learning rate, momentum and dispersion of initial link weights.

21. A method for selecting a detailed architecture of an artificial neural network, the method comprising the steps of

a) choosing a population of symbol strings, each symbol string representing a respective neural network architecture including a plurality of neurons, which population has at least two members, wherein each symbol string includes a representation of a number of neurons and how the neurons are connected,

b) for each symbol string,

i) providing a set of neural network inputs including a training set and an evaluation set, which evaluation set is not identical to the training set,

ii) training the neural network using the training set of inputs,

iii) evaluating the trained neural network using the evaluation set of inputs,

c) modifying the population according to a true genetic algorithm, based on results of the evaluating step, and repeating b) until a stopping criterion is reached, and

d) choosing a symbol string representing the selected architecture from among the population after the stopping criterion is reached,

wherein the symbol string includes a representation of the number of layers of hidden neurons, and the number of neurons within each hidden layer of the network, and

wherein the training step is carried out by back propagation, and the symbol string further represents back propagation parameters of learning rate, momentum, and dispersion of initial link weights.