Method and apparatus for adaptive learning in neural networks
First Claim
1. A method of controlling a computer to adjust a neural network, said computer including an artificial neural network having multiple layers including an input layer and an output layer, each layer having a plurality of neurons, each neuron in a first layer is connected to each neuron in a second layer by a weight, said weights organized into a weight vector, comprising the steps of:
- assigning in a memory in said computer, a first memory location for storing a first learning rate, a second memory location for storing a momentum factor, a memory block for storing said weight vector, and a third memory location for storing a second learning rate;
selecting said first learning rate, said momentum factor, and said weight vector;
storing said first learning rate, said momentum factor, and said weight vector into said first memory location, said second memory location and said memory block;
saving said first learning rate in said second learning rate by storing said first learning rate into said third memory location;
obtaining the total error from said neural network using said weight vector;
using a search technique to adjust said first learning rate to adjust said weight vector;
updating said first memory location and said memory block;
adapting said momentum factor using said first learning rate and said second learning rate by computing the change in the learning rate by subtracting said first learning rate from said second learning rate and allowing said momentum factor to drift in a direction dependent upon said change in the learning rate;
meeting a predetermined convergence criterion; and
updating said weight vector to said artificial neural network.
0 Assignments
0 Petitions
Accused Products
Abstract
A method enabling a neural network, having weights organized into a weight vector, to learn, comprising the steps of: (a) assigning a first memory location for storing a first learning rate, a second memory location for storing a momentum factor, a memory block for storing the weight vector, and a third memory location for storing a second learning rate; (b) initializing the learning rate, momentum factor, and weight vector; (c) storing the first learning rate, the momentum factor, and the weight vector into their respective memory locations; (d) saving the first learning rate in the second learning rate by storing it into the third memory location; (e) using a search technique to adjust the first learning rate to adjust the weight vector, and updating the first memory location and the memory block; (f) adapting the momentum factor using the first learning rate and the second learning rate; and repeating steps (c) through (f) until a predetermined convergence criterion has been met.
75 Citations
10 Claims
-
1. A method of controlling a computer to adjust a neural network, said computer including an artificial neural network having multiple layers including an input layer and an output layer, each layer having a plurality of neurons, each neuron in a first layer is connected to each neuron in a second layer by a weight, said weights organized into a weight vector, comprising the steps of:
-
assigning in a memory in said computer, a first memory location for storing a first learning rate, a second memory location for storing a momentum factor, a memory block for storing said weight vector, and a third memory location for storing a second learning rate; selecting said first learning rate, said momentum factor, and said weight vector; storing said first learning rate, said momentum factor, and said weight vector into said first memory location, said second memory location and said memory block; saving said first learning rate in said second learning rate by storing said first learning rate into said third memory location; obtaining the total error from said neural network using said weight vector; using a search technique to adjust said first learning rate to adjust said weight vector; updating said first memory location and said memory block; adapting said momentum factor using said first learning rate and said second learning rate by computing the change in the learning rate by subtracting said first learning rate from said second learning rate and allowing said momentum factor to drift in a direction dependent upon said change in the learning rate; meeting a predetermined convergence criterion; and updating said weight vector to said artificial neural network. - View Dependent Claims (2, 3, 4, 5)
-
-
6. A method of controlling a computer to adjust a neural network, said computer including an artificial neural network having multiple layers including an input layer and an output layer, each layer having a plurality of neurons, each neuron in a first layer is connected to each neuron in a second layer by a weight, said weights organized into a weight vector, comprising the steps of:
-
assigning in said computer a first memory location for storing a first learning rate, a second memory location for storing a momentum factor, a memory block for storing said weight vector, a third memory location for storing a second learning rate; selecting said first learning rate, momentum factor, and said weight vector; storing said first learning rate, said momentum factor, and said weight vector into their respective memory locations assigned in said assigning step; utilizing an error function and a set of test patterns to compute a total error for said neural network using said weight vector; designating said first learning rate to said second learning rate by storing said first learning rate into said third memory location; using a search technique to adjust said first learning rate as a function of said total error by allowing said momentum factor to drift in a direction dependent upon said first learning rate and said second learning rate; reversing said direction if said momentum factor is equal to an upper limit or a lower limit; adapting said momentum factor based upon said second learning rate and said first learning rate; utilizing said first learning rate and said momentum factor to adjust said weight vector; meeting a predetermined convergence criterion; and updating said weight vector to said artificial neural network. - View Dependent Claims (7, 8, 9)
-
-
10. An improved neural network adjusted to match a known set of test patterns comprising:
-
a neural network having multiple layers including an input layer and an output layer, each layer having a plurality of neurons, each of said plurality of neurons in a first layer connected to each of said plurality of neurons in a second layer by a weight, said weights organized into a weight vector; a memory storing a set of signals associated with said known set of test patterns, a first learning rate, a second learning rate, a momentum factor, and said weight vector; and processor means coupled to said neural network and said memory for comparing a set of signals from said output layer to said set of signals associated with said known set of test patterns to determine errors;
said processor means if said errors exceed a predetermined error convergence criteria replacing in said memory said second learning rate with said first learning rate, readjusting said first learning rate and adjusting said momentum factor by a predetermined incremental value and multiplying said predetermined incremental value by a negative value if said first learning rate is increasing or if said momentum factor is equal to an upper limit or a lower limit, and readjusting said weight vector stored in said memory using the readjusted first learning rate and the adjusted momentum factor.
-
Specification