Method for Training Neural Networks

• US 20080281767A1
• Filed: 11/15/2006
• Published: 11/13/2008
• Est. Priority Date: 11/15/2005
• Status: Active Grant

• Current Assignee: Bernadette Garner
• Original Assignee: Bernadette Garner
• Inventors: Garner, Bernadette
• Granted Patent: US 8,862,527 B2
• Time in Patent Office: Days
• US Class Current: 706/25
• CPC Class Codes: G06N 3/048 Activation functions
• CPC Class Codes: G06N 3/082 modifying the architecture, e.g. adding, deleting or silencing nodes or connections
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning

Abstract

The present invention provides a method (30) for training an artificial neural network (NN). The method (30) includes the steps of: initialising the NN by selecting an output of the NN to be trained and connecting an output neuron of the NN to input neuron(s) in an input layer of the NN for the selected output; preparing a data set to be learnt by the NN; and, applying the prepared data set to the NN to be learnt by applying an input vector of the prepared data set to the first hidden layer of the NN, or the output layer of the NN if the NN has no hidden layer(s), and determining whether at least one neuron for the selected output in each layer of the NN can learn to produce the associated output for the input vector. If none of the neurons in a layer of the NN can learn to produce the associated output for the input vector, then a new neuron is added to that layer to learn the associated output which could not be learnt by any other neuron in that layer. The new neuron has its output connected to all neurons in next layer that are relevant to the output being trained. If an output neuron cannot learn the input vector, then another neuron is added to the same layer as the current output neuron and all inputs are connected directly to it. This neuron learns the input the old output could not learn. An additional neuron is added to the next layer. The inputs to this neuron are the old output of the NN, and the newly added neuron to that layer.

Claims

1. A method for training an artificial neural network, said method comprising:

(i) initialising the neural network by selecting an output of the neural network to be trained and connecting an output neuron of the neural network to input neuron(s) in an input layer of the neural network for the selected output;

(ii) preparing a data set to be learnt by the neural network; and

(iii) applying the prepared data set to the neural network to be learnt by applying an input vector of the prepared data set to a first hidden layer of the neural network, or an output layer of the neural network if the neural network has no hidden layer(s), and determining whether at least one neuron for the selected output in each layer of the neural network can learn to produce the associated output for the input vector, wherein;

if at least one neuron for the selected output in each layer of the neural network can learn to produce the associated output for the input vector, and if there are more input vectors of the prepared data set to learn, repeat (iii) for the next input vector, else repeat (i) to (iii) for the next output of the neural network if there are more outputs to be trained;

if no neuron in a hidden layer for the selected output of the neural network can learn to produce the associated output for the input vector, a new neuron is added to that layer to learn the associated output which could not be learnt by any other neurons in that layer for the selected output, and if there are more input vectors of the data set to learn, repeat (iii) for the next input vector, else repeat (i) to (iii) for the next output of the neural network if there are more outputs to be trained;

if the output neuron for the selected output of the neural network cannot learn to produce the associated output for the input vector, that output neuron becomes a neuron of a hidden layer of the neural network, a new neuron is added to this hidden layer to learn the associated output which could not be learnt by the output neuron, and a new output neuron is added to the neural network for the selected output, and if there are more input vectors of the data set to learn, repeat (iii) for the next input vector, else repeat (i) to (iii) for the next output of the neural network if there are more outputs to be trained.

2. A method as claimed in claim 1, wherein (ii) preparing the data set is performed before (i) initializing the neural network.

3. The method as claimed in claim 1, wherein the neurons of the neural network are Linear Threshold Gates (LTGs).

4. The method as claimed in claim 3, wherein in said (iii), to determine whether an LTG can learn to produce the associated output for the input vector is to determine whether a relationship between weights and a threshold of the LTG has a solution given what the LTG has previously learnt.

5. The method as claimed in claim 4, wherein said relationship is a constraint, and wherein the input vector and the LTG'"'"'s weight vector form a relationship with the LTG'"'"'s threshold based on the selected output of the neural network.

6. The method as claimed in claim 5, wherein to learn a constraint is to be able to add the constraint to a constraint set of an LTG.

7. The method as claimed in claim 6, wherein to be able to add the constraint to a constraint set of an LTG there must be a solution between all the constraints.

8. The method as claimed in claim 6, wherein initialising the neural network further includes clearing the constraints set of the output LTG so that the constraints set of the output LTG is empty.

9. The method as claimed in claim 1, wherein preparing the data set to be learnt by the neural network includes in any order:

converting the data set into a predefined data format before the data set is presented to the neural network for training;

determining whether there are any inconsistencies in the data set before the data set is presented to the neural network for training;

sorting the data set before the data set is presented to the neural network for training; and

, determining whether the 0 input vector is available in the data set before the data set is presented to the neural network for training, and if the 0 input vector is available the data set, the data set is ordered so that the 0 input vector is presented to the neural network to be trained first.

10. The method as claimed in claim 9, wherein said predefined data format is binary or floating-point data format.

11. The method as claimed in claim 9, wherein determining whether there are any inconsistencies in the data set before the data set is presented to the neural network includes:

determining whether there are two or more identical input vectors which produce different output.

12. The method as claimed in claim 11, wherein if it is determined that two or more identical input vectors produce a different output, only one of the input vectors is used.

13. The method as claimed in claim 9, wherein sorting the data set before the data set is presented to the neural network for training includes:

sorting the input vectors of the data set into two sets, separating those that output 1 from those that produce 0 for that output, and selecting one of the two sets to be trained first;

sorting the data with a Self Organising Map (SOM); and

/or sorting the data using any other suitable method.

14. The method as claimed in claim 13, wherein a single list for each input layer is created from the sorted data before the data is presented to the neural network for training.

15. The method as claimed in claim 5, wherein if a new LTG is added to a layer to learn a constraint that could not be learnt by any other LTG in accordance with (iii):

the new LTG is connected to all LTGs in the next layer which contribute to the selected output of the neural network, and the constraints set of the LTGs in the next layer which receive input from the new LTG are updated to accept input from the new LTG;

if the layer with the new LTG is not the first layer of the neural network, the new LTG is connected to and receives input from all LTGs in a preceding layer which contribute to the selected output of the neural network; and

, the constraints set of the new LTG is updated to include a copy of the modified constraints set of the previous last LTG in that layer and the constraint which could not be learnt by any other LTG in that layer.

16. The method as claimed in claim 5, wherein if a new output LTG is added to the neural network in accordance with (iii):

the new output LTG is connected to and receives input from the LTGs in the hidden layer;

if the hidden layer is not the first layer of the neural network, the new LTG in the hidden layer is connected to and receives input from all LTGs in a preceding layer which contribute to the selected output of the neural network;

the constraints set of the new LTG added to the hidden layer is updated to include a copy of the modified constraints set of the previous output LTG in that layer and the constraint which could not be learnt by the previous output LTG; and

, the new output LTG combines its inputs in a predefined logical relationship according to what could not be learnt by the previous output LTG.

17. The method as claimed in claim 16, wherein when a new output LTG is added to the neural network in accordance with (iii), the predefined logical relationship formed between the inputs into this new output LTG is logical OR, logical AND, or any other suitable logical relationship.

18. The method as claimed in claim 17, wherein logical OR is used if the input vector that could not be learnt by the previous output LTG produces an output 1, and logical AND is used if the input vector that could not be learnt by the previous output LTG produces an output 0.

19. A method for adding a new neuron into a layer of a neural network during training, the new neuron being added to the neural network when no other neuron in that layer for the selected output can learn a relationship associated with an input vector of a data set being learnt, said method including:

updating the new neuron with a copy of all the modified data from a previous last neuron that contributes to the selected output of the neural network in that layer and the relationship which could not be learnt by any other neuron in that layer; and

,

updating the output neuron(s) to accept input from the new neuron.

20. The method as claimed in claim 19, wherein the neurons of the neural network are LTGs.

21. The method as claimed in claim 20, wherein said relationship is a relationship between weights and a threshold of an LTG.

22. The method as claimed in claim 20, wherein said relationship is a constraint, and wherein the input vector of the data set and an LTG'"'"'s weight vector form a relationship with the LTG'"'"'s threshold based on the output of the neural network.

23-39. -39. (canceled)