Non-algorithmically implemented artificial neural networks and components thereof

• US 5,845,271 A
• Filed: 01/26/1996
• Issued: 12/01/1998
• Est. Priority Date: 01/26/1996
• Status: Expired due to Term

• Current Assignee: Stephen L. Thaler
• Original Assignee: Stephen L. Thaler
• Inventors: Thaler, Stephen L.
• Primary Examiner(s): Hafiz, Tarig R.
• Assistant Examiner(s): Shah, Sanjiv
• Application Number:
  US08/592,767
• Time in Patent Office: 1,040 Days
• Field of Search: 395/22, 395/76, 395/11, 364/807, 706/16, 706/25, 706/60, 706/10
• US Class Current: 706/16
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/105 Shells for specifying net layout

Abstract

Constructing and simulating artificial neural networks and components thereof within a spreadsheet environment results in user friendly neural networks which do not require algorithmic based software in order to train or operate. Such neural networks can be easily cascaded to form complex neural networks and neural network systems, including neural networks capable of self-organizing so as to self-train within a spreadsheet, neural networks which train simultaneously within a spreadsheet, and neural networks capable of autonomously moving, monitoring, analyzing, and altering data within a spreadsheet. Neural networks can also be cascaded together in self training neural network form to achieve a device prototyping system.

Claims

1. A computer based neural network training system, comprising:

a computer including a spreadsheet application program operable therewith for electronically generating a spreadsheet including a plurality of spreadsheet cells arranged in a column and row format such that each spreadsheet cell is identifiable by a column and row designation, said computer and spreadsheet application program operable to enable interrelating of said plurality of spreadsheet cells through relative cell referencing;

a first functional neural network constructed within said spreadsheet and including a plurality of imaging cells for relatively referencing a set of training inputs to said first neural network, said first neural network further including at least one hidden layer including a first plurality of neurons and an output layer including a second plurality of neurons, wherein each neuron of said hidden layer and said output layer is formed by a first plurality of cells each containing a numeric weight value of said neuron and an activation cell containing an activation function which activation function relatively references each of said first plurality of cells such that when a calculate function of said spreadsheet is performed a numeric value which is representative of an activation level of said neuron is determined, said hidden layer and output layer neurons interrelated through relative cell referencing to form said first neural network;

a training network constructed within said spreadsheet, said training network including a second functional neural network constructed within said spreadsheet and having substantially the same configuration as the first neural network; and

wherein, when a calculate function of said spreadsheet is performed, a given set of training inputs is applied to said first neural network and each training input of the given set of training inputs is adjusted by a predetermined incremental amount before being applied to said second neural network.

2. A computer based neural network training system in accordance with claim 1 wherein at least a portion of said training network is integrated with said first neural network within said spreadsheet.

3. A computer based neural network training system in accordance with claim 1 wherein said training network further includes a derivative module constructed within said spreadsheet such that when the calculate function of said spreadsheet is performed said derivative module is operable to determine, for each of said hidden layer neurons and each of said output layer neurons, a partial derivative of the activation level thereof with respect to a net input thereto based at least in part on a difference in the activation levels of corresponding activation cells of said first neural network and said second neural network.

4. A computer based neural network training system in accordance with claim 3 wherein said training network further includes an error module constructed within said spreadsheet such that when the calculated function of the spreadsheet is performed said error module is operable to determine an error vector associated with said given set of training inputs applied to said first neural network.

5. A computer based neural network training system in accordance with claim 4 further comprising a program associated with said training network and said first neural network, at least a portion of said program operable to effect alteration of said numeric weight values of said first neural network based upon weight update terms calculated by said training network.

6. A computer based neural network training system in accordance with claim 5 wherein sets of training inputs are stored as numeric values associated with cells of said spreadsheet and at least a portion of said program is operable to effect movement of both said first neural network and said training network to a new location within said spreadsheet such that for a given movement of said neural network and said training network to a given new location a calculate function of said spreadsheet is performed and at least some of said numeric weight values of each neuron of said first neural network are altered to incorporate a knowledge domain represented by a given set of training inputs associated with said given new location within said spreadsheet.

7. A computer based neural network training system in accordance with claim 1, further comprising means for providing a dynamic data exchange between said spreadsheet and an external system so that sets of training inputs are input into predetermined cells within said spreadsheet, and, as said sets of training inputs flow through said spreadsheet a calculate function of said spreadsheet is repeatedly performed.

8. A computer based neural network training system in accordance with claim 1, further comprising means for dynamically pruning at least one of said hidden layer neurons from said first neural network in an automatic manner during training.

9. A computer based neural network training system in accordance with claim 8 wherein said means for dynamically pruning at least one hidden layer neuron from said first neural network includes a program associated with said first neural network, said program effecting determination of whether said at least one hidden layer neuron is significantly involved in training, and, if said at least one hidden layer neuron is not significantly involved in training, to set the activation function associated with said at least one hidden layer neuron to zero (0).

10. A computer based neural network training system in accordance with claim 1, further comprising means for adding a new hidden layer neuron to said first neural network in an automatic manner during training.

11. A computer based neural network training system in accordance with claim 10 wherein said means for adding a new hidden layer neuron to said first neural network includes a program associated with said first neural network, said program effecting determination of whether an error value associated therewith exceeds a predetermined threshold.

12. A computer based neural network training system in accordance with claim 11 wherein said program further effects, at predetermined intervals during a training operation, addition of a new hidden layer neuron to said first neural network if said error value exceeds said predetermined threshold.

13. A self training neural network object implemented utilizing a computer including processing means operable to run a spreadsheet application, comprising:

a first functional neural network constructed in a spreadsheet of the spreadsheet application, said first neural network including a plurality of neurons each formed of a plurality of spreadsheet cells including a first plurality of cells each with an associated numeric weighting value of such neuron entered therein and an activation cell having an activation function of such neuron entered therein which activation function makes relative reference to each of said first plurality of cells, wherein said neurons are interrelated through relative cell referencing to form said first neural network;

a training network constructed in the spreadsheet, said training network including a second functional neural network having the same configuration as said first neural network, said training network further including at least one other module constructed within the spreadsheet for calculating weight update terms,

a program associated with said training network and said first neural network, said training network operable in conjunction with said program during a training operation to alter said numeric weighting value associated with at least some of said first plurality of cells of each neuron of said neural network based upon the weight update terms calculated by said training network,

wherein a given set of training inputs is applied to said self training neural network object by initiating a calculate function of said spreadsheet and said numeric weighting value associated with at least some of said plurality of cells of each neuron is altered to incorporate into said neural network a knowledge domain represented by said given set of applied training inputs.

14. A self training neural network object in accordance with claim 13 wherein, for said given set of applied training inputs said program is operable to effect addition of one of said weight update terms to said numeric weighting value associated with each cell of said first plurality of cells of each neuron of said first neural network.

15. A self training neural network object in accordance with claim 13 wherein, for said given set of applied training inputs said program is operable to effect replacement of said numeric weighting value associated with each cell of said first plurality of cells of each neuron of said first neural network with one of said calculated weight update terms.

16. A method of training a neural network, utilizing a computer including a processing means and an associated spreadsheet application operable therewith, said method comprising the steps of:

(a) constructing a first neural network to be trained within a spreadsheet of the spreadsheet application by interrelating cells of the spreadsheet through relative cell referencing, wherein each hidden layer neuron and each output layer neuron of the constructed first neural network is formed by a plurality of cells each having a respective weight value of such neuron associated therewith and an activation cell containing an activation function of such neuron, such that for a given calculate operation of the spreadsheet the first neural network functions to produce outputs in accordance with its then current structure;

(b) constructing a training network within the spreadsheet of the spreadsheet application, the training network including a second neural network constructed within the spreadsheet and having the same configuration as the first neural network, the training network further including a plurality of interrelated cells containing equations for calculating weight update terms for the first neural network being trained, such that for a given calculate operation of the spreadsheet during a training operation the training network functions to produce such weight update terms;

(c) applying a set of training inputs to the first neural network being trained,

(d) adjusting each training input of the plurality of training inputs by an incremental amount and applying each of the adjusted training inputs to the second neural network;

(e) establishing weight update terms within the training network based at least in part upon a difference in activation levels between corresponding activation cells of the first and second neural networks;

(f) altering the weight values associated with each neuron of the first neural network being trained based upon the weight update terms established by the training network to reflect a knowledge domain represented by the set of training inputs.

17. A method of training a neural network in accordance with claim 16 wherein step (f) includes adding each weight update term to one of the weight values of the neural network being trained.

18. A method of training a neural network in accordance with claim 16 wherein step (f) includes replacing each weight value of the neural network being trained with one of the weight update terms.

19. A method of training a neural network in accordance with claim 16 wherein the training network includes a derivative module constructed in the spreadsheet and in step (e) the derivative module calculates, for each of the activation cells, a derivative of activation level with respect to net input.

20. A method of training a neural network according to claim 16 wherein the training network includes an error module constructed within the spreadsheet and in step (e) the error module calculates an error representative of a difference between a set of outputs produced by the first neural network being trained and a set of training outputs corresponding to the set of training inputs applied thereto.

21. A method of training a neural network in accordance with claim 20, further comprising the step of:

(g) repeating steps (c), (d), (e) and (f) until said error falls below a predetermined value.

22. A method of training a neural network in accordance with claim 16 wherein step (c) includes providing relative movement within the spreadsheet between the first neural network and a plurality of sets of training data located within the spreadsheet.

23. A method of training a neural network in accordance with claim 16 further comprising the step of scanning the spreadsheet for a set of training data prior to initiating the calculate function.

24. A method of simultaneously training at least two neural networks, utilizing a computer including processing means and an associated spreadsheet application operable therewith, said method comprising the steps of:

(a) constructing a first functional neural network to be trained within a spreadsheet produced by the spreadsheet application by interrelating cells of the spreadsheet through relative cell referencing, wherein each hidden layer neuron and each output layer neuron of the first neural network is formed by plurality of cells each having a respective weight value of such neuron associated therewith and an activation cell containing an activation function of such neuron,

(b) constructing a first training network within the spreadsheet of the spreadsheet application for use in training the first neural network, the first training network including a plurality of interrelated cells containing equations for calculating weight update terms for the first neural network,

(c) constructing a second functional neural network to be trained within the spreadsheet produced by the spreadsheet application by interrelating cells of the spreadsheet through relative cell referencing, wherein each hidden layer neuron and each output layer neuron of the second neural network is formed by plurality of cells each having a respective weight value of such neuron associated therewith and an activation cell containing an activation function of such neuron,

(d) constructing a second training network within the spreadsheet of the spreadsheet application for use in training the second neural network, the second training network including a plurality of interrelated cells containing equations for calculating weight update terms for the second neural network,

(e) simultaneously applying training data located within the spreadsheet to both the first neural network and the second neural network by initiating a calculate function of the spreadsheet,

(f) altering at least a portion of the first neural network in accordance with weight update terms produced by the first training network, and

(g) altering at least a portion of the second neural network in accordance with weight update terms produced by the second training network.

25. A method of simultaneously training at least two neural networks in accordance with claim 24 wherein step (e) includes applying a first set of training data to the first neural network and simultaneously applying a second set of training data to the second neural network, said first set of training data and said second set of training data having at least one variable in common.

26. A method of simultaneously training at least two neural networks in accordance with claim 24 wherein step (e) includes applying a first set of training data to the first neural network and simultaneously applying a second set of training data to the second neural network, said first set of training data and said second set of training data made up of distinct variables.

27. A computer based neural network training system, comprising:

processing means operable to electronically generate a data space including a plurality of cells;

means associated with said data space and said processing means for maintaining a numeric value associated with each cell,

means associated with said data space and said processing means for interrelating said cells through relative cell referencing,

a neural network constructed within said data space, said neural network including a plurality of imaging cells for relatively referencing a plurality of training inputs to said neural network, at least one hidden layer including a plurality of neurons, and an output layer including a plurality of neurons, each neuron of said hidden layer and said output layer formed by a plurality of cells including a first plurality of cells each for containing a numeric weight value of said neuron and an activation cell containing an activation function which makes relative reference to each of said first plurality of cells to establish a numeric value which is dependent upon said numeric weight values and is representative of an activation level of said neuron,

means associated with said neural network for altering said numeric weight values of said neurons during training of said neural network,

whereby, for a given set of training inputs and corresponding training outputs on which said neural network is being trained, at least some of said numeric weight values of each neuron are altered to incorporate into said neural network a knowledge domain represented by said given set; and

a data filtering neural network including an autoassociative neural network constructed in said data space, said autoassociative neural network having been trained on a plurality of control sets of inputs thereto, whereby, for a given set of inputs within a knowledge domain represented by said plurality of control sets of inputs, said autoassociative neural network is operable to map said given set of inputs to themselves.

28. A computer based neural network training system in accordance with claim 27, further comprising a program associated with said data filtering neural network, said neural network and said training network, at least a portion of said data filtering neural network operable to determine an error between a given set of inputs and a resulting set of outputs of said autoassociative neural network, at least a portion of said program operable to determine if said error exceeds a predetermined value, and, only if said error exceeds said predetermined value, to alter at least some of said numeric weight values of each neuron of said neural network, so that said neural network is trained on only novel sets of training inputs and corresponding training outputs.

29. A self training neural network object implemented utilizing a computer including processing means operable to run a spreadsheet application, comprising:

a neural network constructed in a spreadsheet of the spreadsheet application, said neural network including a plurality of neurons each formed of a plurality of cells including a first plurality of cells each with an associated numeric weighting value entered therein and an activation cell having a function entered therein which makes relative reference to each of said first plurality of cells,

a training network constructed in the spreadsheet,

a program associated with said training network and said neural network, said training network operable in conjunction with said program during a training operation to alter said numeric weighting value associated with at least some of said first plurality of cells of each neuron of said neural network,

whereby, for a given set of training inputs and corresponding training outputs applied to said self training neural network object, said numeric weighting value associated with at least some of said plurality of cells of each neuron is alterable to incorporate into said neural network a knowledge domain represented by said given set of applied training inputs and corresponding training outputs; and

an autoassociative neural network constructed in said spreadsheet, a plurality of the variables making up said given set of training inputs and corresponding training outputs being applied as inputs to said autoassociative neural network, said autoassociative neural network operable during training to determine, for a given set of inputs thereto, an error value, said error value representing a difference between said given set of inputs thereto and a resulting set of outputs therefrom, wherein said program is operable to effect determination of whether said error exceeds a predetermined value and, if said error is less than said predetermined value, to prevent alteration of said numeric weighting value associated with each cell of said plurality of cells of each neuron of said neural network.