Artificial neural networks based on a low-order model of biological neural networks

• US 8,990,132 B2
• Filed: 05/11/2012
• Issued: 03/24/2015
• Est. Priority Date: 01/19/2010
• Status: Active Grant

• Current Assignee: Lo, James Ting-Ho
• Original Assignee: Lo, James Ting-Ho
• Inventors: Lo, James Ting-Ho
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Figueroa, Kevin W
• Application Number:
  US13/469,090
• Publication Number:
  US 20130304683A1
• Time in Patent Office: 1,047 Days
• Field of Search: 706/20
• US Class Current: 706/20
• CPC Class Codes: G06N 3/049 Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
• CPC Class Codes: G06N 3/06 Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning
• CPC Class Codes: G06N 3/12 using genetic models

Abstract

A low-order model (LOM) of biological neural networks and its mathematical equivalents including the clusterer interpreter probabilistic associative memory (CIPAM) are disclosed. They are artificial neural networks (ANNs) organized as networks of processing units (PUs), Each PU comprising artificial neuronal encoders, synapses, spiking/nonspiking neurons, and a scheme for maximal generalization. If the weights in the artificial synapses in a PU have been learned (and then fixed) or can be adjusted by the unsupervised accumulation rule and the unsupervised covariance rule (or supervised covariance rule), the PU is called unsupervised (or supervised) PU. The disclosed ANNs, with these Hebbian-type learning rules, can learn large numbers of large input vectors with temporally/spatially hierarchical causes with ease and recognize such causes with maximal generalization despite corruption, distortion and occlusion. An ANN with a network of unsupervised PUs (called clusterer) and offshoot supervised PUs (called interpreter) is an architecture for many applications.

Claims

1. An artificial neural network for processing data, comprising at least one processing unit, a first processing unit including

(a) at least one artificial neuronal encoder for encoding a vector into a neuronal code;

(b) a means for evaluating a code deviation vector that is the deviation of a neuronal code obtained by said artificial neuronal encoder from a neuronal code average;

(c) a plurality of artificial synapse memories each for storing a component of a code deviation accumulation vector;

(d) a first means for evaluating a first product of a component of a code deviation accumulation vector, a masking factor, and a component of a code deviation vector;

(e) an artificial nonspiking neuron processor for evaluating a first sum of first products obtained by said first means;

(f) a plurality of artificial synapse memories each for storing an entry of a code covariance matrix;

(g) a second means for evaluating a second product of an entry of a code covariance matrix, a masking factor, and a component of a code deviation vector; and

(h) at least one artificial spiking neuron processor for evaluating a second sum of second products obtained by said second means, and for using at least said second sum and a first sum obtained by said artificial nonspiking neuron processor to evaluate a representation of a first empirical probability distribution of a component of a label of a vector that is input to said first processing unit.

2. The artificial neural network of claim 1, wherein a plurality of code covariance matrices are submatrices of a general code covariance matrix, a plurality of code deviation accumulation vectors are subvectors of a general code deviation accumulation vector, a plurality of masking factors are entries of a masking matrix, a plurality of masking matrices are submatrices of a general masking matrix, a plurality of neuronal codes are subvectors of a general neuronal code, and a plurality of neuronal code averages are subvectors of a general neuronal code average.

3. The artificial neural network of claim 1, further comprising at least one feedback connection with a time delay means.

4. The artificial neural network of claim 1, said first processing unit further including an unsupervised accumulation learning means for adjusting at least one component of a code deviation accumulation vector by an unsupervised accumulation rule in response to a vector that is input to said first processing unit.

5. The artificial neural network of claim 1, said first processing unit further including a pseudorandom number generation means for generating at least one pseudo-random number in accordance with a first empirical probability distribution obtained by said at least one artificial spiking neuron processor.

6. The artificial neural network of claim 5, said first processing unit further including an unsupervised learning means for using at least a pseudorandom number generated by said pseudorandom number generation means and an entry of a code deviation vector to adjust an entry of a code covariance matrix by an unsupervised covariance rule in response to a vector that is input to said first processing unit.

7. The artificial neural network of claim 1, said first processing unit further including

(a) a third means for evaluating a third product of a component of a code deviation accumulation vector, a learning masking factor, and a component of a code deviation vector;

(b) a summing means for evaluating a third sum of third products obtained by said third means;

(c) a fourth means for evaluating a fourth product of an entry of a code covariance matrix, a learning masking factor, and a component of a code deviation vector;

(d) a summing-evaluating means for evaluating a fourth sum of fourth products obtained by said fourth means and for using at least said fourth sum and a third sum obtained by said summing means to evaluate a representation of a second empirical probability distribution of a component of a label of a vector that is input to said first processing unit; and

(e) an unsupervised learning means for using at least a pseudorandom number generated in accordance with a second empirical probability distribution generated by said summing-evaluation means and a component of a code deviation vector to adjust an entry of a code covariance matrix by an unsupervised covariance rule in response to a vector that is input to said first processing unit.

8. The artificial neural network of claim 1, said first processing unit further including a supervised learning means for using at least one entry of a code deviation vector and one component of a given label of a vector that is provided from outside said first processing unit to adjust at least one entry of a code covariance matrix by a supervised covariance rule in response to a vector that is input to said first processing unit.

9. The artificial neural network of claim 1, wherein a plurality of processing units are unsupervised processing units, which form a clusterer, and at least one processing unit is a supervised processing unit, the set of said at least one supervised processing unit being an interpreter.

10. A learning machine for processing data, said learning machine comprising at least one processing unit, a first processing unit including

(a) an encoding means for encoding a vector into a code;

(b) a means for evaluating a code deviation vector;

(c) a memory means for storing at least one first weighted sum of an entry of a code covariance matrix and a component of a code deviation accumulation vector;

(d) a memory means for storing at least one masking factor;

(e) a multiplying means for evaluating a product of a second weighted sum of an entry of a code covariance matrix and a component of a code deviation accumulation vector, a masking factor, and a component of a code deviation vector;

(f) a summing means for evaluating a sum of products obtained by said multiplying means;

(g) an evaluation means for using at least a sum obtained by said summing means to evaluate a representation of an empirical probability distribution of a component of a label of a vector input to said first processing unit.

11. The learning machine defined in claim 10, wherein said first weighted sum of an entry of a code covariance matrix and a component of a code deviation accumulation vector is a component of a code deviation accumulation vector.

12. The learning machine defined in claim 10, wherein said first weighted sum of an entry of a code covariance matrix and a component of a code deviation accumulation vector is an entry of a code covariance matrix.

13. The learning machine defined in claim 10, wherein said first weighted sum of an entry of a code covariance matrix and a component of a code deviation accumulation vector is a sum of an entry of a code covariance matrix and a component of a code deviation accumulation vector.

14. The learning machine of claim 10, said first processing unit further including a memory means for storing at least one learning masking factor;

and a second evaluation means for using at least a weighted sum of an entry of a code covariance matrix and a component of a code deviation accumulation vector;

a learning masking factor;

and a component of a code deviation vector to evaluate a representation of an empirical probability distribution of a component of a label of a vector input to said first processing unit.

15. The learning machine of claim 10, said first processing unit further including a pseudorandom number generation means for generating at least one pseudo-random number in accordance with an empirical probability distribution obtained by said evaluation means.

16. The learning machine of claim 15, said first processing unit further including an unsupervised learning means for using at least a pseudorandom number generated by said pseudorandom number generation means and an entry of a code deviation vector to adjust at least one first weighted sum of an entry of a code covariance matrix and a component of a code deviation accumulation vector.

17. The learning machine of claim 16, wherein said unsupervised learning means adjusts at least one component of a code deviation accumulation vector by an unsupervised accumulation rule.

18. The learning machine of claim 16, wherein said unsupervised learning means adjusts at least one entry of a code covariance matrix by an unsupervised covariance rule.

19. The learning machine of claim 15, further comprising a feedback means for feedbacking at least one pseudorandom number generated by said pseudorandom number generation means to a processing unit after a time delay.

20. The learning machine of claim 10, said first processing unit further including a supervised learning means for adjusting at least one entry of a code covariance matrix by a supervised covariance rule.

21. The learning machine of claim 10, wherein a plurality of processing units are unsupervised processing units, which form a clusterer, and at least one processing unit is a supervised processing unit, which forms a interpreter.

22. A method for processing data, said method comprising steps of:

(a) encoding a subvector of a first vector into a code;

(b) evaluating a code deviation vector that is the deviation of a code from a code average;

(c) evaluating a product of a weighted sum of a component of a code deviation accumulation vector and an entry of a code covariance matrix, a masking factor, and a component of a code deviation vector;

(d) evaluating a sum of products obtained by said step of evaluating a product; and

(e) using at least a sum of products obtained by said step of evaluating a sum to evaluate a representation of an empirical probability distribution of a component of a label of said first vector.

23. The method of claim 22, wherein a weighted sum of a component of a code deviation accumulation vector and an entry of a code covariance matrix is a component of a code deviation accumulation vector.

24. The method of claim 22, wherein a weighted sum of a component of a code deviation accumulation vector and an entry of a code covariance matrix is an entry of a code covariance matrix.

25. The method of claim 22, wherein a weighted sum of a component of a code deviation accumulation vector and an entry of a code covariance matrix is a sum of an entry of a code covariance matrix and a component of a code deviation accumulation vector.

26. The method of claim 22, further comprising a step of using at least a component of a code deviation vector and a component of a given label of said first vector to adjust at least one entry of a code covariance matrix by a supervised covariance rule, said component of a given label being provided for said method to use.

27. The method of claim 22, further comprising a step of generating a pseudo-random number in accordance with an empirical probability distribution of a component of a label of said first vector.

28. The method of claim 27, further comprising a step of including a pseudo-random number generated in accordance with an empirical probability distribution of a component of a label as a component in said first vector after a time delay.

29. The method of claim 27, further comprising a step of adjusting a weighted sum of a component of a code deviation accumulation vector and an entry of a code covariance matrix.

30. The method of claim 29, wherein said step of adjusting a weighted sum is a step of adjusting a component of a code deviation accumulation vector by an unsupervised accumulation rule.

31. The method of claim 29, wherein said step of adjusting a weighted sum is a step of adjusting an entry of a code covariance matrix by an unsupervised covariance rule.

32. The method of claim 29, further comprising a step of using at least a component of a code deviation vector and a component of a given label of said first vector to adjust a component of a code covariance matrix by a supervised covariance rule, said given label being provided for said method to use.

33. The method of claim 22, further comprising steps of:

(a) evaluating a second product of a weighted sum of a component of a code deviation accumulation vector and an entry of a code covariance matrix, a learning masking factor, and a component of a code deviation vector;

(b) evaluating a sum of second products obtained by said step of evaluating a second product; and

(c) using at least a sum of second products obtained by said step of evaluating a sum of second products to evaluate a representation of a second empirical probability distribution of a component of a label of said first vector.

34. The method of claim 33, further comprising a step of using at least a sum of second products obtained by said step of evaluating a sum of second products and a component of a given label of said first vector to adjust an entry of a code covariance matrix by a supervised covariance rule, said given label being provided for said method to use.