Dynamically stable associative learning neural network system

• US 5,822,742 A
• Filed: 02/24/1995
• Issued: 10/13/1998
• Est. Priority Date: 05/17/1989
• Status: Expired due to Term

• Current Assignee: MDA Information Systems LLC (Galileo Parent, Inc.), The United States of America, as represented by the Secretary, Department of Health and Human Services
• Original Assignee: ERIM International, Inc. (Altarum Institute), United States Department Of Health And Human Services (Government of the United States of America)
• Inventors: Barbour, Garth S., Alkon, Daniel L., Vogl, Thomas P., Blackwell, Kim T.
• Primary Examiner(s): Downs, Robert W.
• Application Number:
  US08/331,554
• Time in Patent Office: 1,327 Days
• Field of Search: 395/21, 395/24, 395/22, 395/23, 382/159, 382/181, 382/185, 706/31, 706/20, 706/25
• US Class Current: 706/31
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/18057 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]

Abstract

A dynamically stable associative learning neural system includes a plurality of neural network architectural units. A neural network architectural unit has as input both condition stimuli and unconditioned stimulus, an output neuron for accepting the input, and patch elements interposed between each input and the output neuron. The patches in the architectural unit can be modified and added. A neural network can be formed from a single unit, a layer of units, or multiple layers of units.

Claims

1. A neural network architectural unit comprising:

(a) a first input for receiving a set of conditioned stimuli;

(b) a second input for receiving an unconditioned stimulus;

(c) a patch element coupled to the first and second inputs and for receiving and processing the set of conditioned stimuli and the unconditioned stimulus, the patch element producing an output; and

(d) an output neuronal element for accepting the output of the patch element and for providing an output for the neural network architectural unit.

2. The neural network of claim 1, wherein the output neuronal element has a receptive field which comprises a subset of the conditioned stimuli and a component of the unconditioned stimulus.

3. The neural network of claim 2, wherein the patch element is associated with the receptive field which retains at least one connection to the patch element but from which at least one connection to the patch has been removed, provided that the signal on each of the removed connections, at the time of its removal, varied significantly when the patch element was active relative to the input'"'"'s variance overall.

4. The neural network of claim 1, further comprising a plurality of patch elements, each of which receives a plurality of inputs and is connected to the output neuronal element.

5. The neural network of claim 4, wherein each patch element spans a volume of hyperspace determined by a plurality of thresholds used as criteria in deciding either to create a new patch element or to modify a previously existing patch element when a conditioned stimulus is input to the network, wherein a first threshold sets the maximum size for a patch element, and a second threshold sets a minimum size for a patch element.

6. The neural network of claim 5, wherein the thresholds are further defined as functions of the similarity computed between a value of the conditioned stimulus and a value of a previously existing patch element.

7. The neural network of claim 1, wherein the set of conditioned stimuli arises from a pattern to be classified, and the unconditioned stimulus is a classification category.

8. The neural network of claim 1, wherein the conditioned stimuli are generated by optical scanning of a photograph.

9. The neural network of claim 1, wherein the conditioned stimuli are signals representing values at previous specific points in time, and the unconditioned stimuli are signals representing values at a present time.

10. The neural network of claim 1, further comprising:

(a) a patch vector which is the average value of the conditioned stimuli with which the patch element has matched;

(b) the expected value of one component of the unconditioned stimulus that is associated with the conditioned stimuli; and

(c) a weight based on frequency of use of the patch.

11. The neural network of claim 1, wherein the patch element is generated or modified by the first and second inputs, are storage areas in a physical apparatus, and is a site of local learning in the immediate vicinity of paired conditioned and unconditioned stimuli.

12. The neural network of claim 11, wherein the physical apparatus is a VSLI chip.

13. The neural network of claim 1, wherein the patch element responds to conditioned and unconditioned stimuli in accordance with rules and methods to process the signals by organizing the signals into classes or features to condense or modify information to achieve the goals of an application of the network.

14. A neural network comprising a plurality of the neural network architectural units of claim 1.

15. A method for deciding whether to create a new patch in a neural network, said method comprising:

(a) presenting as input to the neural network a conditioned stimulus and an unconditioned stimulus with which the conditioned stimulus is to be associated;

(b) for each output neuron of the neural network, comparing values of the presented conditioned stimulus within the receptive field of the output neuron, with values of conditioned stimuli stored in all preexisting patches of the output neuron;

(c) selecting the preexisting patch with the stored value most similar to the value of the presented conditioned stimulus; and

(d) determining whether the similarity between the stored value of the selected preexisting patch and the value of the presented conditioned stimulus requires formation of a new patch.

16. The method of claim 15, wherein the similarity is determined as a correlation coefficient computed between the conditioned stimulus within the receptive field of the output neuron and the stored values in the preexisting patches.

17. The method of claim 16, further comprising determining the set of preexisting patches with expected values of unconditioned stimuli which match the value of the presented unconditioned stimulus.

18. The method of claim 15, wherein the selected preexisting patch is updated if the similarity between the presented conditioned stimulus and the selected preexisting patch is greater than a predefined threshold and if the value of the presented unconditioned stimulus matches the unconditioned stimulus value of the selected preexisting patch.

19. The method of claim 15, wherein the step of selecting comprises comparing only the active area of each patch with the value of the presented conditioned stimulus.

20. A neural network apparatus comprising;

input terminals for receiving signals;

signal processing means configured in a plurality of layers including;

(i) a first layer of neuron circuits configured to isolate features from the signals received at the input terminals, the first layer being both an input layer and a non-output layer;

(ii) a second layer of neuron circuits connected to communicate with the first layer of neuron circuits to receive feature information carrying signals from the first layer, the second layer being both a non-input layer and an output layer and being operative to establish classifications of the features in the absence of a priori knowledge of the features;

(iii) output terminals receiving classification carrying signals from the second layer;

wherein at least one intermediate layer of neuron circuits connected successively between said first and second layers of neuron circuits, each said intermediate layer being both a non-input and a non-output layer and having at least one neuron circuit connected to receive outputs of neuron circuits from a preceding layer and to provide inputs to neuron circuits of a subsequent layer;

wherein the neuron circuits of the input layer include means for storing in a memory associations between an input pattern received by the neuron circuits and an input unconditioned stimulus value, thereby generating patches; and

wherein the non-input layers include means for storing arbitrary labels of patches of neuron circuits from a preceding layer, thereby generating patches designated as meta-patches.

21. The apparatus recited in claim 20, wherein in each layer is stored a measure of similarity between each of its patches and an input pattern of the layer.

22. The apparatus recited in claim 21, further comprising means for storing a measure of similarity between patches of the input layer and a conditioned stimulus pattern, the conditioned stimulus pattern being the input pattern of the input layer.

23. The apparatus recited in claim 21, further comprising means for storing measures of similarity between patches of a non-input layer and arbitrary labels of patches from the neuron circuits of the preceding layer, the arbitrary labels identifying the input pattern of the non-input layer.

24. The apparatus recited in claim 21, wherein a neuron circuit in a non-output layer propagates to a subsequent layer the label of a patch having the greatest similarity to the input pattern of the neuron circuit.

25. The apparatus recited in claim 24, wherein the non-output layer is an intermediate layer and the similarity between the patch and the input pattern of the neuron circuit is determined by a means for comparing patch labels being propagated to the neuron circuit with patch labels stored in the meta-patch.

26. The apparatus recited in claim 25, wherein the comparing means comprises a counter counting at least one of matching and mismatching patch labels of the patch labels being propagated to the neuron circuit and the patch labels stored in the meta-patch.

27. The apparatus recited in claim 24, wherein the neuron circuit in the non-output layer further propagates an ordinal measure of similarity of the patch having the greatest similarity to the input pattern.

28. The apparatus recited in claim 27, further comprising circuitry for determining a weighted average of matching, wherein matches are assigned a value of +1 times the ordinal measure of similarity and mismatches are assigned a value of -1 times the ordinal measure of similarity, the weighted value of matching providing a measure of confidence for each match.

29. The apparatus recited in claim 21, wherein a neuron circuit in a non-output layer propagates to a subsequent layer the labels and similarity measures of all patches of the neuron circuit.

30. The apparatus recited in claim 29, wherein meta-patches of the neuron circuit in the subsequent layer compute a similarity value from patch labels previously stored in a patch vector of the meta-patch.

31. The apparatus recited in claim 21, further comprising means for storing a patch vector having similarity values and patch labels for a subset of all patches of a neuron circuit, the patch labels of the subset being one of either the top n patches or all patches exceeding a predetermined threshold.

32. The apparatus recited in claim 31, further comprising means for determining a similarity between a meta-patch of a neuron circuit of a non-input layer and the input pattern of the non-input neuron circuit as an average of similarity values paired with the patch labels stored in the patch vector of the meta-patch.

33. The apparatus recited in claim 32, further comprising means for identifying input similarity values of the meta-patch below a second predetermined threshold and ignoring the identified input similarities in classifying the input pattern of the neuron circuit.

34. The apparatus recited in claim 31, wherein meta patches having a similarity closest to a stored similarity value are propagated.

35. The apparatus recited in claim 20, wherein each neuron circuit has its own set of patches.

36. The apparatus recited in claim 20, further comprising means for storing a list of patches accessible to all neuron circuits of each layer and wherein the neuron circuit of each layer compares the input pattern of the neuron circuit with patterns in the patches in the list and generates a new patch only if no patch in the list is similar to the input pattern of the neuron circuit.

37. The apparatus recited in claim 36, further comprising means for partitioning the input pattern of a neuron into data vectors and comparing the data vectors to predetermined codes to identify matches.

38. A neural network comprising:

a plurality of sequential subnetworks having at least a first subnetwork and a final subnetwork, each subnetwork having input neurons, a plurality of patches, and output neurons;

wherein output signals are generated at the output neurons of each subnetwork by transmitting input signals received at the input neurons of the subnetwork through the plurality of patches of the subnetwork;

wherein the input neurons of the first subnetwork receive input signals from sources external to the neural network;

wherein each patch of the plurality of patches of the first subnetwork is generated in response to the input signals received by the input neurons of the first subnetwork;

wherein the input neurons of each subnetwork, except for the first subnetwork, are coupled to the output neurons of the previous subnetwork and receive as input signals the output signals generated at the output neurons of the previous subnetwork;

wherein the input neurons of the final subnetwork further receive unconditioned stimuli; and

wherein one of the subnetworks, except the first subnetwork, receives identifying values from only a subset of the plurality of patches of the first subnetwork.

39. The neural network of claim 38, wherein the patches of the first subnetwork respond to features of the received input signals.

40. The neural network of claim 38, wherein the subset of patches of the first subnetwork are those having the highest similarity measures across a training set of signals presented to the first subnetwork.

41. The neural network of claim 38, wherein the subset of patches of the first subnetwork are those within a predefined range of variance with an unconditioned stimuli-specific signal.

42. The neural network of claim 38, wherein the subset of patches of the first subnetwork are those having the least variance with an unconditioned stimuli-specific signal, wherein patches having a variance greater than said least variance are added to the subset of patches to improve network performance.

43. The neural network of claim 38, wherein the neural network is used for classifying a pattern, wherein:

(a) the subnetworks are connected in series by signal communication paths;

(b) each subnetwork comprises a plurality of sequential layers having at least a first layer comprising the input neurons of the subnetwork and a final layer comprising the output neurons of the subnetwork, wherein the first layer of each subnetwork, except for the first subnetwork, is coupled to the final layer of the previous subnetwork;

(c) the patches in the first subnetwork define a first level of features;

(d) each subnetwork subsequent to the first subnetwork is selected by the output signals of the first subnetwork; and

(e) the plurality of patches of each subsequent subnetwork increasingly refine classification levels.

44. The neural network of claims 43, wherein the plurality of subnetworks further comprises a plurality of second subnetworks coupled to the first subnetwork, wherein the input neurons of each second subnetwork receives a single patch group identification output from the first subnetwork.

45. The neural network of claim 44, wherein a group of patches is designated as a meta-patch.

46. The neural network of claim 43, wherein the plurality of subnetworks further comprises a second subnetwork having a first layer and two second layers, each second layer connected to the first layer of the second subnetwork, and wherein the first of the two second layers contains patches coupled to the final layer of the first subnetwork, and directs an input signal to be classified to a patch group, and wherein the second of the two second layers contains patches distinguishing individual patterns.

47. A method for classifying a pattern using the neural network of claim 43, the method comprising:

(a) recognizing a set of features of the pattern to be classified by creating patches relating to the features in a first layer of the first subnetwork;

(b) combining patch identification labels into a group using a second layer of the first subnetwork;

(c) transferring the patch identification labels from the first subnetwork to a first layer of a second subnetwork, wherein patches based on the transferred patch identification labels are prevented from forming in the second subnetwork, and wherein new patches formed in the second subnetwork reflect the difference between a pattern generated by the first subnetwork patches and the pattern to be classified; and

(d) grouping the new patches using a second layer of the second subnetwork, the second layer associated with unconditioned stimuli specific for the pattern to be classified.

48. The method of claim 47, wherein the pattern to be classified is a human face.

49. The method of claim 48, wherein the human face is presented to the neural network system using a digitized photograph of the face.

50. The method of claim 48, wherein the human face is presented to the neural network system using a frame obtained from a video camera output.

51. The method of claim 47, wherein the pattern to be classified is a Japanese Kanji character, and wherein the first layer of the first subnetwork recognizes a combination of strokes which is a component of the pattern to be classified, and wherein groupings by the second layer of the first subnetwork are a function of the size of the receptive field of the first subnetwork.