SPANN: Sequence processing artificial neural network

• US 5,067,095 A
• Filed: 01/09/1990
• Issued: 11/19/1991
• Est. Priority Date: 01/09/1990
• Status: Expired due to Term

• Current Assignee: Motorola, Inc. (Motorola Solutions, Inc.)
• Original Assignee: Motorola, Inc. (Motorola Solutions, Inc.)
• Inventors: Garrison, Sidney C., Leivian, Robert, Anderson, Howard C., Peterson, William M.
• Primary Examiner(s): MacDonald, Allen R.
• Assistant Examiner(s): Joyner Roger S.
• Application Number:
  US07/462,203
• Time in Patent Office: 679 Days
• Field of Search: 364/513, 364/807, 307/201, 357/23.5, 382/15, 381/43
• US Class Current: 706/33
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology

Abstract

An artificial neural network is provided using a modular, self-organizing approach wherein a separate neural field is contained within each module for recognition and synthesis of particular characteristics of respective input and output signals thereby allowing several of these modules to be interconnected to perform a variety of operations. The first output and second input of one module is respectively coupled to the first input and second output of a second module allowing each module to perform a bi-directional transformation of the information content of the first and second input signals for creating first and second output signals having different levels of information content with respect thereto. In the upward direction, the first low-level input signal of each module is systematically delayed to create a temporal spatial vector from which a lower frequency, high-level first output signal is provided symbolic of the incoming information content. Since the first output signal contains the same relevant information as the first input signal while operating at a lower frequency, the information content of the latter is said to be compressed into a first high-level output signal. In the downward direction, a second output signal having a low-level of information content is synthesized from a second input signal having a high-level of information content. The second input signal is the best prediction of the first output signal available from the knowledge base of the module, while similarly the second output signal is the prediction of the first input signal.

Claims

1. A neural network responsive to first and second input signals applied at first and second inputs for providing first and second output signals at first and second outputs symbolic of the information content thereof, comprising:

first means responsive to the first output signal for providing a prediction of the first input signal of the neural network at an output;

second means having first and second outputs and first and second inputs coupled for receiving the first input signal of the neural network and said prediction of the first input signal respectively for monitoring the correlation between the first input signal and said prediction of the first input signal and gating a signal through said second means to said first output of said second means in response to first and second degrees of correlation between the first input signal and said prediction of the first input signal, wherein the first input signal is gated through to said first output of said second means upon detecting said first degree of correlation and said prediction of the first input signal is gated through to said first and second outputs of said second means upon detecting said second degree of correlation, said second output of said second means being coupled to the second output of the neural network;

third means for progressively delaying said signal gated through said second means to provide a spatial vector of said signal gated through said second means at a plurality of tap points; and

fourth means responsive to said spatial vector of said signal gated through said second means and to the second input signal of the neural network for providing the first output signal symbolic of the information content of the first and second input signals.

2. The neural network of claim 1 wherein the first output signal symbolic of the information content of the first and second input signals comprises a plurality of output signals and the second input signal comprises a plurality of input signals.

3. The neural network of claim 2 wherein said first means comprises a first neuron having a plurality of inputs coupled for receiving said plurality of outputs signals of the first output signal and having an output coupled to said second input of said second means.

4. The neural network of claim 3 wherein said first neuron comprises:

a plurality of synapses each having an input respectively coupled for receiving said plurality of outputs signals of the first output signal and having an output for providing a weighted output signal in response thereto; and

a summing junction having a plurality of inputs respectively coupled to said outputs of said plurality of synapses and having an output coupled to said output of said first neuron.

5. The neural network of claim 4 wherein said second means comprises:

a first gain controllable amplifier having an input coupled to said first input of said second means and having an output and a gain control input;

a second gain controllable amplifier having an input coupled to said output of said first neuron and an output coupled to said second output of said second means and having a gain control input;

a second neuron having first and second synapses and an output, said first synapse of said second neuron being coupled to said output of said first gain controllable amplifier, said second synapse of said second neuron being coupled to said output of said second gain controllable amplifier, said output of said second neuron being coupled to said first output of said second means for providing said signal gated through said second means as the weighted sum of the signals applied at said first and second synapses; and

a third neuron having first and second synapses coupled to said inputs of said first and second gain controllable amplifiers respectively for providing an output signal as the weighted sum of the signals applied thereto for controlling the gain of said first and second gain controllable amplifiers.

6. The neural network of claim 5 wherein said second means further comprises:

an averaging circuit responsive to said output signal of said third neuron for providing an output signal as the time average thereof;

a fourth neuron having first and second synapses and having an output, said first synapse of said fourth neuron being coupled for receiving said output signal of said averaging circuit, said output of said fourth neuron being coupled to said gain control input of said first gain controllable amplifier; and

a fifth neuron having first and second synapses and having an output, said first synapse of said fifth neuron being coupled to a power supply conductor, said second synapse of said fifth neuron being coupled to said output of said fourth neuron, said output of said fifth neuron being coupled to said gain control input of said second gain controllable amplifier and to said second synapse of said fourth neuron.

7. The neural network of claim 6 wherein said third means comprises a plurality of serially coupled delay circuits having an input coupled to said output of said second neuron and having a plurality of tap points coupled to said fourth means for providing said spatial vector of said signal gated through said second means.

8. The neural network of claim 7 wherein said fourth means comprises a first plurality of neurons each having a plurality of adaptive synapses coupled for receiving said spatial vector provided at said plurality tap points of said delay circuits and each having a plurality of non-adaptive synapses and an output, said outputs of said first plurality of neurons being respectively coupled to said plurality of inputs of said first neuron.

9. The neural network of claim 8 wherein said outputs of said first plurality of neurons are respectively coupled to said plurality of non-adaptive synapses thereof wherein the weights of said non-adaptive synapses of said first plurality of neurons are assigned in a predetermined manner.

10. The neural network of claim 9 wherein said fourth means further comprises:

a second plurality of neurons each having a plurality of non-adaptive synapses and an output, said plurality of non-adaptive synapses being coupled to a predetermined group of said outputs of said first plurality of neurons, said outputs of said second plurality of neurons being respectively coupled to predetermined ones of said plurality of output signals of the first output signal symbolic of the information content of the first and second input signals; and

a third plurality of neurons each having a plurality of non-adaptive synapses and an output, said plurality of non-adaptive synapses being coupled to said plurality of inputs of the second input signal, said outputs of said third plurality of neurons being respectively coupled to predetermined ones of said plurality of non-adaptive synapses of said first plurality of neurons.

11. The neural network of claim 10 wherein said one of said first plurality of neurons comprises a summing junction having a plurality of inputs respectively coupled to said outputs of said plurality of adaptive and non-adaptive synapses and having an output coupled to said output of one of said first plurality of neurons.

12. The neural network of claim 11 wherein said one of said plurality of non-adaptive synapses of said first plurality of neurons comprises a resistor coupled between the input of said one of said plurality of non-adaptive synapses and one of said plurality of inputs of said summing junction.

13. The neural network of claim 12 wherein one of said plurality of adaptive synapses of one of said first plurality of neurons comprises:

a first transistor having a drain, a source and a gate, said drain being coupled to the input of said one of said plurality of adaptive synapses, said source being coupled to one of said plurality of inputs of said summing junction;

a comparator having first and second inputs and an output, said first input of said comparator being coupled to said drain of said first transistor;

a scaling circuit coupled between said gate of said first transistor and said second input of said comparator for adjusting the potential therebetween;

a first switching circuit having first and second conduction terminals and having a control input, said first conduction terminal being coupled to a first source of operating potential, said control input being coupled to said output of said comparator;

a second switching circuit having first and second conduction terminals and having a control input, said first conduction terminal of said second switching circuit being coupled to said second conduction terminal of said first switching circuit, said second conduction terminal of said second switching circuit being coupled to a second source of operating potential; and

an inverter coupled between said output of said comparator and said control input of said second switching circuit.

14. The neural network of claim 13 wherein one of said plurality of adaptive synapses of said first plurality of neurons further comprises:

a third switching circuit having first and second conduction terminals and having a control input, said first conduction terminal of said third switching circuit being coupled to said second terminal of said first switching circuit;

a second transistor having a drain, a source and a gate, said source of said second transistor being coupled to a third source of operating potential, said gate of said second transistor being coupled to said second terminal of said third switching circuit, said drain of said second transistor being coupled to said gate of said first transistor;

a third transistor having a drain, a source and a gate, said drain and said gate of said third transistor being coupled together to a fourth source of operating potential, said source of said third transistor being coupled to said drain of said second transistor;

a fourth switching circuit having first and second conduction terminals and having a control input, said first conduction terminal of said fourth switching circuit being coupled to said control input of said third switching circuit; and

a multiplier circuit having first and second inputs and an output, said first input of said multiplier circuit being coupled to said output of said fourth neuron, said second input of said multiplier circuit being coupled to said output of one of said first plurality of neurons, said output of said multiplier circuit being coupled to said control input of said fourth switching circuit.

15. The neural network of claim 14 further comprising an oscillator having an output coupled to said second conduction terminal of said fourth switching circuit for providing an output signal having a predetermined frequency.

16. The neural network of claim 1 wherein said first degree of correlation indicates a degree of difference between the first input signal and said prediction of the first input signal.

17. The neural network of claim 16 wherein said second degree of correlation indicates a degree of similarity between the first input signal and said prediction of the first input signal.

18. A neural network comprising a plurality of serially coupled modules each being responsive to first and second input signals applied at first and second inputs for providing first and second output signals at first and second outputs symbolic of the information content thereof, wherein the first output and the second input of a first one of the plurality of serially coupled modules are respectively coupled to the first input and the second output of a second one of the plurality of serially coupled modules for increasing the recognition capacity thereof, each module comprising:

first means responsive to the first output signal for providing a prediction of the first input signal of the neural network at an output;

second means having first and second outputs and first and second inputs coupled for receiving the first input signal of the neural network and said prediction of the first input signal respectively for monitoring the correlation between the first input signal and said prediction of the first input signal and gating a signal through said second means to said first output of said second means in response to first and second degrees of correlation between the first input signal and said prediction of the first input signal, wherein the first input signal is gated through to said first output of said second means upon detecting said first degree of correlation and said prediction of the first input signal is gated through to said first and second outputs of said second means upon detecting said second degree of correlation, said second output of said second means being coupled to the second output of the neural network;

third means for progressively delaying said signal gated through said second means to provide a spatial vector of said signal gated through said second means at a plurality of tap points; and

fourth means responsive to said spatial vector of said signal gated through said second means and to the second input signal of the neural network for providing the first output signal symbolic of the information content of the first and second input signals.

19. The neural network of claim 18 wherein said first degree of correlation indicates a degree of difference between the first input signal and said prediction of the first input signal.

20. The neural network of claim 19 wherein said second degree of correlation indicates a degree of similarity between the first input signal and said prediction of the first input signal.

21. The neural network of claim 20 wherein the first output signal symbolic of the information content of the first and second input signals comprises a plurality of output signals and the second input signal comprises a plurality of input signals.

22. The neural network of claim 21 wherein said first means comprises a first neuron having a plurality of inputs coupled for receiving said plurality of outputs signals of the first output signal and having an output coupled to said second input of said second means.

23. The neural network of claim 22 wherein said first neuron comprises:

a plurality of synapses each having an input respectively coupled for receiving said plurality of outputs signals of the first output signal and having an output for providing a weighted output signal in response thereto; and

a summing junction having a plurality of inputs respectively coupled to said outputs of said plurality of synapses and having an output coupled to said output of said first neuron.

24. The neural network of claim 23 wherein said second means comprises:

a first gain controllable amplifier having an input coupled to said first input of said second means and having an output and a gain control input;

a second gain controllable amplifier having an input coupled to said output of said first neuron and an output coupled to said second output of said second means and having a gain control input;

a second neuron having first and second synapses and an output, said first synapse of said second neuron being coupled to said output of said first gain controllable amplifier, said second synapse of said second neuron being coupled to said output of said second gain controllable amplifier, said output of said second neuron being coupled to said first output of said second means for providing said signal gated through said second means as the weighted sum of the signals applied at said first and second synapses; and

a third neuron having first and second synapses coupled to said inputs of said first and second gain controllable amplifiers respectively for providing an output signal as the weighted sum of the signals applied thereto for controlling the gain of said first and second gain controllable amplifiers.

25. The neural network of claim 24 wherein said second means further comprises:

an averaging circuit responsive to said output signal of said third neuron for providing an output signal as the time average thereof;

a fourth neuron having first and second synapses and having an output, said first synapse of said fourth neuron being coupled for receiving said output signal of said averaging circuit, said output of said fourth neuron being coupled to said gain control input of said first gain controllable amplifier; and

a fifth neuron having first and second synapses and having an output, said first synapse of said fifth neuron being coupled to a power supply conductor, said second synapse of said fifth neuron being coupled to said output of said fourth neuron, said output of said fifth neuron being coupled to said gain control input of said second gain controllable amplifier and to said second synapse of said fourth neuron.

26. The neural network of claim 25 wherein said third means comprises a plurality of serially coupled delay circuits having an input coupled to said output of said second neuron and having a plurality of tap points coupled to said fourth means for providing said spatial vector of said signal gated through said second means.

27. The neural network of claim 26 wherein said fourth means comprises a first plurality of neurons each having a plurality of adaptive synapses coupled for receiving said spatial vector provided at said plurality tap points of said delay circuits and each having a plurality of non-adaptive synapses and an output, said outputs of said first plurality of neurons being respectively coupled to said plurality of inputs of said first neuron.

28. The neural network of claim 27 wherein said outputs of said first plurality of neurons are respectively coupled to said plurality of non-adaptive synapses thereof wherein the weights of said non-adaptive synapses of said first plurality of neurons are assigned in a predetermined manner.

29. The neural network of claim 28 wherein said fourth means further comprises:

a second plurality of neurons each having a plurality of non-adaptive synapses and an output, said plurality of non-adaptive synapses being coupled to a predetermined group of said outputs of said first plurality of neurons, said outputs of said second plurality of neurons being respectively coupled to predetermined ones of said plurality of output signals of the first output signal symbolic of the information content of the first and second input signals; and

a third plurality of neurons each having a plurality of non-adaptive synapses and an output, said plurality of non-adaptive synapses being coupled to said plurality of inputs of the second input signal, said outputs of said third plurality of neurons being respectively coupled to predetermined ones of said plurality of non-adaptive synapses of said first plurality of neurons.

30. The neural network of claim 29 wherein said one of said first plurality of neurons comprises a summing junction having a plurality of inputs respectively coupled to said outputs of said plurality of adaptive and non-adaptive synapses and having an output coupled to said output of one of said first plurality of neurons.

31. The neural network of claim 30 wherein said one of said plurality of non-adaptive synapses of said first plurality of neurons comprises a resistor coupled between the input of said one of said plurality of non-adaptive synapses and one of said plurality of inputs of said summing junction.

32. The neural network of claim 31 wherein one of said plurality of adaptive synapses of one of said first plurality of neurons comprises:

a first transistor having a drain, a source and a gate, said drain being coupled to the input of said one of said plurality of adaptive synapses, said source being coupled to one of said plurality of inputs of said summing junction;

a comparator having first and second inputs and an output, said first input of said comparator being coupled to said drain of said first transistor;

a scaling circuit coupled between said gate of said first transistor and said second input of said comparator for adjusting the potential therebetween;

a first switching circuit having first and second conduction terminals and having a control input, said first conduction terminal being coupled to a first source of operating potential, said control input being coupled to said output of said comparator;

a second switching circuit having first and second conduction terminals and having a control input, said first conduction terminal of said second switching circuit being coupled to said second conduction terminal of said first switching circuit, said second conduction terminal of said second switching circuit being coupled to a second source of operating potential; and

an inverter coupled between said output of said comparator and said control input of said second switching circuit.

33. The neural network of claim 32 wherein one of said plurality of adaptive synapses of said first plurality of neurons further comprises:

a third switching circuit having first and second conduction terminals and having a control input, said first conduction terminal of said third switching circuit being coupled to said second terminal of said first switching circuit;

a second transistor having a drain, a source and a gate, said source of said second transistor being coupled to a third source of operating potential, said gate of said second transistor being coupled to said second terminal of said third switching circuit, said drain of said second transistor being coupled to said gate of said first transistor;

a third transistor having a drain, a source and a gate, said drain and said gate of said third transistor being coupled together to a fourth source of operating potential, said source of said third transistor being coupled to said drain of said second transistor;

a fourth switching circuit having first and second conduction terminals and having a control input, said first conduction terminal of said fourth switching circuit being coupled to said control input of said third switching circuit; and

a multiplier circuit having first and second inputs and an output, said first input of said multiplier circuit being coupled to said output of said fourth neuron, said second input of said multiplier circuit being coupled to said output of one of said first plurality of neurons, said output of said multiplier circuit being coupled to said control input of said fourth switching circuit.

34. The neural network of claim 33 further comprising an oscillator having an output coupled to said second conduction terminal of said fourth switching circuit for providing an output signal having a predetermined frequency.