TRAINABLE ENTROPY SYSTEM

• US 3,702,986 A
• Filed: 07/06/1970
• Issued: 11/14/1972
• Est. Priority Date: 07/06/1970
• Status: Expired due to Term

• Current Assignee: Fredrick James Taylor, William C. Choate
• Original Assignee: Fredrick James Taylor, William C. Choate
• Inventors: Choate, William C., Taylor, Fredrick James
• Primary Examiner(s): Henon, Paul J.
• Assistant Examiner(s): Chirlin, Sydney R.
• Application Number:
  US05/052,611
• Time in Patent Office: 862 Days
• Field of Search: 340/172.5,146.3T
• US Class Current: 712/25
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

A system is comprised of a series of trainable nonlinear processors in cascade. The processors are trained in sequence as follows. In a first phase of the sequence, a set of input signals comprising input information upon which the system is to be trained and a corresponding set of desired responses to these input signals are introduced into the first processor. When the first processor has been trained over the entire set, a second phase commences in which a second set of input signals along with the output of the first processor and corresponding set of desired responses are introduced into the second processor. During this second phase the input signals to the first and second processors are in sequential correspondence. In one embodiment of the invention the set of input signals to the first processor comprises the same set of input signals being introduced into the second processor delayed by a fixed time interval. The training sequence continues until all processors in the series have been trained in a similar manner. The input to the kth or last processor will comprise a set of input signals, the desired output responses to those input signals and the output of the (k-l)the processor. The input to each preceding processor will th separate sets of input signals which in one embodiment are the set of input signals to the kth processor, retrogressively, delayed in time by one additional time interval and the output of the previous processor. The system may be looked upon as a minimum entropy system in which the entropy or measure of uncertainty is decreased at each stage. When all of the processors have been trained, the system is ready for execution and the actual output of the last stage is a minimum entropy approximation of a proper desired output when an input signal, without a corresponding desired response, is introduced into the completed system of cascaded processors.

Claims

1. A trainable system of cascaded processors comprised of:

a plurality of non-linear signal processors in cascade, each processor including;

a. means for applying at least one input signal provided by peripheral equipment and one probabilistic signal generated by the previous processor in the casecade thereto;

b. means for applying at least one desired output signal provided by peripheral equipment thereto when such processor is operated in a training mode;

c. a multi-level tree-arranged storage array having at least a root level and a leaf level;

d. means for defining a path through the levels of the tree from said input and probabilistic signals;

said leaf level including means for accumulating and storing the number of occurrences that each desired output signal was associated with each of said defined paths during training; and

e. means for generating at least one output signal comprising the probabilistic signal for the subsequent processor in the cascade when the processor is operated in an execution mode.

2. The system of claim 1 wherein said probabilistic signal for the subsequent processor in the cascade is generated from said accumulated and stored number of occurrences when the processor is operated in an execution mode.

3. The system of claim 2 including preprocessor means for encoding said at least one input signal into one or more key components, and each of said processors including:

means for encoding said probabilistic signal into one or more key components, all of said key components being utilized for defining said path through the levels of said tree-arranged storage array.

4. The system of claim 3 wherein said preprocessor means includes:

a. means for providing An initial signal for the first processor in the cascade taking the place in structure of the probabilistic signal generated by each of the processors for the subsequent processor in the cascade, and b. means for encoding said initial signal into one or more key components.

5. The system of claim 4 including means for sequentially comprising the key components of the present input and probabilistic signals with the key components of input and probabilistic signals which have previously defined paths through the levels of said three-arranged storage array.

6. The system of claim 1 wherein said root level is directly addressable and said leaf level is addressable in a defined path extending from a storage unit of said root level.

7. The system of claim 6 including:

preprocessor means for encoding said at least one input signal into one I key component, and each of said processors including;

means for encoding said probabilistic signal into one X key component, said I key component providing means for directly addressing a storage unit of said root level and said X key component providing means for addressing a storage unit of said leaf level extending in a path from the addressed storage unit in said root level.

8. The system of claim 7 wherein said preprocessor means includes:

a. means for generating an initial signal for the first processor in the cascade analogous to the probabilistic signal generated by each of the processors for the subsequent processor in the cascade, and b. means for encoding said initial signal into one X key component.

9. The system of claim 8 including means for sequentially comprising I and X key components of the present input and probabilistic signals with the I and X key components of input and probabilistic signals which have previously defined paths through the root and leaf levels of said tree-arranged storage array.

10. The system of claim 2 wherein the last processor in the cascade includes means for converting the probabilistic signal generated thereby to an actual output signal which is a best estimate of a desired response to an input signal applied to the first processor when the system is operated in an execution mode.

11. In a method of operating a trainable system of cascaded processors, the steps of:

a. training a (k-1)th trainable non-linear processor to store therein (k-1)th statistical data based upon an applied first input signal and an applied desired response to such first input signal, b. executing said (k-1)th processor to generate from such stored (k-1)th statistical data a (k-1)th probabilistic signal which is a statistical estimate of the desired response to a second applied input signal, c. training a kth trainable non-linear processor to store therein kth statistical data based upon a third input signal comprising said first probabilistic signal generated by said (k-1)th processor and applied desired response to such third applied input signal applied thereto, d. executing said (k-1)th processor to generate from such stored (k-1)th statistical data a second probabilistic signal which is a statistical estimate of the desired response to a fourth applied input signal, and e. executing said kth processor to generate from such stored second statistical data an actual output signal which is a lower entropy estimate of the desired response to said fourth input signal when a fifth input signal comprising said second probabilistic signal generated by said (k-1)th processor is applied thereto.

12. The process of claim 11 including the step of delaying at least one signal comprising said third input signal to provide at least one signal comprising said second input signal.

13. In a method of operating a trainable system of cascaded processors, the stEps of:

a. storing first statistical data based upon an applied first input signal and an applied desired response to such first input signal in a first trainable non-linear processor operated in a training mode, b. generating from such stored first statistical data a first probabilistic signal which is a statistical estimate of the desired response to a second signal applied to said first processor operated in an execution mode, c. storing second statistical data based upon a third applied input signal comprising said first probabilistic signal generated by said first processor and applied desired response to such third input signal in a second trainable non-linear processor operated in a training mode, d. generating from such stored first statistical data a second probabilistic signal which is a statistical estimate of the desired response to a fourth signal applied to said first processor operated in an execution mode, and e. generating from such stored second statistical data an actual output signal which is a lower entropy estimate of the desired response to said fourth signal when a fifth input signal comprising said second probabilistic signal generated by said first processor is applied to said second processor operated in an execution mode.

14. The method of claim 13 wherein the method of storing first statistical data includes the steps of:

a. accumulating and storing the number of occurrences that each possible desired response has been associated with each same first input signal, and the method of storing second statistical data includes the steps of;

b. accumulating and storing the number of occurrences that each possible desired response has been associated with each same third input signal.

15. In a method of operating a trainable system of cascaded processors, the steps of:

a. encoding a first applied input signal into a plurality of key components, b. defining a path through the levels of a tree-arranged storage array of a first trainable non-linear signal processor said storage array having a plurality of levels and said path defined to a storage unit in the leaf level thereof in accordance with at least two of said key components of said first encoded input signal, c. storing first statistical data based upon an applied desired response to such first input signal in said storage unit at the leaf level of the tree-arranged storage array of said first processor, d. encoding a second applied input signal into a plurality of key components, e. defining a path through the levels of the tree-arranged storage array of said first processor to a storage unit in the leaf level thereof in accordance with at least two of said key components of said second encoded input signal, f. generating from such stored first statistical data a first probabilistic signal which is a statistical estimate of the desired response to said second input signal, g. encoding a third applied input signal comprising said first probabilistic signal generated by said first processor into a plurality of key components, h. defining a path through the levels of a tree-arranged storage array of a second trainable non-linear signal processor said storage array having a plurality of levels and said path defined to a storage unit in the leaf level thereof in accordance with at least two of said key components of said third encoded input signal, i. storing second statistical data based upon an applied desired response to such third input signal in said storage unit at the leaf level of the tree-arranged storage array of said second processor, j. encoding a fourth applied input signal into a plurality of key components, k. defining a path through the levels of the tree-arranged storage array of said first processor to a storage unit in the leaf level thereof in accordance with at least two of said key components of said fourth encoded input signal, l. generating from such stored first statistical data a secOnd probabilistic signal which is a statistical estimate of the desired response to said fourth input signal, m. encoding a fifth applied input signal comprising said second probabilistic signal generated by said first processor into a plurality of key components, n. defining a path through the levels of the tree-arranged storage array of said second processor to a storage unit in the leaf level thereof in accordance with at least two of said key components of said fifth encoded input signal, and o. generating from such stored second statistical data an actual output signal which is a lower entropy estimate of the desired response to said fourth input signal.

16. The method of claim 15 wherein the method of defining a path through the levels of the tree-arranged storage arrays of said first and second processors includes the steps of:

a. directly addressing a storage unit in a first root level of the storage array, and b. addressing the storage unit in said leaf level in a defined path extending from the storage unit of said root level.

17. The method of claim 15 wherein the method of storing first statistical data includes the steps of:

a. accumulating and storing the number of occurrences that each possible desired response has been associated with each same first input signal, and the method of storing second statistical data includes the steps of;

b. accumulating and storing the number of occurrences that each possible desired response has been associated with each same third input signal.

18. In a method of operating a trainable system of cascaded processors, the steps of:

a. encoding a first applied input signal into a plurality of key components, b. defining a path through the levels of a tree-arranged storage array of a first trainable non-linear signal processor said storage array having a plurality of levels and said path defined to a storage unit in the leaf level thereof in accordance with at least two of said key components of said first encoded input signal, c. sequentially comparing the key components of said first input signal with the key components of input signals which have previously defined paths through the levels of the tree-arranged storage array of said first processor. d. storing statistical data based upon an applied desired response to such first input signal in said storage unit at the leaf level of the tree-arranged storage array of said first processor when no such path has been previously defined. e. updating the statistical data stored in said storage unit at the leaf level of the tree-arranged storage array of said first processor when such a path has been previously defined. f. encoding a second applied input signal into a plurality of key components, g. defining a path through the levels of the tree-arranged storage of said first processor to a storage unit in the leaf level thereof in accordance with at least two of said key components of said second encoded input signal, h. sequentially comparing the key components of said second input signal with the key components of input signals which have previously defined paths through the levels of the tree-arranged storage array of said first processor to locate first statistical data which provides a best statistical estimate for said first processor of the desired response to said second input signal. i. generating from such located first statistical data a first probabilistic signal, j. encoding a third applied input signal comprising said first probabilistic signal generated by said first processor into a plurality of key components, k. defining a path through the levels of a tree-arranged storage array of a second trainable non-linear signal processor said storage array having a plurality of levels and said path defined to a storage unit in the leaf level thereof in accordance with at least two of said key components of said third encoded input signal, l. sequentially comparing the keY components of said third input signal with the key components of input signals which have previously defined paths through the levels of the tree-arranged storage array of said second processor. m. storing statistical data based upon an applied desired response to such third input signal in said storage unit at the leaf level of the tree-arranged storage array of said second processor when no such path has been previously defined. n. updating the statistical data stored in said storage unit at the leaf level of the tree-arranged storage array of said second processor when such a path has been previously defined. o. encoding a fourth applied input signal into a plurality of key components, p. defining a path through the levels of the tree-arranged storage array of said first processor to a storage unit in the leaf level thereof in accordance with at least two of said key components of said fourth encoded input signal, q. sequentially comparing the key components of said fourth input signal with the key components of input signals which have previously defined paths through the levels of the tree-arranged storage array of said first processor to locate first statistical data which provides a best statistical estimate for said first processor of the desired response to said fourth input signal. r. generating from such located first statistical data a second probabilistic signal, s. encoding a fifth applied input signal comprising said second probabilistic signal generated by said first processor into a plurality of key components, t. defining a path through the levels of the tree-arranged storage array of said second processor to a storage unit in the leaf level thereof in accordance with at least two of said key components of said fifth encoded input signal, u. sequentially comparing the key components of said fifth input signal with the key components of input signals which have previously defined paths through the levels of the tree-arranged storage array of said second processor to locate second statistical data which provides a lower entropy estimate of the desired response to said fourth input signal than provided by said first processor. v. generating from such located second statistical data an actual output signal.

19. The method of claim 18 wherein the method of storing first statistical data includes the steps of:

a. accumulating and storing the number of occurrences that each possible desired response has been associated with each same first input signal, and the method of storing second statistical data includes the steps of;

b. accumulating and storing the number of occurrences that each possible desired response has been associated with each same third input signal.

20. In a method of operating a system of cascaded processors, the steps of:

a. encoding a first applied input signal into a plurality of key components, b. defining a path through the levels of a tree-arranged storage array of a first non-linear signal processor said storage array having a plurality of levels and said path defined to locate a storage unit in the leaf level thereof in accordance with at least two of said key components of said first encoded input signal, c. generating from first statistical data stored in such storage unit of the first processor a probabilistic signal which is a statistical estimate of the desired response to said first input signal, d. encoding a second applied input signal comprising said probabilistic signal generated by said first processor into a plurality of key components, e. defining a path through the levels of a tree-arranged storage array of a second trainable non-linear signal processor said storage array having a plurality of levels and said path defined to locate a storage unit in the leaf level thereof in accordance with at least two of said key components of said third encoded input signal, f. generating from such stored second statistical data stored in such storage unit of the second processor an actual output signal which is a lower entropy estimate of the desired response to said first input signal.

21. The method of claim 15 wherein the method of defining a path through the levels of the tree-arranged storage arrays of said first and second processors includes the steps of:

a. directly addressing a storage unit in a first root level of the storage array, and b. addressing the storage unit in said leaf level in a defined path extending for the storage unit of said root level.

22. In a method of operating a system of cascaded processors, the steps of:

a. encoding a first applied input signal into a plurality of key components, b. defining a path through the levels of a tree-arranged storage array of a first trainable non-linear signal processor said storage array having a plurality of levels and said path defined to a storage unit in the leaf level thereof in accordance with at least two of said key components of said first encoded input signal, c. sequentially comparing the key components of said first input signal with the key components which define paths through the levels of the tree-arranged storage array of said first processor to locate first statistical data which provides a best statistical estimate for said first processor of the desired response to said first input signal. d. generating from such located first statistical data a probabilistic signal, e. encoding a second applied input signal comprising said probabilistic signal generated by said first processor into a plurality of key components, f. defining a path through the levels of a tree-arranged storage array of a second trainable non-linear signal processor said storage array having a plurality of levels and said path defined to a storage unit in the leaf level thereof in accordance with at least two of said key components of said encoded input signal, g. sequentially comparing the key components of said second input signal with the key components which define paths through the levels of the tree-arranged storage array of said second processor to locate second statistical data stored in such storage unit of the second processor which provides a lower entropy estimate of the desired response to said first input signal than provided by said first processor. h. generating from such located second statistical data an actual output signal.

23. A trainable system of cascaded processors comprised of:

a plurality of non-linear signal processors in cascade, each processor including;

a. means for applying at least one input signal from an external source and at least one probabilistic signal generated by the previous processor in the cascade thereto;

b. a multi-level tree-arranged storage array having at least a root level and a leaf level;

c. means for defining a path through the levels of the tree in accordance with said input and probabilistic signals; and

d. means for generating at least one output signal comprising a probabilistic signal for the subsequent processor in the cascade.

24. The system of claim 23 wherein each processor is operable in a training mode additionally including means for applying at least one desired output signal from an external source thereto when such processor is operated in said training mode.

25. The system of claim 24 wherein said leaf level includes means for accumulating and storing the number of occurrences that each desired output signal was associated with each of said defined paths during training.

26. The system of claim 25 wherein said means for generating at least one output signal is responsive during execution to said means for storing the number of occurrences that each desired output signal was associated with each of said defined paths during training.