Pattern recognizer with independent feature learning

• US 6,058,206 A
• Filed: 12/01/1997
• Issued: 05/02/2000
• Est. Priority Date: 12/01/1997
• Status: Expired due to Fees

• Current Assignee: Chris Alan Kortge
• Original Assignee: Chris Alan Kortge
• Inventors: Kortge, Chris Alan
• Primary Examiner(s): Au, Amelia
• Assistant Examiner(s): Wu, Jingge
• Application Number:
  US08/980,838
• Time in Patent Office: 883 Days
• Field of Search: 382/157, 382/158, 382/159, 382/156, 706/20, 706/16, 706/25, 706/30
• US Class Current: 382/159
• CPC Class Codes: G06F 18/21 Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
• CPC Class Codes: G06N 3/045 Combinations of networks

Abstract

A pattern recognition device having modifiable feature detectors (28) which respond to a transduced input signal (26) and communicate a feature activity signal (30) to allow classification and an appropriate output action (70). A memory (40) stores a set of comparison patterns, and is used by an assigner (66) to find likely features, or parts, in the current input signal (26). Each part is assigned to a feature detector (28[m]) judged to be responsible for it. An updater (42) modifies each responsible feature detector (28[m]) so as to make its preferred feature more similar to its assigned part. The modification embodies a strong constraint on the feature learning process, in particular an assumption that the ideal features for describing the pattern domain occur independently. This constraint allows improved learning speed and potentially improved scaling properties.

A first preferred embodiment uses a group of noisy-OR type neural networks (50) to implement the feature detectors (28) and memory (40), and to obtain the parts by a soft segmentation of the current input signal (26). A second preferred embodiment maintains a lossless memory (40) separate from the feature detectors (28), and the parts consist of differences between the current input signal (26) and comparison patterns stored in the memory (40).

Claims

1. A device for recognizing and responding to physical patterns, comprising:

(a) transducer means for producing an input signal representing a physical pattern in an environment;

(b) a plurality of feature detectors responsive to said input signal, each feature detector having weight means for storing a representation of a preferred feature, for producing a feature activity signal representing degrees to which each of said preferred features exists in said input signal, and for producing a feature description signal representing said preferred features;

(c) classifier means responsive to said feature activity signal, for producing an output signal representing a system action corresponding to said input signal;

(d) effector means responsive to said output signal, for committing an action in said environment;

(e) memory means responsive to said input signal, for approximately storing a representation of said input signal, and for producing a retrieval signal representing previously stored input signals;

(f) assigner means responsive to said input signal and to said retrieval signal and to said feature description signal, for producing a part mapping signal representing a mapping between a plurality of parts and at least one responsible feature detector, such that each part corresponds to a likely feature of said input signal and of said previously stored input signals;

(g) updater means responsive to said part mapping signal, for modifying each of said responsible feature detectors so as to make its preferred feature more similar to its assigned part;

whereby the modification of each of said responsible feature detectors is largely independent of the modifications of the other feature detectors;

whereby said device can be effectively trained with fewer physical pattern examples than a device having correlated feature training.

2. The device of claim 1 wherein said part mapping signal represents a mapping between said plurality of parts and a plurality of responsible feature detectors, and is such that each responsible feature detector has a high correspondence to its assigned part relative to the other feature detectors.

3. The device of claim 2 wherein said memory means is responsive to said feature activity signal and to said feature description signal, and said retrieval signal is dependent upon said feature activity signal and upon said feature description signal.

4. The device of claim 3 wherein said feature detectors, and said classifier means, and said memory means, and said assigner means, and said updater means comprise executable instruction code on a digital computing machine.

5. The device of claim 3 wherein said feature detectors are implemented with a neural network, such that the weight means for each feature detector comprises an array of modifiable connections configured for receiving said input signal.

6. The device of claim 5 wherein said neural network comprises executable instruction code on a digital computing machine.

7. The device of claim 5 wherein at least one unit of said neural network acts according to a noisy-OR function.

8. The device of claim 7, further including means for storing a contribution to the activation probability of said at least one unit, such that said contribution may be accessed on a plurality of activation cycles.

9. The device of claim 8 wherein said contribution is a sum over each inactive child unit of a negative logarithm of a quantity representing one minus the weight from said at least one unit to said inactive child unit.

10. The device of claim 5 wherein said assigner means is configured to perform a soft segmentation of said input signal to obtain said parts.

11. The device of claim 2 wherein said memory means is a lossless storage device.

12. The device of claim 11 wherein each of said parts is a difference vector representing a difference between said input signal and a previously stored comparison pattern represented by said retrieval signal.

13. The device of claim 12 wherein said assigner means is configured to assign each of said parts to a winning feature detector, said winning feature detector having the preferred feature which has a minimum distance from said difference vector.

14. The device of claim 2 wherein said updater means is configured to modify each of said responsible feature detectors so as to make its preferred feature move to a new input space location which is substantially along the vector from its current input space location to the input space location of its assigned part.

15. A method for creating a pattern recognition device, comprising the steps of:

(a) providing transducer means for producing an input signal representing a physical pattern in an environment;

(b) providing a plurality of feature detectors responsive to said input signal, each feature detector having weight means for storing a representation of a preferred feature, for producing a feature activity signal representing degrees to which each of said preferred features exists in said input signal, and for producing a feature description signal representing said preferred features;

(c) providing classifier means responsive to said feature activity signal, for producing an output signal representing a system action corresponding to said input signal;

(d) providing effector means responsive to said output signal, for committing an action in said environment;

(e) providing memory means for approximately storing input patterns, and for producing a retrieval signal representing previously stored input patterns;

(f) using said memory means to approximately store a sequence of comparison patterns;

(g) providing a training pattern;

(h) identifying a plurality of parts in said training pattern, such that each part corresponds to a likely feature of said training pattern and of said comparison patterns;

(i) assigning each of said parts to a corresponding responsible feature detector;

(j) modifying each of said responsible feature detectors so as to make its preferred feature substantially directly more similar to its assigned part;

(k) training said feature detectors by repeating steps (g) through (j) on a significant portion of a training set until a training criterion is reached;

whereby the modification of each of said responsible feature detectors is largely independent of the modifications of the other feature detectors;

whereby said method allows effective creation of a pattern recognition device with fewer pattern presentations than a device having correlated feature training.

16. The method of claim 15, further including the steps of:

(l) repeating steps (a) through (d) to create a comparable pattern recognition device;

(m) transferring the preferred feature of at least one of the trained feature detectors to at least one corresponding feature detector of said comparable pattern recognition device.

17. The method of claim 15 wherein said memory means is responsive to said feature activity signal and to said feature description signal, and said retrieval signal is dependent upon said feature activity signal and upon said feature description signal.

18. The method of claim 17 wherein said feature detectors are implemented with a neural network, such that the weight means for each feature detector comprises an array of modifiable connections configured for receiving said input signal.

19. The method of claim 18 wherein at least one unit of said neural network acts according to a noisy-OR function.

20. A device for recognizing and responding to physical patterns, comprising:

(a) a transducer capable of producing an input signal representing a physical pattern in an environment;

(b) a plurality of feature detectors each responsive to said input signal, each feature detector having weight storage capable of representing a preferred feature, each of said feature detectors being capable of producing a feature activity signal element representing a degree to which its preferred feature exists in said input signal, and being capable of producing a feature description signal element representing its preferred feature;

(c) a classifier responsive to each said feature activity signal element, capable of producing an output signal representing a system action corresponding to said input signal;

(d) an effector responsive to said output signal, capable of committing an action in said environment;

(e) a memory responsive to said input signal, capable of approximately storing a representation of said input signal, and capable of producing a retrieval signal representing previously stored input signals;

(f) an assigner responsive to said input signal and to said retrieval signal and to each of said feature description signal elements, capable of producing a part mapping signal representing a mapping between a plurality of parts and a plurality of responsible feature detectors, such that each part corresponds to a likely feature of said input signal and of said previously stored input signals, and such that each responsible feature detector has a high correspondence to its assigned part relative to the other feature detectors;

(g) an updater responsive to said part mapping signal, capable of modifying each of said responsible feature detectors so as to make its preferred feature vector move substantially directly toward its assigned part vector;

whereby the modification of each of said responsible feature detectors is largely independent of the modifications of the other feature detectors;

whereby said device can be effectively trained with fewer physical pattern examples than a device having correlated feature training.