System and method for cognitive memory and auto-associative neural network based pattern recognition

• US 20060173560A1
• Filed: 10/07/2005
• Published: 08/03/2006
• Est. Priority Date: 10/07/2004
• Status: Active Grant

• Current Assignee: Bernard Widrow
• Original Assignee: Bernard Widrow
• Inventors: Widrow, Bernard
• Granted Patent: US 7,702,599 B2
• Time in Patent Office: Days
• US Class Current: 700/48
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 18/2135 based on approximation criteria, e.g. principal component analysis
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06V 10/70 using pattern recognition or machine learning optical pattern recognition or electronic computations therefor G06V10/88
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: Y10S 706/934 Information retrieval or Information management

Abstract

Designs for cognitive memory systems storing input data, images, or patterns, and retrieving it without knowledge of where stored when cognitive memory is prompted by query pattern that is related to sought stored pattern. Retrieval system of cognitive memory uses autoassociative neural networks and techniques for pre-processing query pattern to establish relationship between query pattern and sought stored pattern, to locate sought pattern, and to retrieve it and ancillary data. Cognitive memory, when connected to computer or information appliance introduces computational architecture that applies to systems and methods for navigation, location and recognition of objects in images, character recognition, facial recognition, medical analysis and diagnosis, video image analysis, and to photographic search engines that when prompted with a query photograph containing faces and objects will retrieve related photographs stored in computer or other information appliance, and will identify URL'"'"'s of related photographs and documents stored on the World Wide Web.

Claims

1. A cognitive memory system for storing in the form of patterns input data or information, wherein subsequent retrieval of said patterns from said cognitive memory system is accomplished in response to related, but not necessarily identical, input query patterns.

2. A cognitive memory system for storing sensory input data and patterns, said data and patterns stored in memory folders, each memory folder capable of storing a plurality of patterns, storing simultaneously inputted patterns from a plurality of sensors, storing other ancillary data, a retrieval system capable of retrieving the contents of each said memory folder when presented with a related prompt pattern, derived from an input query pattern, and a system capable of relating said prompt pattern to one of the patterns stored in said memory folders.

3. The cognitive memory system of claim 2, wherein said contents of each said memory folder could serve as a “

self-prompt”

query pattern or patterns to initiate further retrieval of contents of additional memory folders of said cognitive memory system of claim 2.

4. The cognitive memory system of claim 2, wherein said system:

(a) is capable of relating said prompt patterns to one or more of said data and patterns stored in said memory folders; and

(b) utilizes an autoassociative neural network trained with all patterns stored in all said memory folders of said cognitive memory system, said trained autoassociative neural network being capable of indicating if said prompt pattern matches one of the patterns stored in said memory folders, a match indicated by said autoassociative neural network triggering a component-by-component comparison between the prompt pattern and all of the patterns in said memory folders of said cognitive memory system, wherein a component-by-component match with one pattern in one memory folder results in the entire contents of said memory folder being delivered as the memory output.

5. A cognitive memory system capable of receiving input data, images, or patterns, storing said input data, images or patterns wherever storage space is available, and retrieving said input data, images, or patterns upon receipt of a prompt input, the cognitive memory system further comprising:

(a) a conventional memory organized into memory folders for storing said input data, images or patterns;

(b) a trainable autoassociative neural network connected to said conventional memory to receive said input data, images, or patterns as training patterns;

(c) a stored adaptive algorithm for training said autoassociative neural network with said training patterns;

(d) at least one pre-processor(s) that modify input query images or patterns in order to generate multiple prompt patterns from a single input query pattern;

(e) means for testing each of said multiple prompt patterns, utilizing said autoassociative neural network, to select the successful prompt pattern that matches most closely to a pattern stored in one of the memory folders;

(f) means for scanning said memory folders to find a hit memory folder that stores the pattern that most closely matches said successful prompt pattern; and

(g) means for delivering the entire contents of said “

hit”

memory folder as the output of said cognitive memory system.

6. The cognitive memory system of claim 5, wherein said autoassociative neural network comprises a multi-layer perceptron that is trained by the backpropagation algorithm.

7. The cognitive memory system of claim 5, wherein said successful prompt pattern is selected from said multiple prompt patterns by:

(i) applying in sequence said multiple prompt patterns to said autoassociative neural network, (ii) measuring the mean square error between the input and output patterns of said autoassociative neural network, (iii) selecting the prompt pattern with the lowest mean square error, and (iv) calling this prompt pattern successful if its mean square error is below a pre-determined threshold.

8. The cognitive memory system of claim 5, wherein said hit memory folder is selected from all said memory folders by:

(i) comparing in sequence all training patterns of all memory folders to the successful prompt pattern, (ii) measuring the mean square error between each training pattern and the successful prompt pattern, and (iii) selecting the hit memory folder as the memory folder containing the pattern whose difference from said successful prompt pattern has the smallest mean square error of all the patterns in all of said memory folders.

9. The cognitive memory system of claim 5, wherein said input data, images, or patterns comprise visual patterns.

10. The cognitive memory system of claim 5, wherein said prompt patterns are derived from a single visual query pattern by pre-processing, including a pre-processing selected from the set consisting of at least one of a rotation, a translation, a scaling, a brightness change, a contrast change, a frequency or spatial frequency filter, and any combination of these.

11. The cognitive memory system of claim 5, wherein said input data, images, or patterns comprise both visual and auditory patterns.

12. The cognitive memory system of claim 5, wherein said prompt patterns are derived from:

(a) a single auditory input query pattern by pre-processing, including time dilation and contraction, amplitude scaling, spectrum-analysis, filtering, and translation in time and any combination of these; and

(b) a single visual input query pattern by pre-processing, including a rotation, a translation, a scale change, a brightness change, a contrast change, a frequency or spatial frequency filtering, and any combination of these.

13. The cognitive memory system of claim 5, wherein said cognitive memory system can be expanded in memory capacity by replicating said cognitive memory system, adding more segments, each segment having its own memory folders and autoassociative neural network, wherein:

(a) input data, images or patterns arrive at the inputs of all segments on a common data bus;

(b) each segment makes empty memory folders available for input data recording;

and

(c) prompt patterns arrive at all the same segments on a common data bus for simultaneous sensing of said autoassociative neural networks, so that access time does not increase as size of memory is scaled up.

14. A cognitive memory system as in claim 5, wherein said system is applied as an aircraft navigation system, and (a) wherein said input data, images or patterns are multiple photographs of a geographic area of the Earth, wherein each photograph is stored together with its geographic location in a single folder;

and the system further comprising

(a) a telescope on an aircraft looking down toward the earth; and

(b) a window scanning over the field of view of said telescope, wherein said window scans by rotation, translation, scaling, brightness adjustment, and contrast adjustment, to provide prompt patterns for said cognitive memory system; and

wherein the track of the aircraft, comprising a sequence of determined locations and headings of the aircraft, is calculated from the amount of translation and rotation required to find successive successful prompt patterns for said cognitive memory system and the geographic locations of the photographs of the Earth in the hit memory folders of said cognitive memory system.

15. A cognitive memory system as in claim 5, wherein said system is applied as a satellite or aerial surveillance system, and wherein:

(a) said input data, images, or patterns to be stored in the memory folders of said cognitive memory system are satellite or aerial photographs of the objects of interest, and the respective identities of said objects of interest are stored as ancillary data, wherein the autoassociative neural network of said cognitive memory system is trained with said input data, images, or patterns stored in said memory folders; and

said system further comprising;

(b) a telescope or other optical sensor device, which is onboard an aircraft or on a satellite, looking down toward the earth taking photographs or collecting images of the earth to provide said input data, images, or patterns to said cognitive memory system; and

(c) a window scanning over said photographs taken of the field of view of said telescope, wherein said window scans by rotation, translation, scaling, brightness adjustment, and contrast adjustment, to provide prompt patterns for said cognitive memory system;

wherein the identities of objects of interest that may be present in said satellite or aerial photographs are delivered as outputs of the “

hit”

memory folders of said cognitive memory system, and whereby locations and orientations of objects of interest that may be present in said satellite or aerial photographs are calculated from the amount of translation and rotation required to find the successful prompt patterns for said cognitive memory system.

16. A cognitive memory system as in claim 5, wherein:

(a) the system is applied as a character recognition system, and wherein said input data, images, or patterns to be stored in said memory folders are written or printed characters, logos, street traffic signs, and/or other character or symbolic carrying media, wherein many versions of each type of said characters stored in a separate memory folder, and the identification of each said character is stored as ancillary data in the same said memory folder as said versions of said character, wherein variations of each character corresponding to distortion, perspective, rotation, translation, brightness adjustment, scale, contrast adjustment, partial occlusion, wherein the autoassociative neural network of said cognitive memory system is trained with said input data, images, or patterns stored in said memory folders; and

the system further comprising;

(b) a camera photographing characters on paper, on other media, or elsewhere to provide said input data, images or patterns to said cognitive memory system; and

(c) a window scanning over the photographs taken by said camera wherein said window scans by rotation, translation, scaling, brightness adjustment, and contrast adjustment, to provide the prompt patterns for said cognitive memory system;

wherein the identities of said characters that may be present in said photographs are delivered as outputs of “

hit”

memory folders of said cognitive memory system, and whereby locations, orientations, and relative positions of said characters that may be present in said photographs are calculated from the amount of translation and rotation required to find the successful prompt patterns for said cognitive memory system.

18. A cognitive memory system as in claim 5, wherein:

(a) the system is applied as a human face recognition system and wherein said input data, images, or patterns to be stored in the memory folders of said cognitive memory system are high-resolution photographs of persons of interest, including their faces, wherein one or many photographs of the same person are stored in the same memory folder, and the identity of each said person is stored as ancillary data in the same said memory folder as said one or many photographs of said person, wherein the autoassociative neural network of said cognitive memory system is trained with said input data, images, or patterns stored in said memory folders;

(b) high-resolution photographs of said persons of interest received by said cognitive memory system providing said input data, images, or patterns to said cognitive memory system; and

said system further comprising;

(c) means for producing or receiving a low-resolution versions of said high-resolution photographs stored in said memory folders, said low-resolution photographs versions having lower spatial resolution than said high-resolution photographs;

(d) a low-resolution autoassociative neural network trained by said low-resolution versions of said high-resolution photographs stored in memory folders;

(e) means providing or receiving high-resolution photographs of persons to provide high-resolution input query patterns for said cognitive memory system;

(f) means for producing or receiving low-resolution versions of said high-resolution input query patterns;

(g) a low-resolution window scanning over said low-resolution versions of said high-resolution input query patterns, wherein said low-resolution window scans and optionally applies at least one of a rotation, a translation, a scaling, a brightness adjustment, and a contrast adjustment, to provide prompt patterns to said low-resolution autoassociative neural network, whereby the location, orientation and scale of detected facial images within said high-resolution input query patterns are calculated from the amount of any translation, rotation and scaling required to find the successful prompt patterns for said low-resolution autoassociative neural network;

(h) a high-resolution window, having an initial location, orientation and scale corresponding to said location, orientation and scale of said detected facial images as determined by said low-resolution autoassociative neural network, and scanning over said high-resolution input query patterns, wherein said high-resolution window scans and optionally applies at least one of a rotation, a translation, a scaling, a brightness adjustment, and a contrast adjustment to provide high-resolution prompt patterns for said autoassociative neural network of said cognitive memory system;

(i) successful high-resolution prompt patterns that produce errors lower than a pre-set threshold when sensed by said autoassociative neural network;

(j) means for comparing said successful high-resolution prompt patterns on a pixel-by-pixel or pixel group by pixel group basis with patterns stored in said memory folders, wherein said memory folder that contains patterns that best match each of said successful high-resolution prompt patterns is identified as the hit folder; and

(k) means for delivering as output the contents of said hit folder, which contains the identity of the stored facial images that correspond with the faces in said input query photograph.

23. A cognitive memory system as in claim 18, wherein:

mid-resolution photographs, intermediate in resolution between the high-resolution and low-resolution photographs, of said persons of interest received by said cognitive memory system providing said input data, images, or patterns to said cognitive memory system; and

said system further comprising;

means for producing or receiving a mid-resolution versions of said mid-resolution photographs stored in said memory folders, said mid-resolution photographs versions having lower spatial resolution or detail than said high-resolution photographs and a higher spatial resolution or more detail than said low-resolution photographs;

a mid-resolution autoassociative neural network trained by said mid-resolution versions of said high-resolution photographs stored in memory folders;

means for producing or receiving mid-resolution versions of said high-resolution input query patterns; and

a mid-resolution window scanning over said mid-resolution versions of said high-resolution input query patterns, wherein said mid-resolution window scans and optionally applies at least one of a rotation, a translation, a scaling, a brightness adjustment, and a contrast adjustment, to provide prompt patterns to said low-resolution autoassociative neural network, whereby the location, orientation and scale of detected facial images within said high-resolution input query patterns are calculated from the amount of any translation, rotation and scaling required to find the successful prompt patterns for said low-resolution autoassociative neural network.

17. A human face recognition system for recognizing person'"'"'s faces contained in a query photograph comprising:

(a) images of faces of interest and respective ancillary data stored in memory folders of a computer or other information appliance, where said faces are trained into an autoassociative neural network;

(b) a variable window that can be scanned over said query photograph by, for example, translation, rotation, scaling, brightness adjustment, or contrast adjustment that provides prompt input patterns to said autoassociative neural network;

(c) means for measuring the error between the input and output patterns of said autoassociative neural network, and comparing said error to a pre-set threshold;

(d) means for selecting successful prompt patterns whose measured errors are less than said pres-set threshold value;

(e) means for comparing on a pixel-by-pixel basis the successful,prompt patterns with the patterns stored in said memory folders;

(f) means for selecting the hit memory folder that contains patterns that most closely match the successful prompt patterns; and

(g) means for delivering the contents of the hit folder as the output, where said contents include the identity of the person'"'"'s face that most closely matches a person'"'"'s face in said query photograph.

19. A surveillance system comprising:

(a) at least one security checkpoint system, each security checkpoint system comprising;

(i) a human face recognition system functioning as a detection system;

(ii) at least one camera or sensor for obtaining high-resolution photographs at a security checkpoint that provide the high-resolution input query patterns for said detection system; and

(iii) an alarm or notification system generating at least one alarm or notification upon the identification on an alarm or notification event; and

wherein the persons of interest passing through said security checkpoint system are detected and identified by said detection system, and said alarm or notification system can be activated based on the outcome of the detection and identification process.

20. A surveillance system as in claim 19, wherein the human face recognition system further comprising:

a cognitive memory system capable of receiving input data, images, or patterns, storing said input data, images or patterns wherever storage space is available, and retrieving said input data, images, or patterns upon receipt of a prompt input; and

the cognitive memory system further comprising;

(a) a memory organized into memory folders for storing said input data, images or patterns;

(b) a trainable autoassociative neural network connected to said conventional memory to receive said input data, images, or patterns as training patterns;

(c) a stored adaptive algorithm for training said autoassociative neural network with said training patterns;

(d) at least one pre-processor(s) that modify input query images or patterns in order to generate multiple prompt patterns from a single input query pattern;

(e) means for testing each of said multiple prompt patterns, utilizing said autoassociative neural network, to select the successful prompt pattern that matches most closely to a pattern stored in one of the memory folders;

(f) means for scanning said memory folders to find a hit memory folder that stores the pattern that most closely matches said successful prompt pattern; and

(g) means for delivering the entire contents of said hit memory folder as the output of said cognitive memory system.

21. A surveillance system as in claim 20, wherein:

(a) the system is applied as a human face recognition system and wherein said input data, images, or patterns to be stored in the memory folders of said cognitive memory system are high-resolution photographs of persons of interest, including their faces, wherein one or many photographs of the same person are stored in the same memory folder, and the identity of each said person is stored as ancillary data in the same said memory folder as said one or many photographs of said person, wherein the autoassociative neural network of said cognitive memory system is trained with said input data, images, or patterns stored in said memory folders;

(b) high-resolution photographs of said persons of interest received by said cognitive memory system providing said input data, images, or patterns to said cognitive memory system;

said system further comprising;

(c) means for producing or receiving a low-resolution versions of said high-resolution photographs stored in said memory folders, said low-resolution photographs versions having lower spatial resolution than said high-resolution photographs;

(d) a low-resolution autoassociative neural network trained by said low-resolution versions of said high-resolution photographs stored in memory folders;

(e) means providing or receiving high-resolution photographs of persons to provide high-resolution input query patterns for said cognitive memory system;

(f) means for producing or receiving low-resolution versions of said high-resolution input query patterns;

(g) a low-resolution window scanning over said low-resolution versions of said high-resolution input query patterns, wherein said low-resolution window scans and optionally applies at least one of a rotation, a translation, a scaling, a brightness adjustment, and a contrast adjustment, to provide prompt patterns to said low-resolution autoassociative neural network, whereby the location, orientation and scale of detected facial images within said high-resolution input query patterns are calculated from the amount of any translation, rotation and scaling required to find the successful prompt patterns for said low-resolution autoassociative neural network;

(h) a high-resolution window, having an initial location, orientation and scale corresponding to said location, orientation and scale of said detected facial images as determined by said low-resolution autoassociative neural network, and scanning over said high-resolution input query patterns, wherein said high-resolution window scans and optionally applies at least one of a rotation, a translation, a scaling, a brightness adjustment, and a contrast adjustment to provide high-resolution prompt patterns for said autoassociative neural network of said cognitive memory system;

(i) successful high-resolution prompt patterns that produce errors lower than a pre-set threshold when sensed by said autoassociative neural network;

(j) means for comparing said successful high-resolution prompt patterns on a pixel-by-pixel or pixel group by pixel group basis with patterns stored in said memory folders, wherein said memory folder that contains patterns that best match each of said successful high-resolution prompt patterns is identified as the hit folder; and

(k) means for delivering as output the contents of said hit folder, which contains the identity of the stored facial images that correspond with the faces in said input query photograph.

22. A surveillance system as in claim 19, wherein:

(a) the surveillance system comprises a plurality of security checkpoint systems;

(b) an intelligence center coupled with the plurality of security checkpoint systems;

(c) means of communication between said plurality of security checkpoint systems and said intelligence center;

(d) at least one data base located at or coupled with said intelligence center that contains facial images of persons of interest and identities of said persons of interest; and

(e) a computer or other information appliance located at or coupled with said intelligence center that is capable of training said facial images of said persons of interest into an autoassociative neural network, and transmitting the weights, the structure, and the training patterns of said autoassociative neural network via said means of two-way communication to said security checkpoint systems, whereupon the weights, structure and training patterns of said autoassociative neural network are copied into the cognitive memories of the detection systems of said security checkpoint systems;

(f) whereby said persons of interest who pass through said security checkpoint systems are detected and identified, whereupon the detection event is transmitted via said means of two-way communication to said intelligence center, and new high-resolution photographs of said persons of interest that were taken at said security checkpoint are transmitted via said means of two-way communication to said intelligence center to be added to said data base for further training.

24. A method for accessing sensory data and patterns, said method comprising:

storing sensory input data and patterns in a memory element of a memory data structure, each memory element of said memory data structure capable of simultaneously receiving a plurality of input data and patterns from a plurality of sensors, and storing said plurality of input data and patterns and optionally storing other ancillary data associated with said input data and patterns;

retrieving a contents of at least one of said plurality of memory elements of said memory data structure in response to receiving a related prompt pattern, the prompt pattern derived from an input query pattern; and

relating said prompt pattern to one of the plurality of input data or patterns stored in one of said memory elements of said memory data structure.

25. A method for accessing sensory data and patterns, wherein said memory elements comprise memory folders and said memory data structure comprises a data structure formed in a memory storage device.

26. A computer program product for accessing sensory data and patterns, the computer program product comprising a computer readable storage and a computer program stored therein, the computer program comprising:

(a) instructions for storing sensory input data and patterns in a memory element of a memory data structure, each memory element of said memory data structure capable of simultaneously receiving a plurality of input data and patterns from a plurality of sensors, and storing said plurality of input data and patterns and optionally storing other ancillary data associated with said input data and patterns;

(b) instructions for retrieving a contents of at least one of said plurality of memory elements of said memory data structure in response to receiving a related prompt pattern, the prompt pattern derived from an input query pattern; and

(c) instructions for relating said prompt pattern to one of the plurality of input data or patterns stored in one of said memory elements of said memory data structure.