Cognitive memory and auto-associative neural network based search engine for computer and network located images and photographs

• US 20060133699A1
• Filed: 10/07/2005
• Published: 06/22/2006
• Est. Priority Date: 10/07/2004
• Status: Active Grant

• Current Assignee: Bernard Widrow, Brian Mitchell Percival, Juan Carlos Aragon
• Original Assignee: Bernard Widrow, Brian Mitchell Percival, Juan Carlos Aragon
• Inventors: Aragon, Juan Carlos, Percival, Brian Mitchell, Widrow, Bernard
• Granted Patent: US 7,333,963 B2
• Time in Patent Office: Days
• US Class Current: 382/305
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 18/2135 based on approximation criteria, e.g. principal component analysis
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06V 10/70 using pattern recognition or machine learning optical pattern recognition or electronic computations therefor G06V10/88
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: Y10S 706/934 Information retrieval or Information management

Abstract

Designs for cognitive memory systems storing input data, images, or patterns, and retrieving it without knowledge of where stored when cognitive memory is prompted by query pattern that is related to sought stored pattern. Retrieval system of cognitive memory uses autoassociative neural networks and techniques for pre-processing query pattern to establish relationship between query pattern and sought stored pattern, to locate sought pattern, and to retrieve it and ancillary data. Cognitive memory, when connected to computer or information appliance introduces computational architecture that applies to systems and methods for navigation, location and recognition of objects in images, character recognition, facial recognition, medical analysis and diagnosis, video image analysis, and to photographic search engines that when prompted with a query photograph containing faces and objects will retrieve related photographs stored in computer or other information appliance, and will identify URL'"'"'s of related photographs and documents stored on the World Wide Web.

Claims

1. A photographic or image search engine for a computer or other information appliance, the photographic or image search engine characterized in that photographs or images stored in said computer or other information appliance are accessible to said photographic search engine, and are retrieved in response to receipt of a query comprising at least one query photograph or image, the query photograph or image having a relationship to said photographs or images stored in said computer or other information appliance.

3. The photographic or image search engine for a computer or other information appliance of claim 1, wherein said query photograph or image has one or more objects of interest or people'"'"'s faces, and said photographs or images stored in said computer or other information appliance are retrieved in response to receipt of a query photograph containing at least one object of interest or person'"'"'s face in common with said photographs stored in said computer or other information appliance.

2. A photographic or image search engine for a computer or other information appliance, the photographic or image search engine characterized in that photographs or images stored in said computer or other information appliance are accessible to said photographic or image search engine, and are retrieved in response to receipt of a query comprising at least one text or symbolic information and at least one photograph or image, wherein both the at least one text or symbolic information and at least one photograph or image have a relationship to said photographs or images stored in said computer or other information appliance.

4. A photographic or image search engine for a computer or other information appliance, wherein photographs or images stored in said computer or other information appliance are accessible to said photographic or image search engine, and are retrieved in response to receipt of a query comprising of a query photograph, where said query photograph has one or more objects of interest or at least one persons face, and said photographs or images stored in said computer or other information appliance contain objects of interest or at least one persons face, and wherein said photographic or image search engine further comprises:

(a) first means for locating said objects of interest or people'"'"'s faces in both said query photograph and in said photographs stored in said computer or other information appliance; and

(b) second means for relating said objects of interest in said query photograph to corresponding objects in said photographs stored in said computer or other information appliance.

5. The photographic search engine for a computer or other information appliance of claim 4, wherein:

(a) the first means of said photographic search engine for locating objects of interest or people'"'"'s faces is based on a low-resolution autoassociative neural network trained on low-resolution photographs of an arbitrary selection of said people'"'"'s faces or multiple variations of said objects of interest; and

(b) the second means of said photographic search engine for relating objects of interest or people'"'"'s faces is based on high-resolution autoassociative neural networks, each trained on high-resolution photographs of a single person'"'"'s face or on a single object of interest.

6. The photographic search engine for a computer or other information appliance of claim 5, wherein:

(a) the low-resolution autoassociative neural network of said photographic search engine, already trained, senses input prompt patterns generated by a window scanning over each said photograph stored in said computer or information appliance by rotation, translation, scaling, variation in brightness, and variation in contrast, seeking a low error in the difference between the input and sensed output of said low-resolution autoassociative neural network, where said low error in said difference, when below a pre-set threshold, indicates a detected object of interest or person'"'"'s face, and the corresponding position, orientation, and scale is recorded;

(b) each high-resolution autoassociative neural network of said photographic search engine, already trained on a specific object of interest or a specific person'"'"'s facial image, receives input patterns generated by a window scanning over each said photograph stored in said computer or information appliance, initiated in each said position, orientation and scale of said detected objects of interest or persons'"'"' faces recorded from the low-resolution procedure;

(c) said scanning performs rotation, translation, scaling, brightness adjustment, and contrast adjustment, seeking a low error in the difference between the input and sensing output of said high-resolution autoassociative neural network; and

(d) said low error in said difference, when below a pre-set threshold, indicates a hit and the corresponding photograph stored in said computer or information appliance is delivered as an output of the search.

17. A photographic or image search engine as in claim 4, further comprising:

(a) third means for locating said objects of interest or people'"'"'s faces in both said query photograph and in said photographs stored in said computer or other information appliance; and

(b) fourth means for relating said objects of interest in said query photograph to corresponding objects in said photographs stored in said computer or other information appliance.

18. A photographic or image search engine as in claim 17,

(a) the third means of said photographic search engine for locating objects of interest or people'"'"'s faces is based on at least one of a low-resolution, mid-resolution, or high-resolution autoassociative neural network trained on at least one of a low-resolution, mid-resolution, or high-resolution photographs of an arbitrary selection of said people'"'"'s faces or multiple variations of said objects of interest;

and

(b) the fourth means of said photographic search engine for relating objects of interest or people'"'"'s faces is based on at least one of a low-resolution, mid-resolution, or high-resolution autoassociative neural networks, each trained on at least one of a low-resolution, mid-resolution, or high-resolution photographs of a single person'"'"'s face or on a single object of interest.

7. A photographic search engine for the World Wide Web or other types of interconnected networks of computers, servers, information networks, or information appliances, wherein:

(a) photographs and documents containing photographs or images stored on network or web servers or other types of information storage appliances are accessible to said photographic or image search engine, and paths, addresses, universal resource locators (URL'"'"'s), or other source or location information that refer to said photographs or images and said documents containing photographs; and

(b) said paths, addresses, URL'"'"'s, and/or other source or location information are retrieved in response to receipt of a query consisting of a query photograph or image which has a relationship to said photographs or images and documents containing photographs stored on said web servers or other types of information storage appliances.

9. The photographic search engine of claim 7, wherein said query photograph has one or more objects of interest or people'"'"'s faces, and said photographs stored on said web servers or said other types of information storage appliances are retrieved in response to receipt of a query photograph containing at least one object of interest or person'"'"'s face in common with said photographs stored on said web servers or said other types of information storage appliances.

19. The photographic or image search engine of claim 7, wherein:

a relationship exists between a query photograph or query image and said photographs or images and documents containing photographs or images stored on said web servers or other types of information storage appliances; and

the relationship is detected and confirmed by sensing a plurality of autoassocitive neural networks, each having weights that were trained off-line with training patterns which were or were derived from said photographs or images and documents containing photographs stored on said web servers or other types of information appliances.

20. The photographic search engine of claim 19, wherein:

said weights, having been trained on images of a first subject of interest such as a person or an object of interest, and said training patterns are downloaded, labeled by said subject of interest category description, and stored in the memory of a conventional computer or information appliance, thus freeing said photographic search engine to be retrained on a second subject of interest, and returning and downloading said weights and said training patterns into said photographic search engine when a new query arrives relating to said first subject of interest, thus greatly extending the usefulness, speed, and flexibility of said photographic search engine.

8. A photographic search engine for the World Wide Web or other types of interconnected networks of computers, servers, information networks, or information appliances, wherein:

(a) photographs and documents containing photographs stored on web servers or other types of information storage appliances are accessible to said photographic search engine, and paths, addresses, or universal resource locators (URL'"'"'s) that refer to said photographs and said documents containing photographs; and

(b) said paths addresses, or URL'"'"'s are retrieved in response to receipt of a query consisting of text and a query photograph, which have a relationship to said photographs and documents containing photographs stored on said web servers or other types of information storage appliances.

10. A photographic search engine for the World Wide Web or interconnected networks of computers, servers, information networks, or information appliances, wherein photographs or images are stored on web servers or other information storage appliances are accessible to said photographic search engine, and are retrieved in response to receipt of a query consisting of a photograph, the query photograph, where said query photograph has one or more objects of interest or people'"'"'s faces, and said photographs stored on said web servers or said other information storage appliances contain objects of interest or people'"'"'s faces, where said photographic search engine has:

(a) first means for locating said objects of interest or people'"'"'s faces in both said query photograph and in said photographs stored on said web servers or said other information storage appliances; and

(b) second means for relating said objects of interest in said query photograph to corresponding objects in said photographs stored on said web servers or said other information storage appliances.

11. The photographic search engine for the World Wide Web or other network of information appliances of claim 10 comprising:

(a) means for crawling or searching said World Wide Web or other network of information appliances to collect photographs or images and the paths, addresses, or URL'"'"'s of said photographs or images or of documents containing said photographs or images;

(b) means for locating images of objects of interest or people'"'"'s faces from said photographs, and storing the located images with the paths, addresses, or URL'"'"'s of said photographs or said documents containing said photographs in a buffer;

(c) means for storing said located images and their respective paths, addresses or URL'"'"'s in said photographic search engine;

(d) means for detecting if said located images contain objects of interest or people'"'"'s faces that are contained in located images already stored in said photographic search engine;

(e) means for storing together in a list all paths, addresses and URL'"'"'s of all said located images containing the same object of interest or person'"'"'s face; and

(f) means for responding to a query photograph, and providing as a search output said list of all paths, addresses, and URL'"'"'s for each located image related to said query photograph.

12. In a photographic search engine for the World Wide Web or interconnected networks of computers, servers, information networks, or information appliances, wherein photographs or images are stored on web servers or other information storage appliances are accessible to said photographic search engine, and are retrieved in response to receipt of a query consisting of a photograph, the query photograph, where said query photograph has one or more objects of interest or people'"'"'s faces, and said photographs stored on said web servers or said other information storage appliances contain objects of interest or people'"'"'s faces, a method for performing a search for photographs or images comprising:

(a) locating said objects of interest or people'"'"'s faces in both said query photograph and in said photographs stored on said web servers or said other information storage appliances; and

(b) relating said objects of interest in said query photograph to corresponding objects in said photographs stored on said web servers or said other information storage appliances.

13. The method of claim 12, further comprising:

(a) crawling or searching said World Wide Web or other network of information appliances to collect photographs or images and the paths, addresses, or URL'"'"'s of said photographs or images or of documents containing said photographs or images;

(b) locating images of objects of interest or people'"'"'s faces from said photographs, and storing the located images with the paths, addresses, or URL'"'"'s of said photographs or said documents containing said photographs in a buffer;

(c) storing said located images and their respective paths, addresses or URL'"'"'s in said photographic search engine;

(d) detecting if said located images contain objects of interest or people'"'"'s faces that are contained in located images already stored in said photographic search engine;

(e) storing together in a list all paths, addresses and URL'"'"'s of all said located images containing the same object of interest or person'"'"'s face; and

(f) responding to a query photograph, and providing as a search output said list of all paths, addresses, and URL'"'"'s for each located image related to said query photograph.

14. A computer program product for performing a search for photographs or images on a single computer or information appliance or over an interconnected network of computers and/or information appliances, the computer program product comprising a computer readable storage and a computer program stored therein, the computer program comprising:

(a) instructions for locating objects of interest or people'"'"'s faces in both a query photograph and in photographs or images stored on said computers or information appliances;

web servers or said other information storage appliances; and

(b) instructions for relating said objects of interest in said query photograph to corresponding objects in said photographs stored on said web servers or said other information storage appliances.

15. A computer program product as in claim 14, wherein said photographic search engine is applied for a search of the World Wide Web or interconnected networks of computers, servers, information networks, or information appliances, and wherein photographs or images are stored on web servers or other information storage appliances are accessible to said photographic search engine, and are retrieved in response to receipt of a query consisting of a photograph, the query photograph, where said query photograph has one or more objects of interest or people'"'"'s faces, and said photographs stored on said web servers or said other information storage appliances contain objects of interest or people'"'"'s faces.

16. A computer program product as in claim 15, wherein the computer program further comprising:

(a) instructions for crawling or searching said World Wide Web or other network of information appliances to collect photographs or images and the paths, addresses, or URL'"'"'s of said photographs or images or of documents containing said photographs or images;

(b) instructions for locating images of objects of interest or people'"'"'s faces from said photographs, and storing the located images with the paths, addresses, or URL'"'"'s of said photographs or said documents containing said photographs in a buffer;

(c) instructions for storing said located images and their respective paths, addresses or URL'"'"'s in said photographic search engine;

(d) instructions for detecting if said located images contain objects of interest or people'"'"'s faces that are contained in located images already stored in said photographic search engine;

(e) instructions for storing together in a list all paths, addresses and URL'"'"'s of all said located images containing the same object of interest or person'"'"'s face; and

(f) instructions for responding to a query photograph, and providing as a search output said list of all paths, addresses, and URL'"'"'s for each located image related to said query photograph.