Methods and apparatus for automated true object-based image analysis and retrieval

• US 20050271304A1
• Filed: 05/05/2005
• Published: 12/08/2005
• Est. Priority Date: 05/05/2004
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Facet Technology Corp.
• Inventors: Laumeyer, Robert A., Retterath, Jamie E.
• Granted Patent: US 7,590,310 B2
• Time in Patent Office: Days
• US Class Current: 382/305
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5862 using texture
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

The present invention is an automated and extensible system for the analysis and retrieval of images based on region-of-interest (ROI) analysis of one or more true objects depicted by an image. The system uses a Regions Of Interest (ROI) database that is a relational or analytical database containing searchable vectors that represent the images stored in a repository. Entries in the ROI database are created by an image locator and ROI classifier that work in tandem to locate images within the repository and extract the relevant information that will be stored in the ROI database. Unlike existing region-of-interest search systems, the ROI classifier analyzes objects in an image to arrive at the actual features of the true object, instead of merely describing the features of the image of that object. Graphical searches are performed by the collaborative workings of an image retrieval module, an image search requestor and an ROI query module. The image search requestor is an abstraction layer that translates user or agent search requests into the language understood by the ROI query.

Claims

1. A computerized system for automated retrieval of images from an unstructured repository comprising:

a region of interest (ROI) database that contains searchable entries representing images stored in the repository;

an image locator that automatically locates images within the repository to populate the ROI database;

an ROI classifier that segments images located by the image locator into a plurality of ROIs wherein each ROI contains content associated with one or more actual features of a physical object, the ROI classifier having a plurality of computer-implemented transform filters adapted to extract physical descriptors from the content for each ROI to be stored in the ROI database, each physical descriptor corresponding to one of the one or more actual features associated with the content and having a searchable data structure that includes at least an ROI type field and an ROI descriptor field;

an ROI query module that interrogates the ROI database by searching the ROI database for images containing content associated with at least one true object requested by a user search request;

an image search requestor that receives and formats the user search request for input to the ROI query module; and

an image retrieval module that utilizes the searchable entries in the ROI database to access the images stored in the repository that have content generally matching the true object requested by the user search request.

2. The system of claim 1 wherein the ROI database is selected from the group consisting of a relational database and an analytical database.

3. The system of claim 1 wherein the data structure is a vector of physical features belonging to a physical object feature space representing a range of features of true objects corresponding to the images stored in the repository.

4. The system of claim 3 wherein the ROI database further includes at least one lookup table (LUT) that contains at least one entry describing at least one field in the data structure.

5. The system of claim 1 wherein the searchable data structure is a vector wherein the ROI type field contains a type of the region of interest (ROI) and the ROI descriptor field contains a description of the ROI having a format indicated by the type of the ROI.

6. The system of claim 1 wherein the searchable data structure further comprising at least one of a file name field, a cache key field, a physical color vector field, a physical shape vector field, a physical texture vector field, and optionally a physical size vector field, a physical location vector field and a physical content vector field.

7. The system of claim 1 wherein the repository is a physical storage environment selected from a group consisting of a local hard drive, a shared drive on a networked computer, one or more file servers attached to a WAN, a LAN or the Internet.

8. The system of claim 1 wherein the image locator further comprises automated routines for continually searching the repository for new images utilizing one or more of a web crawler, a spider, intelligent agent or a user input.

9. The system of claim 1 wherein the region of interest classifier segments the at least one image into regions of interest using one or more of segmentation, blob analysis, high frequency filter and Fourier transforms.

10. The system of claim 1 wherein the ROI is selected from a set consisting of a bounding box, a polygon, and a mask.

11. The system of claim 1 wherein the searchable data structure is a physical color vector.

12. The system of claim 11 wherein the physical color vector belongs to a physical object color space that describes a range of colors for the true objects that correspond to the images stored in the repository.

13. The system of claim 12 wherein the physical color vector has a single-bit field.

14. The system of claim 13 wherein the physical color vector has a multi-bit field representing an available quantum level selected from the set consisting of a probability that the object is a designated color with a designated color volume, a percentage of the designated color volume is contained within a ROI physical color representation, a distance from the center of mass of a ROI physical color representation to a center of mass of the designated color volume and a measure signifying the relative strength that a representative color contributes to a physical color of a original object.

15. The system of claim 1 wherein the searchable data structure is a physical shape vector.

16. The system of claim 15 wherein the physical shape vector belongs to a physical object shape space that contains two dimensional representations of true objects that correspond to the images stored in the repository.

17. The system of claim 15 wherein the physical shape vector contains fields that represent two-dimensional representations of true objects, the system further including a shape vector lookup table containing projections that correspond to the fields within the physical shape vector.

18. The system of claim 17 wherein the physical shape vector has a multi-bit field representing an available quantum level selected from the group consisting of a probability that the true object is a designated shape, that the true object contains feature aspects of the designated shape, and that a measure that a representative shape contributes to a physical shape of the true object.

19. The system of claim 1 wherein the searchable data structure is a physical texture vector.

20. The system of claim 19 wherein the physical texture vector belongs to a physical object texture space that describes a range of textures of the true objects that correspond to the images stored in the repository.

21. The system of claim 20 wherein the physical texture vector has a single-bit field.

22. The system of claim 21 wherein the physical texture vector has a multi-bit field representing an available quantum level selected from the group consisting of a probability that the true object is a designated texture, that the true object is in a similar class of textures, and that a measure that a representative texture contributes to a physical texture of the true object.

23. The system of claim 22 wherein the physical texture vector further includes fields with physical descriptors selected from the group consisting of shiny, dull, metallic, transparent, translucent, and reflective.

24. A computer-implemented method for automated retrieval of images from an unstructured repository comprising the steps of:

providing a region of interest (ROI) database containing searchable entries representing images stored in the repository;

automatically locating images within the repository;

populating the ROI database with the located images;

segmenting the located images into a plurality of regions of interest (ROIs) wherein each ROI contains content associated with one or more actual features of a physical object;

extracting at least one physical descriptor from the content for each ROI;

storing the at least one physical descriptor in the ROI database;

responsive to a user search request, interrogating the ROI database for images containing content associated with at least one true object requested by the user search request; and

accessing the images stored in ROI database, the images having content generally matching the at least one true object requested by the user search request.

25. The method of claim 24 wherein the step of automatically locating the images comprises using an image locator.

27. The method of claim 29 wherein the data structure is a vector of physical features belonging to a physical object feature space representing a range of features of true objects corresponding to the images stored in the repository.

26. The method of claim 27 wherein the step of segmenting the located images further comprises the step of:

providing an ROI classifier that segments images located by the image locator into a plurality of ROIs, the ROI classifier having a plurality of computer-implemented transform filters adapted to extract physical descriptors from the content for each ROI to be stored in the ROI database, each physical descriptor corresponding to one of the one or more actual features associated with the content and having a searchable data structure that includes at least an ROI type field and an ROI descriptor field;

28. The method of claim 30 wherein the vector of physical features is one or more of a physical color vector, a physical shape vector, and a physical texture vector.