Methods and apparatus for automated true object-based image analysis and retrieval

• US 8,908,996 B2
• Filed: 01/31/2012
• Issued: 12/09/2014
• Est. Priority Date: 05/05/2004
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Retterath, Jamie E., Laumeyer, Robert A.
• Primary Examiner(s): Carter, Aaron W
• Application Number:
  US13/362,774
• Publication Number:
  US 20120134583A1
• Time in Patent Office: 1,043 Days
• Field of Search: 382/165, 382/203, 382/305, 707/709, 707/769, 707/E17.03, 707/E17.108
• US Class Current: 382/305
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5862 using texture
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

An automated and extensible system is provided for the analysis and retrieval of images based on region-of-interest (ROI) analysis of one or more true objects depicted by an image. The system uses an ROI database that is a relational or analytical database containing searchable vectors representing images stored in a repository. Entries in the ROI database are created by an image locator and ROI classifier that locate images within the repository and extract relevant information to be stored in the ROI database. The ROI classifier analyzes objects in an image to arrive at actual features of the true object. Graphical searches may also be performed.

Claims

1. A computerized system for automated retrieval of images from an unstructured repository, the computerized system comprising:

an image locator that automatically locates images for analysis within the repository to populate a region of interest (ROI) database that contains searchable entries representing images stored in the repository;

an ROI classifier that segments one or more images located by the image locator into a plurality of ROIs, each ROI containing image content associated with one or more actual features of a physical object in the ROI, the ROI classifier having at least one computer-implemented transform filter adapted to extract one or more physical descriptors from the image content for each ROI to be stored in the ROI database as a physical feature vector, each physical descriptor corresponding to at least one of the one or more actual features of a given physical object in the ROI associated with the image content; and

an ROI query module adapted to search the ROI database for images containing content associated with at least one true physical object.

2. The system of claim 1 wherein the ROI database is selected from the group consisting of a relational database and an analytical database.

3. The system of claim 1 wherein the physical feature vector has a searchable data structure of physical features belonging to a physical object feature space representing a range of features of true objects.

4. The system of claim 3, wherein the searchable data structure contains an ROI type and an ROI descriptor field contains a description of the ROI having a format indicated by the ROI type.

5. The system of claim 3, wherein the searchable data structure comprises at least one of a physical color vector field, a physical shape vector field, or a physical texture vector field.

6. The system of claim 1, wherein the image locator further comprises one or more automated routines for continually searching the repository for new images.

7. The system of claim 1, wherein the region of interest classifier segments the one or more images into regions of interest using one or more of segmentation, blob analysis, high frequency filter and Fourier transforms.

8. The system of claim 1 wherein, the ROI is selected from a set consisting of a bounding box, a polygon, and a mask.

9. The system of claim 1, wherein the physical feature vector has a searchable data structure that comprises a physical color vector.

10. The system of claim 9, wherein the physical color vector belongs to a physical object color space that describes a range of colors for the true objects that correspond to the images stored in the repository.

11. The system of claim 10 wherein the physical color vector has a single-bit field.

12. The system of claim 1, wherein the physical feature vector has a searchable data structure that comprises a physical shape vector.

13. The system of claim 12, wherein the physical shape vector belongs to a physical object shape space that contains two dimensional representations of true objects that correspond to the images stored in the repository.

14. The system of claim 1, wherein the physical feature vector has a searchable data structure that comprises a physical texture vector.

15. The system of claim 14, wherein the physical texture vector belongs to a physical object texture space that describes a range of textures of the true objects that correspond to the images stored in the repository.

16. A computer-implemented method for automated retrieval of images from an unstructured repository, the method comprising:

automatically locating images for analysis within the repository;

populating a region of interest (ROI) database with the located images, the ROI database containing searchable entries representing images stored in the repository;

segmenting the located images into a plurality of regions of interest (ROIs), each ROI containing content associated with one or more actual features of a physical object;

extracting at least one physical descriptor from the content for each ROI, each physical descriptor corresponding to one or more of the actual features of a given physical object in the ROI; and

storing the at least one physical descriptor in the ROI database.

17. The method of claim 16, wherein segmenting the located images comprises:

segmenting images located by the image locator into a plurality of ROIs using one or more computer-implemented transform filters adapted to extract physical descriptors from the content for each ROI to be stored in the ROI database.

18. The method of claim 17, wherein each physical descriptor is a data structure having a vector of physical features belonging to a physical object feature space representing a range of features of true objects.

19. The method of claim 18, wherein the vector of physical features is one or more of a physical color vector, a physical shape vector, and a physical texture vector.

20. A tangible, non-transitory computer readable recording medium recorded with a program that, when executed by a processor, causes the processor to implement a method for automated retrieval of images from an unstructured repository, the method comprising:

automatically locating images for analysis within the repository;

populating a region of interest (ROI) database with the located images, the ROI database containing searchable entries representing images stored in the repository;

segmenting the located images into a plurality of regions of interest (ROIs), each ROI containing content associated with one or more actual features of a physical object;

extracting at least one physical descriptor from the content for each ROI, each physical descriptor corresponding to one or more of the actual features of a given physical object in the ROI; and

storing the at least one physical descriptor in the ROI database.

21. The recording medium of claim 20, wherein segmenting the located images comprises segmenting images located by the image locator into a plurality of ROIs using one or more computer-implemented transform filters adapted to extract physical descriptors from the content for each ROI to be stored in the ROI database.

22. The method of claim 21, wherein each physical descriptor is a data structure having a vector of physical features belonging to a physical object feature space representing a range of features of true objects.

23. The method of claim 22, wherein the vector of physical features is one or more of a physical color vector, a physical shape vector, and a physical texture vector.