Methods and apparatus for automated true object-based image analysis and retrieval

• US 8,903,199 B2
• Filed: 02/06/2012
• Issued: 12/02/2014
• Est. Priority Date: 05/05/2004
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Retterath, Jamie E., Laumeyer, Robert A.
• Primary Examiner(s): Carter, Aaron W
• Application Number:
  US13/366,870
• Publication Number:
  US 20120140990A1
• Time in Patent Office: 1,030 Days
• Field of Search: 382/100, 382/305, 707/709, 707/E17.03, 707/E17.108
• US Class Current: 382/305
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5862 using texture
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

An automated and extensible system for analysis and retrieval of images based on region-of-interest (ROI) analysis of one or more true objects depicted by an image is provided. The system uses an database that is a relational or analytical database containing searchable vectors that represent the images stored in a repository. Entries in the database are created by an image locator and ROI classifier working together to locate images within the repository and extract relevant information to be stored in the ROI database. The ROI classifier analyzes objects in an image to arrive at actual features of the true object. Graphical searches are performed by the collaborative workings of an image retrieval module, an image search requestor and an ROI query module. The image search requestor is an abstraction layer that translates user or agent search requests into the language understood by the ROI query.

Claims

1. A system for processing imagery, the system comprising:

a database configured to store searchable data structures that represent images;

a computer operatively coupled to the database, the computer including an image locator that locates images in the database for analysis and a region of interest classifier that extracts relevant information from the database;

the image locator being configured to;

locate an image file associated with a given image,

determine whether a region of interest entry exists in the database for the given image, and

when the region of interest entry does not exist for the given image, the given image is sent to the region of interest classifier for analysis; and

the region of interest classifier being configured to;

find a unique region of interest for the given image;

perform one or more transforms to describe the unique region of interest within a physical space, and

create one or more physical vectors based on the one or more transforms.

2. The system of claim 1, wherein when the region of interest entry does exist for the given image, the image locator is further configured to:

create a pseudo-unique cache key comprising a semi-unique qualifier for the image file; and

compare the pseudo-unique cache key to one or more entries in the database.

3. The system of claim 2, wherein when the comparison fails, the image locator is further configured to invalidate some or all region of interest data structures in the database associated with the given image.

4. The system of claim 3, wherein the image locator is further configured to send the given image to the region of interest classifier.

5. The system of claim 2, wherein when the comparison passes, the image locator is further configured to determine whether versions of the physical vectors associated with the unique region of interest stored in the database are current versions.

6. The system of claim 5, wherein when the versions are not current versions, the image locator is further configured to send the given image to the region of interest classifier.

7. The system of claim 1, wherein the region of interest classifier is further configured to construct a region of interest data structure from the one or more physical vectors, the region of interest data structure being configured for storage in the database as a searchable data structure.

8. The system of claim 7, wherein the searchable data structure contains a region of interest type and a region of interest descriptor field including a description of the region of interest having a format indicated by the region of interest type.

9. The system of claim 7, wherein the searchable data structure comprises at least one of a physical color vector field, a physical shape vector field, or a physical texture vector field.

10. The system of claim 9, wherein the physical color vector of the physical color vector field belongs to a physical object color space that describes a range of colors for true objects that correspond to respective images stored in the database.

11. The system of claim 9, wherein the physical shape vector of the physical shape vector field belongs to a physical object shape space that contains two dimensional representations of true objects that correspond to respective images stored in the database.

12. The system of claim 9, wherein the physical texture vector of the physical texture vector field belongs to a physical object texture space that describes a range of textures of true objects that correspond to respective images stored in the database.

13. The system of claim 7, wherein upon construction of a region of interest data structure for a given image, the system creates a pseudo-unique cache key comprising a semi-unique qualifier for that image.

14. A computer-implemented method for processing imagery, the method comprising:

locate an image file associated with a given image;

determine whether a region of interest entry exists in a database for the given image;

when the region of interest entry does not exist for the given image, sending the given image to a region of interest classifier for analysis;

finding, with the region of interest classifier, a unique region of interest for the given image;

performing, with the region of interest classifier, one or more transforms to describe the unique region of interest within a physical space; and

creating one or more physical vectors for storage in the database, the one or more physical vectors being based on the one or more transforms.

15. The method of claim 14, wherein when the region of interest entry does exist for the given image, the method further comprises:

creating a pseudo-unique cache key comprising a semi-unique qualifier for the image file; and

comparing the pseudo-unique cache key to one or more entries in the database.

16. The method of claim 15, wherein when the comparison fails, the method further comprises invalidating some or all region of interest data structures in the database associated with the given image.

17. The method of claim 15, wherein when the comparison passes, the method further comprises determining whether versions of the physical vectors associated with the unique region of interest stored in the database are current versions.

18. The method of claim 14, wherein the method further comprises constructing a region of interest data structure from the one or more physical vectors, the region of interest data structure being configured for storage in the database as a searchable data structure.

19. The method of claim 18, wherein the searchable data structure contains a region of interest type and a region of interest descriptor field including a description of the region of interest having a format indicated by the region of interest type.

20. The method of claim 18, wherein the searchable data structure comprises at least one of a physical color vector field, a physical shape vector field, or a physical texture vector field.

21. The method of claim 18, wherein upon construction of a region of interest data structure for a given image, the method further includes creating a pseudo-unique cache key comprising a semi-unique qualifier for that image.

22. A tangible, non-transitory computer readable recording medium recorded with a program that, when executed by a processor, causes the processor to implement a method for processing imagery, the method comprising:

locate an image file associated with a given image;

determine whether a region of interest entry exists in a database for the given image;

when the region of interest entry does not exist for the given image, sending the given image to a region of interest classifier for analysis;

finding, with the region of interest classifier, a unique region of interest for the given image;

performing, with the region of interest classifier, one or more transforms to describe the unique region of interest within a physical space; and

creating one or more physical vectors for storage in the database, the one or more physical vectors being based on the one or more transforms.