Methods and apparatus for automated true object-based image analysis and retrieval

• US 9,424,277 B2
• Filed: 10/31/2014
• Issued: 08/23/2016
• Est. Priority Date: 05/05/2004
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Retterath, Jamie E., Laumeyer, Robert A.
• Primary Examiner(s): Carter, Aaron W
• Application Number:
  US14/529,710
• Publication Number:
  US 20150104104A1
• Time in Patent Office: 662 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5862 using texture
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

An automated and extensible system for analysis and retrieval of images based on region-of-interest (ROI) analysis of one or more true objects depicted by an image is provided. The system uses a database that is a relational or analytical database containing searchable vectors that represent the images stored in a repository. Entries in the database are created by an image locator and ROI classifier working together to locate images within the repository and extract relevant information to be stored in the ROI database. The ROI classifier analyzes objects in an image to arrive at actual features of the true object. Graphical searches are performed by the collaborative workings of an image retrieval module, an image search requestor and an ROI query module. The image search requestor is an abstraction layer that translates user or agent search requests into the language understood by the ROI query.

Claims

1. A method comprising:

identifying, by one or more computers, an image in a first storage system;

determining, by the one or more computers, whether a second storage system includes an entry for the image identifying a region of interest in the image;

when the second storage system does not include an entry for the image, identifying, by the one or more computers, a region of interest for the image;

performing, by the one or more computers, one or more transforms on the identified region of interest in order to identify one or more aspects of the identified region of interest.

2. The method of claim 1, wherein the one or more aspects of the region of interest include at least one of color, shape, and texture.

3. The method of claim 1, further comprising:

for each given aspect of the one or more aspects of the identified region of interest, generating a physical vector representing that given aspect; and

generating a new entry in the second storage system for the image based on the identified region of interest and any generated physical vector.

4. The method of claim 3, further comprising:

determining that the second storage system now includes the new entry;

after determining that the second storage system now includes the new entry, generating a cache key;

comparing the cache key to data in the new entry; and

when the compare fails, invalidating the new entry.

5. The method of claim 3, further comprising:

determining that the second storage system now includes the new entry;

after determining that the second storage system now includes the new entry, generating a cache key;

comparing the cache key to data in the new entry; and

when the compare passes, determining whether to reprocess the image.

6. The method of claim 3, further comprising:

receiving a search request identifying a true object from a user;

querying the second storage system to identify the new entry based on the search request; and

after querying the second storage system, providing the image to the requesting user.

7. The method of claim 1, wherein identifying the identified region of interest includes:

segmenting the image into a plurality of regions of interest, each region of interest of the plurality of regions of interest containing content associated with one or more actual features of a physical object; and

extracting at least one physical descriptor from the content for the identified region of interest as a feature vector, each physical descriptor corresponding to one or more actual features of a given physical object in the identified region of interest, wherein the feature vector describes the one or more aspects.

8. The method of claim 7, wherein the extracting includes using one or more transform filters.

9. The method of claim 7, wherein the feature vector includes a searchable data structure including at least one of a physical color vector, a physical shape vector, and a physical texture vector.

10. A system comprising one or more computers configured to:

identify an image in a first storage system;

determine whether a second storage system includes an entry for the image identifying a region of interest in the image;

when the second storage system does not include an entry for the image, identify a region of interest for the image;

perform one or more transforms on the identified region of interest in order to identify one or more aspects of the identified region of interest.

11. The system of claim 10, wherein the one or more aspects of the region of interest include at least one of color, shape, and texture.

12. The system of claim 10, wherein the one or more computers are further configured to:

for each given aspect of the one or more aspects of the identified region of interest, generate a physical vector representing that given aspect; and

generate a new entry in the second storage system for the image based on the identified region of interest and any generated physical vector.

13. The system of claim 12, wherein the one or more computers are further configured to:

determine that the second storage system now includes the new entry;

after determining that the second storage system now includes the new entry, generate a cache key;

compare the cache key to data in the new entry; and

when the compare fails, invalidate the new entry.

14. The system of claim 12, wherein the one or more computers are further configured to:

determine that the second storage system now includes the new entry;

after determining that the second storage system now includes the new entry, generate a cache key;

compare the cache key to data in the new entry; and

when the compare passes, determine whether to reprocess the image.

15. The system of claim 12, wherein the one or more computers are further configured to:

receive a search request identifying a true object from a user;

query the second storage system to identify the new entry based on the search request; and

after querying the second storage system, provide the image to the requesting user.

16. The system of claim 10, wherein the one or more computers are further configured to identify the identified region of interest includes:

segment the image into a plurality of regions of interest, each region of interest of the plurality of regions of interest containing content associated with one or more actual features of a physical object; and

extract at least one physical descriptor from the content for the identified region of interest as a feature vector, each physical descriptor corresponding to one or more actual features of a given physical object in the identified region of interest, wherein the feature vector describes the one or more aspects.

17. The system of claim 16, wherein the one or more computers are further configured to extract the at least one physical descriptor by using one or more transform filters.

18. The system of claim 16, wherein the feature vector includes a searchable data structure including at least one of a physical color vector, a physical shape vector, and a physical texture vector.

19. The system of claim 10, further comprising the first storage system and the second storage system.

20. A non-transistory, tangible computer readable storage medium including a program that, when executed by one or more computers, causes the one or more computers to implement a method, the method comprising:

identifying an image in a first storage system;

determining whether a second storage system includes an entry for the image identifying a region of interest in the image;

when the second storage system does not include an entry for the image, identifying a region of interest for the image;

performing one or more transforms on the identified region of interest in order to identify one or more aspects of the identified region of interest.