Image Search by Query Object Segmentation

• US 20140089326A1
• Filed: 09/21/2012
• Published: 03/27/2014
• Est. Priority Date: 09/21/2012
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Lin, Zhe, Brandt, Jonathan W.
• Granted Patent: US 8,880,563 B2
• Time in Patent Office: Days
• US Class Current: 707/749
• CPC Class Codes: G06F 16/5854 using shape and object relationship

Abstract

Query object localization, segmentation, and retrieval are disclosed. A query image may be received that includes a query object. Based on respective spatially constrained similarity measures between the query image and a plurality of images from an image database, at least some of the plurality of images may be identified and/or retrieved and a location of the query object in the query image may be estimated. The query object may then be automatically segmented from the query image based on the estimated query object location. In some embodiments, the retrieval, localization and/or segmentation may be iterated.

Claims

1. A method, comprising:

performing, by one or more computing devices;

receiving a query image, wherein the query image includes a query object, wherein a location of the query object in the query image is unspecified;

based on respective spatially constrained similarity measures between the query image and a plurality of images from an image database;

identifying at least some of the plurality of images from the image database, wherein the at least some images share a common feature with the query image, and

estimating the location of the query object in the query image based on the identified at least some images; and

automatically segmenting the query object from the query image based on the estimated location.

2. The method of claim 1, further comprising:

updating the query image with the segmented query object and a clean background; and

repeating said identifying a plurality of images based on the updated query image, wherein said repeating said identifying includes retrieving the identified plurality of images based on the updated query image.

3. The method of claim 2, further comprising:

iteratively performing said estimating, said automatically segmenting, said updating, and said identifying.

4. The method of claim 1, wherein said estimating the location of the query object includes, for each respective image of the at least some images, determining a respective mask for the query object.

5. The method of claim 4, said determining the respective mask for the query object includes, for each respective image of the at least some images, voting on a center of the query object based on matched feature pairs between the respective image and the query image resulting in the respective mask of the query object for the respective image, wherein the respective mask includes the voted center of the query object.

6. The method of claim 4, wherein said estimating the location of the query object includes:

removing outlier images from the retrieved images resulting in inlier images; and

aggregating the respective masks of the inlier images according to respective weights resulting in a query object support map.

7. The method of claim 6, wherein said estimating the location of the query object further includes:

updating the inlier images by iteratively performing said removing outlier images and said aggregating the respective masks resulting in a refined query object support map.

8. The method of claim 6, wherein the weights are based on a ranking of the inlier images according to their respective spatially constrained similarity measures.

9. The method of claim 4, wherein said estimating the location of the query object further includes:

generating a query object support map from at least some of the respective masks, wherein the query object support map indicates a region of the query object; and

generating a trimap from the query object support map to use in said automatically segmenting.

10. The method of claim 1, wherein said identifying at least some of the plurality of images from the image database includes:

determining a respective transformation for each of the at least some images based on a maximization of each of the respective spatially constrained similarity measures;

wherein said estimating the location of the query object is based on at least some of the respective transformations.

11. The method of claim 1, wherein the query image is obtained by a mobile device of the one or more computing devices.

12. The method of claim 1, wherein the plurality of images in the image database include multiple different categories of images.

13. A non-transitory computer-readable storage medium storing program instructions, wherein the program instructions are computer-executable to implement:

receiving a query image, wherein the query image includes a query object, wherein a location of the query object in the query image is unspecified;

based on respective spatially constrained similarity measures between the query image and a plurality of images from an image database;

identifying at least some of the plurality of images from the image database, wherein the at least some images share a common feature with the query image, and

estimating the location of the query object in the query image based on the identified at least some images; and

automatically segmenting the query object from the query image based on the estimated location.

14. The non-transitory computer-readable storage medium of claim 13, wherein the program instructions are further computer-executable to implement:

updating the query image with the segmented query object and a clean background;

repeating said identifying a plurality of images based on the updated query image, wherein said repeating said identifying includes retrieving the identified plurality of images based on the updated query image; and

iteratively performing said estimating, said automatically segmenting, said updating, and said identifying.

15. The non-transitory computer-readable storage medium of claim 13, wherein said estimating the location of the query object includes:

for each respective image of the at least some images, determining a respective mask for the query object;

generating a query object support map from at least some of the respective masks, wherein the query object support map indicates a region of the query object; and

generating a trimap from the query object support map to use in said automatically segmenting.

16. The non-transitory computer-readable storage medium of claim 15, wherein said generating a query object support map from at least some of the respective masks includes generating the query object support map from the masks corresponding to mlier images.

17. A system, comprising:

one or more processors; and

a memory comprising program instructions, wherein the program instructions are executable by at least one of the one or more processors to;

receive a query image, wherein the query image includes a query object, wherein a location of the query object in the query image is unspecified;

based on respective spatially constrained similarity measures between the query image and a plurality of images from an image database;

identify at least some of the plurality of images from the image database, wherein the at least some images share a common feature with the query image, and

estimate the location of the query object in the query image based on the identified at least some images; and

automatically segment the query object from the query image based on the estimated location.

18. The system of claim 17, wherein the program instructions are further executable by at least one of the one or more processors to:

update the query image with the segmented query object and a clean background;

repeat said identifying a plurality of images based on the updated query image, wherein said repeating said identifying includes retrieving the identified plurality of images based on the updated query image; and

iteratively perform said estimating, said automatically segmenting, said updating, and said identifying.

19. The system of claim 17, wherein to perform said estimating the location of the query object, the program instructions are further executable by at least one of the one or more processors to:

for each respective image of the at least some images, determine a respective mask for the query object;

generate a query object support map from at least some of the respective masks, wherein the query object support map indicates a region of the query object; and

generate a trimap from the query object support map to use in said automatically segmenting.

20. The system of claim 17, wherein to perform said generating a query object support map from at least some of the respective masks, the program instructions are further executable by at least one of the one or more processors to:

generating the query object support map from the masks corresponding to inlier images.