Salient Object Segmentation

• US 20130223740A1
• Filed: 02/23/2012
• Published: 08/29/2013
• Est. Priority Date: 02/23/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wang, Jingdong, Li, Shipeng, Jiang, Huaizu
• Granted Patent: US 9,042,648 B2
• Time in Patent Office: Days
• US Class Current: 382/171
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06V 10/457 by analysing connectivity, e.g. edge linking, connected component analysis or slices

Abstract

Techniques for identifying a salient object with respect to its context are described. A process receives an input image that includes a salient object. The process segments the input image into multiple regions and calculates a saliency value for each of the segmented regions based on scale image levels. The process constructs saliency maps based at least in part on the calculated saliency value, and combines the saliency maps to construct a total saliency map. Next, the process connects a set of line segments computed from the input image and utilizes the total saliency map to compute a closed boundary, which forms a shape prior from the closed boundary, and extracts the salient object from the total saliency map and the shape prior.

Claims

1. A method implemented at least partially by a processor, the method comprising:

receiving an input image that includes a salient object;

segmenting the input image into multiple regions;

calculating a saliency value for each of the segmented regions based on scale image levels;

constructing saliency maps based at least in part on the calculated saliency value for each of the segmented regions, the saliency maps to be constructed for the scale image levels;

constructing a total saliency map by combining the saliency maps constructed for the scale image levels;

connecting a set of line segments that are computed from the input image and utilizing the total saliency map to define a shape prior that has a defined boundary; and

extracting the salient object from the total saliency map and the shape prior.

2. The method of claim 1, wherein the segmenting the input image comprises further fragmenting the input image into multiple groups having different parameters to generate multiple superpixel scales.

3. The method of claim 1, wherein the calculating the saliency value comprises:

measuring a color distance between a region and a neighboring region;

comparing the saliency value for each of the segmented regions; and

applying a Gaussian falloff weight to the saliency value, which includes an average spatial distance of pixels in a region to the center of the input image.

4. The method of claim 1, further comprising computing a minimum of a total length of gaps along the defined boundary over a total saliency value of pixels located inside the defined boundary.

5. The method of claim 4, further comprising defining at least a spatial distance between a pixel and the defined boundary.

6. The method of claim 1, further comprising:

determining that the salient object includes several parts and representing each part with a contour;

determining an optimal contour by;

rejecting contours of the several parts that have self-intersections, or

rejecting the contours whose average saliency values of pixels located inside contours are smaller than a total saliency value,

merging the contours of the several parts when the contours share same line segments, or

combining the contours of the several parts when one contour is located inside another contour.

7. The method of claim 1, further comprising filling in gaps on the salient object by adding a set of gap-filling segments to form a closed contour by utilizing the total saliency map.

8. The method of claim 1, further comprising:

assigning a label to each pixel in the input image based at least in part on a criteria that each label pixel is an element of 0, or 1, where 0 corresponds to the pixel belonging to a background and 1 corresponds to the pixel belonging to the salient object representing a foreground;

minimizing an energy function of the set of labels to further define the salient object; and

segmenting the salient object representing the foreground from the remaining regions representing the background.

9. The method of claim 8, further comprising:

updating the total saliency map by constructing hue, saturation, and value (HSV) histograms for (a) the salient object that represents a foreground and (b) the regions that represent a background;

updating the shape prior based on the salient object detected; and

performing an iterative energy minimization until convergence based at least in part on the updated saliency map and the updated shape prior to update the salient object.

10. One or more computer-readable storage media encoded with instructions that, when executed by one or more processors, cause the one or more processors to perform acts comprising:

receiving an input image that includes a salient object;

fragmenting the input image into multiple regions to create a saliency map;

calculating a saliency; and

forming a defined boundary with a set of line segments from the input image to define a shape prior.

11. The computer-readable storage media of claim 10, wherein the calculating the saliency value comprises:

computing the saliency value of each of the regions from the input image; and

propagating the saliency value from each of the regions to pixels.

12. The computer-readable storage media of claim 10, further comprising determining an optimal contour of the salient object by:

forming contours around multiple parts;

rejecting the contours of the multiple parts that have self-intersections;

rejecting the contours whose average saliency value is less than a total saliency value;

merging the contours of the multiple parts when the contours share same line segments; and

combining contours of the multiple parts when one contour is located inside another contour.

13. The computer-readable storage media of claim 10, further comprising:

detecting line segments from the input image; and

adding a set of gap-filling segments to fill in gaps of the detected line segments to form a closed contour and based on the closed contour compute the shape prior.

14. The computer-readable storage media of claim 10, further comprising:

assigning a set of labels to each label pixel in the input image by specifying that each label pixel is an element of 0 or 1, where 0 represents the pixel belonging to a background and 1 represents the pixel belonging to the salient object as foreground.

15. The computer-readable storage media of claim 14, further comprising:

presenting a user interface to receive the input image as a user query;

utilizing the shape prior and the saliency map to identify an appearance and a shape of the salient object; and

searching for images similar in appearance to the salient object.

16. A system comprising:

a memory;

one or more processors coupled to the memory;

an object application module executed on the one or more processors to receive an input image that includes a salient object;

a saliency map module executed on the one or more processors to construct a saliency map by fragmenting the input image into multiple regions,

the saliency map module executed on the one or more processor to calculate a saliency value for each region by computing a color difference between a region and neighboring regions, the saliency value of each region being weighted by Gaussian falloff weight that measures a distance of the region to an image center; and

the object application module executed on the one or more processors to propagate the saliency value for each region to pixels.

17. The system of claim 16, wherein the saliency map module is further executed on the one or more processors to:

construct saliency maps based at least in part on the calculated saliency value for each of the segmented regions, the saliency maps to be constructed for scale image levels; and

constructing a total saliency map by combining the saliency maps constructed for the scale image levels.

18. The system of claim 17, further comprising a closed contour module executed on the one or more processors to perform acts of:

connecting a set of line segments in the saliency map;

filling in gaps between the set of line segments by adding a set of gap-filling segments to define a shape prior having a closed boundary; and

extracting a shape prior with based on the closed boundary from the saliency map.

19. The system of claim 18, wherein the object application module is further executed on the one or more processors to assign a set of labels to each pixel in the input image by specifying that each pixel is an element of 0 or 1, where 0 represents pixels belonging to a background and 1 represents pixels belonging to the salient object as a foreground;

and

the object application module is further executed on the one or more processors to perform an iterative energy minimization until convergence to update the total extracted saliency map and the shape prior.

20. The system of claim 19, wherein the object application module is further executed on the one or more processors to present a user interface to receive the input image as a user query;

the object application module is further executed on the one or more processors to search for images similar in appearance to the salient object; and

the object application module is further executed on the one or more processors to present the user interface to display a list of images similar in appearance to the salient object based at least in part on the searching.