Salient object segmentation

• US 9,042,648 B2
• Filed: 02/23/2012
• Issued: 05/26/2015
• Est. Priority Date: 02/23/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wang, Jingdong, Li, Shipeng, Jiang, Huaizu
• Primary Examiner(s): TSAI, TSUNG YIN
• Application Number:
  US13/403,747
• Publication Number:
  US 20130223740A1
• Time in Patent Office: 1,188 Days
• US Class Current: 382/173
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06V 10/457 by analysing connectivity, e.g. edge linking, connected component analysis or slices

Abstract

Techniques for identifying a salient object with respect to its context are described. A process receives an input image that includes a salient object. The process segments the input image into multiple regions and calculates a saliency value for each of the segmented regions based on scale image levels. The process constructs saliency maps based at least in part on the calculated saliency value, and combines the saliency maps to construct a total saliency map. Next, the process connects a set of line segments computed from the input image and utilizes the total saliency map to compute a closed boundary, which forms a shape prior from the closed boundary, and extracts the salient object from the total saliency map and the shape prior.

Claims

1. A method implemented at least partially by a processor, the method comprising:

receiving, by the processor, an input image that includes a salient object;

identifying, by the processor, the salient object, the identifying comprising;

segmenting the input image into multiple regions, wherein the multiple regions are distinguished from each other based at least on saliency values;

calculating a saliency value for each of the segmented regions based on scale image levels;

constructing saliency maps based at least in part on the calculated saliency value for each of the segmented regions, the saliency maps to be constructed for the scale image levels;

constructing a total saliency map by combining the saliency maps constructed for the scale image levels; and

connecting a set of line segments that are computed from the input image and utilizing the total saliency map to define a shape prior that has a defined boundary;

extracting, by the processor, the salient object from the total saliency map and the shape prior; and

determining, by the processor, an optimal contour of the salient object by rejecting contours that have self-intersections and rejecting contours whose average saliency value is less than a total saliency value.

2. The method of claim 1, wherein the segmenting the input image comprises further fragmenting the input image into multiple groups having different parameters to generate multiple superpixel scales.

3. The method of claim 1, wherein the calculating the saliency value comprises:

measuring a color distance between a region and a neighboring region;

comparing the saliency value for each of the segmented regions; and

applying a Gaussian falloff weight to the saliency value, which includes an average spatial distance of pixels in a region to the center of the input image.

4. The method of claim 1, further comprising computing a minimum of a total length of gaps along the defined boundary over a total saliency value of pixels located inside the defined boundary.

5. The method of claim 4, further comprising defining at least a spatial distance between a pixel and the defined boundary.

6. The method of claim 1, further comprising:

determining that the salient object includes several parts and representing each part with a contour; and

determining the optimal contour of the salient object further comprises;

merging the contours of the several parts when the contours share same line segments, or

combining the contours of the several parts when one contour is located inside another contour.

7. The method of claim 1, further comprising filling in gaps on the salient object by adding, by the processor, a set of gap-filling segments to form a closed contour by utilizing the total saliency map.

8. The method of claim 1, further comprising:

assigning a label to each pixel in the input image based at least in part on a criteria that each label pixel is an element of 0 or 1, where 0 corresponds to the pixel belonging to a background and 1 corresponds to the pixel belonging to the salient object representing a foreground;

minimizing an energy function of the set of labels to further define the salient object; and

segmenting the salient object representing the foreground from the remaining regions representing the background.

9. The method of claim 8, further comprising:

updating the total saliency map by constructing hue, saturation, and value (HSV) histograms for (a) the salient object that represents the foreground and (b) the regions that represent the background;

updating the shape prior based on the salient object detected; and

performing an iterative energy minimization until convergence based at least in part on the updated saliency map and the updated shape prior to update the salient object.

10. One or more computer storage devices encoded with instructions that, when executed by one or more processors, cause the one or more processors to perform acts comprising:

receiving an input image that includes a salient object;

fragmenting the input image into multiple regions to create a saliency map;

calculating a saliency value for each of the multiple regions of the input image;

forming a defined boundary with a set of line segments from the input image to define a shape prior; and

determining an optimal contour of the salient object by;

forming contours around multiple parts;

merging the contours of the multiple parts when the contours share same line segments;

combining two or more of the contours when one contour is located inside another contour;

rejecting one or more of the contours that have self-intersections; and

rejecting one or more of the contours whose average saliency value is less than a total saliency value.

11. The computer storage devices of claim 10, wherein the calculating the saliency value comprises

propagating the saliency value from each of the regions to pixels of the corresponding region.

12. The computer storage devices of claim 10, wherein fragmenting the input image into multiple regions to create a saliency map comprises:

fragmenting the input image at a first scale level to create a first saliency map;

fragmenting the input image at a second scale level different than the first scale level to create a second saliency map; and

combining the first saliency map and the second saliency map to create the saliency map.

13. The computer storage devices of claim 10, further comprising:

detecting line segments from the input image; and

adding a set of gap-filling segments to fill in gaps of the detected line segments to form a closed contour and based on the closed contour compute the shape prior.

14. The computer storage devices of claim 10, further comprising:

assigning a set of labels to each label pixel in the input image by specifying that each label pixel is an element of 0 or 1, where 0 represents the pixel belonging to a background and 1 represents the pixel belonging to the salient object as foreground.

15. The computer storage devices of claim 14, further comprising:

presenting a user interface to receive the input image as a user query;

utilizing the shape prior and the saliency map to identify an appearance and a shape of the salient object; and

searching for images similar in appearance to the salient object.

16. A system comprising:

a memory;

one or more processors coupled to the memory;

an object application module executed on the one or more processors to receive an input image that includes a salient object;

a saliency map module executed on the one or more processors to construct a saliency map by fragmenting the input image into multiple regions,

the saliency map module executed on the one or more processors to calculate a saliency value for each region by computing a color difference between a region and neighboring regions, the saliency value of each region being weighted by Gaussian falloff weight that measures a distance of the region to an image center;

the object application module executed on the one or more processors to propagate the saliency value for each region to pixels; and

a closed contour module executed on the one or more processors to determine an optimal contour of the salient object by rejecting contours that have self-intersections and rejecting contours whose average saliency value is less than a total saliency value.

17. The system of claim 16, wherein the saliency map module is further executed on the one or more processors to:

construct saliency maps based at least in part on the calculated saliency value for each region, the saliency maps to be constructed for scale image levels; and

constructing a total saliency map by combining the saliency maps constructed for the scale image levels.

18. The system of claim 17, wherein the closed contour module is further executed on the one or more processors to perform acts of:

connecting a set of line segments in the saliency map;

filling in gaps between the set of line segments by adding a set of gap-filling segments to define a shape prior having a closed boundary; and

extracting a shape prior based on the closed boundary from the saliency map.

19. The system of claim 18, wherein the object application module is further executed on the one or more processors to assign a set of labels to each pixel in the input image by specifying that each pixel is an element of 0 or 1, where 0 represents pixels belonging to a background and 1 represents pixels belonging to the salient object as a foreground;

and

the object application module is further executed on the one or more processors to perform an iterative energy minimization until convergence to update the total extracted saliency map and the shape prior.

20. The system of claim 19, wherein the object application module is further executed on the one or more processors to present a user interface to receive the input image as a user query;

the object application module is further executed on the one or more processors to search for images similar in appearance to the salient object; and

the object application module is further executed on the one or more processors to present the user interface to display a list of images similar in appearance to the salient object based at least in part on the searching.