Salient object detection by composition

• US 8,989,437 B2
• Filed: 05/16/2011
• Issued: 03/24/2015
• Est. Priority Date: 05/16/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wei, Yichen, Feng, Jie, Tao, Litian, Sun, Jian
• Primary Examiner(s): BEKELE, MEKONEN T
• Application Number:
  US13/108,953
• Publication Number:
  US 20120294476A1
• Time in Patent Office: 1,408 Days
• Field of Search: 382/103, 382/104, 382/284, 382/287, 382/224, 382/171, 382/173, 382/106, 382/165, 382/195, 382/197, 382/190, 348/221.12, 348/143, 348/169
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]

Abstract

A computing device configured to determine, for each of a plurality of locations in an image, a saliency measure based at least on a cost of composing parts of the image in the location from parts of the image outside of the location is described herein. The computing device is further configured to select one or more of the locations as representing salient objects of the image based at least on the saliency measures.

Claims

1. A method comprising:

determining, by a computing device, for each of a plurality of locations in an image, a saliency measure based at least on a cost of composing parts of the image in the location from parts of the image outside of the location, the cost being determined based at least in part on appearance distances and spatial distances of the parts of the image in the location from the parts of the image outside the location; and

selecting, by the computing device, one or more of the locations as representing salient objects of the image based at least on the saliency measures.

2. The method of claim 1, wherein the locations are defined by a sliding window that is moved across the image.

3. The method of claim 1, further comprising segmenting the image into a plurality of parts that include the parts in the location and the parts outside of the location.

4. The method of claim 1, wherein at least one part of the image intersects with a boundary of the location such that a first portion of the at least one part is in the location and a second portion of the at least one part is outside of the location.

5. The method of claim 4, further comprising classifying the at least one part as one of the parts in the location or one of the parts outside of the location based on relative sizes of the first and second portions of the at least one part.

6. The method of claim 1, further comprising determining the cost based at least on appearance distances and spatial distances of the parts of the image in the location from the parts of the image outside of the location.

7. The method of claim 1, wherein the parts of the image outside of the location are each used only once in composing the parts of the image in the location.

8. The method of claim 1, further comprising normalizing the saliency measures based on location sizes.

9. The method of claim 1, wherein the composing comprises processing parts of the image in the location that are further from center of the location before those closer to center of the location.

10. The method of claim 1, wherein the composing comprises using parts of the image outside of the location with lowest composition costs relative to parts of the image in the location first.

11. The method of claim 1, wherein selecting the one or more locations comprises selecting locations that are local maxima.

12. The method of claim 11, wherein selecting locations that are local maxima comprises removing locations that overlap with other locations, the other locations having higher saliency scores than the removed locations.

13. The method of claim 1, wherein the determining is performed in parallel for the plurality of locations.

14. The method of claim 1, further comprising extending regions adjacent to a boundary of the image and utilizing an extended portion of the regions as a part or parts of the image outside of the location.

15. The method of claim 1, further comprising utilizing the salient objects as image thumbnails, for image retargeting, or for image summarization.

16. One or more computer-readable storage devices comprising computer-executable instructions for programming one or more computing devices to perform operations comprising:

for each of a plurality of locations in an image, determining a saliency measure based at least on a cost of composing parts of the image in the location from parts of the image outside of the location, the cost being determined based at least on appearance distances and spatial distances of the parts of the image in the location from the parts of the image outside the location; and

selecting one or more of the locations as representing salient objects of the image based at least on the saliency measures.

17. The one or more computer-readable storage devices of claim 16, wherein the cost is based at least in part on a spatial distance between of a part of the image in the location and a part of the image outside the location.

18. The one or more computer-readable storage devices of claim 16, wherein the appearance distances are based on color histograms of parts of the image and the spatial distances are Hausdorff distances between parts of the image.

19. A system comprising:

a processor;

a saliency determination module configured to be operated by the processor to determine, for each of a plurality of locations in an image, a saliency measure based at least on a cost of composing parts of the image in the location from parts of the image outside of the location, the cost being determined based at least in part on spatial distances of the parts of the image in the location from the parts of the image outside the location, wherein determining the cost includes extending, by the processor, regions adjacent to a boundary of the image and utilizing the extended regions as the parts of the image outside of the location; and

an object selection module configured to be operated by the processor to select one or more of the locations as representing salient objects of the image based at least on the saliency measures.

20. The system of claim 19, wherein each location being defined by a sliding window that is moved across the image and a plurality of sliding windows of different shapes and dimensions are used to define the plurality of locations.