Saliency Map Computation

• US 20160104054A1
• Filed: 10/08/2014
• Published: 04/14/2016
• Est. Priority Date: 10/08/2014
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Lin, Zhe, Mech, Radomir, Shen, Xiaohui, Price, Brian L., Zhang, Jianming
• Granted Patent: US 9,454,712 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]

Abstract

Saliency map computation is described. In one or more implementations, a base saliency map is generated for an image of a scene. The base saliency map may be generated from intermediate saliency maps computed for boundary regions of the image. Each of the intermediate saliency maps may represent visual saliency of portions of the scene that are captured in the corresponding boundary region. The boundary regions may include, for instance, a top boundary region, a bottom boundary region, a left boundary region, and a right boundary region. Further, the intermediate saliency maps may be combined in such a way that an effect of a foreground object on the saliency map is suppressed. The foreground objects for which the effect is suppressed are those that occupy a majority of one of the boundary regions.

Claims

1. A method implemented by a computing device, the method comprising:

determining boundary regions of an image that depicts a scene, each of the boundary regions comprising less than an entirety of the image;

computing intermediate saliency maps of each of the boundary regions, the intermediate saliency maps representing visual saliency of portions of the scene that are captured in the boundary regions, the intermediate saliency maps computed based on the visual saliency of the portions of the scene captured in the boundary regions being background of the scene; and

combining the intermediate saliency maps to generate a base saliency map of the image, the intermediate saliency maps being combined to suppress an effect of one or more foreground objects on the base saliency map, the one or more foreground objects that have the effect suppressed occupying a majority of one of the boundary regions.

2. A method as described in claim 1, wherein the boundary regions include a top boundary region determined for a top of the image, a bottom boundary region determined for a bottom of the image, a left boundary region determined for a left side of the image, and a right boundary region determined for a right side of the image.

3. A method as described in claim 1, wherein computing the intermediate saliency maps includes, for a particular said boundary region:

computing an average color of pixels contained in the particular boundary region; and

calculating, for each of the pixels of the image, a saliency value based on a whitened Euclidean distance from the average color, the calculated saliency value usable with saliency values calculated for the other pixels to form a respective said intermediate saliency map.

4. A method as described in claim 3, wherein the average color of the pixels comprises a sample mean color of the pixels.

5. A method as described in claim 3, further comprising computing a covariance matrix for the pixels contained in the particular boundary region, the covariance matrix used to whiten a feature space of the particular boundary region that enables the calculation of the saliency value based on the whitened Euclidean distance from the average color.

6. A method as described in claim 1, further comprising generating a dense saliency map using the base saliency map, the dense saliency map configured to indicate that a greater amount of the scene captured in the image is visually salient than the base saliency map.

7. A method as described in claim 6, wherein the dense saliency map and a saliency edge map are usable to perform an assessment of composition quality of the image, the saliency edge map computed using the dense saliency map and a gradient magnitude map that indicates, for each pixel of the image, an amount the image is changing at the pixel relative to other pixels.

8. A method as described in claim 6, wherein the base saliency map, the dense saliency map, and a row-normalized gradient map are usable to perform an assessment of degrees of preservation of content appearing in the scene for a cropping of the image, the row-normalized gradient map computed by applying a threshold to pixel values of a gradient magnitude map and then normalizing the pixel values of the gradient magnitude map by row, the gradient magnitude map configured to indicate, for each pixel of the image, an amount the image is changing at the pixel relative to other pixels.

9. A method as described in claim 1, wherein the intermediate saliency maps computed for each of the boundary regions are usable to perform an assessment of boundary simplicity of the image.

10. A method as described in claim 1, further comprising resizing the image to a predefined size before computing the intermediate saliency maps of each of the boundary regions.

11. A method as described in claim 1, further comprising smoothing the image before computing the intermediate saliency maps of each of the boundary regions.

12. A method as described in claim 1, further comprising resizing the base saliency map to have a same size as the image.

13. A method implemented by a computing device, the method comprising:

generating an intermediate saliency map of a boundary region of an image that comprises less than an entirety of the image, the intermediate saliency map generated by computing a saliency value for each pixel contained in the boundary region based on a whitened Euclidean distance from an average color of the boundary region; and

combining the intermediate saliency map with other intermediate saliency maps computed for other boundary regions of the image to generate a base saliency map of the image, the other boundary regions of the image comprising less than the entirety of the image.

14. A method as described in claim 13, wherein the intermediate saliency map is combined with the other intermediate saliency maps to suppress an effect of one or more foreground objects on the base saliency map, the one or more foreground objects that have the effect suppressed occupying a majority of the boundary region or one of the other boundary regions.

15. A method as described in claim 13, wherein the average color of the boundary region comprises a sample mean of the color of the boundary region.

16. A method as described in claim 15, wherein the sample mean is computed for each of the pixels contained in the boundary region in a color space.

17. A method as described in claim 16, further comprising generating a dense saliency map by combining the base saliency map with single-channel saliency maps computed for each color channel of the color space.

18. A method as described in claim 13, further comprising computing a covariance matrix for the pixels contained in the boundary region, the covariance matrix used to whiten a feature space of the boundary region that enables the calculation of the saliency value based on the whitened Euclidean distance from the average color of the boundary region.

19. A system comprising:

a memory;

one or more processors; and

computer-readable storage media storing instructions that are executable via the one or more processors to implement an application to perform operations comprising;

receiving input to perform a post-processing action on an image that depicts a scene, the post-processing action involving an assessment of content in the scene that is performed using a base saliency map of the image; and

generating the base saliency map of the image from intermediate saliency maps of boundary regions of the image, each of the boundary regions comprising less than an entirety of the image and each of the intermediate saliency maps generated by computing a saliency value for each pixel contained in a corresponding said boundary region, the saliency value computed based on a whitened Euclidean distance from an average color of the corresponding said boundary region.

20. A system as described in claim 19, wherein the intermediate saliency maps are combined to generate the base saliency map, the combining effective to suppress an effect of one or more foreground objects on the base saliency map, the one or more foreground objects that have the effect suppressed occupying a majority of one of the boundary regions.