Image Cropping Suggestion Using Multiple Saliency Maps

• US 20160104055A1
• Filed: 10/09/2014
• Published: 04/14/2016
• Est. Priority Date: 10/09/2014
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Lin, Zhe, Mech, Radomir, Shen, Xiaohui, Price, Brian L., Zhang, Jianming, Chien, Jen-Chan Jeff, Gilra, Anant
• Granted Patent: US 9,626,584 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2210/22 Cropping
• CPC Class Codes: G06T 3/04 Context-preserving transformations, e.g. by using an importance map panospheric to cylindrical image transformations G06T3/12
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]

Abstract

Image cropping suggestion using multiple saliency maps is described. In one or more implementations, component scores, indicative of visual characteristics established for visually-pleasing croppings, are computed for candidate image croppings using multiple different saliency maps. The visual characteristics on which a candidate image cropping is scored may be indicative of its composition quality, an extent to which it preserves content appearing in the scene, and a simplicity of its boundary. Based on the component scores, the croppings may be ranked with regard to each of the visual characteristics. The rankings may be used to cluster the candidate croppings into groups of similar croppings, such that croppings in a group are different by less than a threshold amount and croppings in different groups are different by at least the threshold amount. Based on the clustering, croppings may then be chosen, e.g., to present them to a user for selection.

Claims

1. A method implemented by a computing device, the method comprising:

computing component scores for candidate image croppings of a scene, the component scores indicative of visual characteristics established for visually pleasing croppings and computed using multiple different saliency maps; and

choosing one or more image croppings from the candidate image croppings using the component scores.

2. A method as described in claim 1, wherein each of the component scores is indicative of a different visual characteristic.

3. A method as described in claim 1, wherein each of the component scores is computed using multiple different saliency maps.

4. A method as described in claim 1, wherein at least one of the component scores indicates a composition quality of the candidate image croppings, the composition quality determined based on a comparison of the candidate image croppings to composition properties derived from well-composed images.

5. A method as described in claim 1, wherein at least one of the component scores is indicative of whether at least a portion of content appearing in the scene is preserved by the candidate image croppings.

6. A method as described in claim 1, wherein at least one of the component scores is indicative of a simplicity of a boundary of the candidate image croppings.

7. A method as described in claim 1, wherein at least one of the component scores indicates a degree to which the candidate image croppings preserve specified regions-to-keep and exclude specified regions-to-remove.

8. A method as described in claim 7, wherein at least one of the regions-to-keep or the regions-to-remove corresponds to an object that is automatically detected in the image without user interaction.

9. A method as described in claim 7, wherein at least one of the regions-to-keep or the regions-to-remove is specified according to user input to keep or remove the at least one region-to-keep or region-to-remove.

10. A method as described in claim 1, further comprising:

ranking the candidate image croppings with regard to each of the visual characteristics based on corresponding component scores;

calculating an average ranking for each of the candidate image croppings based on the rankings of the candidate image croppings in each of the visual characteristics; and

using the average rankings to cluster the candidate image croppings into clusters of similar image croppings.

11. A method as described in claim 10, wherein choosing the one or more image croppings from the candidate image croppings includes choosing the candidate image croppings that are ranked highest in a respective cluster.

12. A method as described in claim 10, wherein choosing the one or more image croppings from the candidate image croppings includes choosing the candidate image croppings from different clusters.

13. A method as described in claim 10, wherein the candidate image croppings in one cluster are different from the candidate image croppings in another cluster by at least a threshold amount.

14. A method as described in claim 10, wherein the candidate image croppings in a given cluster are different, one candidate image cropping from another, by less than a threshold amount.

15. A method implemented by a computing device, the method comprising:

obtaining input to initiate cropping of a scene;

computing multiple different saliency maps from an image of the scene; and

generating one or more suggested image croppings of the scene based on rankings assigned to multiple candidate croppings that reflect an assessment of;

composition quality of the multiple candidate croppings, the assessment of the composition quality performed using a first combination of the multiple different saliency maps;

degrees of preservation of content appearing in the scene for the multiple candidate croppings, the assessment of the degrees of preservation of content performed using a second combination of the multiple different saliency maps; and

boundary simplicity of the multiple candidate croppings, the assessment of the boundary simplicity performed using a third combination of the multiple different saliency maps.

16. A method as described in claim 15, wherein the first combination of saliency maps includes:

a dense saliency map that combines a saliency map computed from red, green, blue (RGB) information of the image and single-channel saliency maps computed for each color channel of the RGB information; and

a saliency edge map computed using the dense saliency map and a gradient magnitude map that indicates, for each pixel of the image, an amount the image is changing at the pixel relative to other pixels.

17. A method as described in claim 15, wherein the second combination of saliency maps includes:

a base saliency map that is computed from RGB information of the image;

a dense saliency map that combines the base saliency map and single-channel saliency maps computed for each color channel of the RGB information; and

a row-normalized gradient map that is computed by applying a threshold to pixel values of a gradient magnitude map and then normalizing the pixel values of the gradient magnitude map by row, the gradient magnitude map configured to indicate, for each pixel of the image, an amount the image is changing at the pixel relative to other pixels.

18. A method as described in claim 15, wherein the third combination of saliency maps includes edge saliency maps computed for each of:

a left edge boundary region of the cropping;

a right edge boundary region of the cropping;

a top edge boundary region of the cropping; and

a bottom edge boundary region of the cropping.

19. A system comprising:

one or more modules implemented at least partially in hardware, the one or more modules configured to perform operations comprising;

displaying, via a user interface, one or more suggested croppings of a scene, the suggested croppings being selected from a plurality of candidate croppings for suggestion based on selection of the suggested croppings from multiple clusters of croppings, each of the clusters having croppings that are different from croppings in the other clusters by at least a threshold amount; and

receiving, via the user interface, a selection of one of the one or more suggested croppings to apply a selected cropping of the scene.

20. A system as described in claim 19, wherein the operations further comprise:

ranking each of the candidate croppings with regard to multiple different visual characteristics;

calculating an average ranking for each of the candidate croppings based on the rankings of the candidate croppings in each of the different visual characteristics; and

using the average rankings to cluster the candidate croppings.