Image cropping suggestion using multiple saliency maps

• US 10,346,951 B2
• Filed: 03/02/2017
• Issued: 07/09/2019
• Est. Priority Date: 10/09/2014
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Inc.
• Inventors: Lin, Zhe, Mech, Radomir, Shen, Xiaohui, Price, Brian L., Zhang, Jianming, Gilra, Anant, Chien, Jen-Chan Jeff
• Primary Examiner(s): Yeh, Euengnan
• Application Number:
  US15/448,138
• Publication Number:
  US 20170178291A1
• Time in Patent Office: 859 Days
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2210/22 Cropping
• CPC Class Codes: G06T 3/04 Context-preserving transformations, e.g. by using an importance map panospheric to cylindrical image transformations G06T3/12
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]

Abstract

Image cropping suggestion using multiple saliency maps is described. In one or more implementations, component scores, indicative of visual characteristics established for visually-pleasing croppings, are computed for candidate image croppings using multiple different saliency maps. The visual characteristics on which a candidate image cropping is scored may be indicative of its composition quality, an extent to which it preserves content appearing in the scene, and a simplicity of its boundary. Based on the component scores, the croppings may be ranked with regard to each of the visual characteristics. The rankings may be used to cluster the candidate croppings into groups of similar croppings, such that croppings in a group are different by less than a threshold amount and croppings in different groups are different by at least the threshold amount. Based on the clustering, croppings may then be chosen, e.g., to present them to a user for selection.

Claims

1. A method implemented by a computing device, the method comprising:

obtaining input to initiate cropping of a scene;

computing multiple different saliency maps from an image of the scene; and

generating one or more suggested image croppings of the scene based on rankings assigned to multiple candidate croppings of the scene, wherein the rankings reflect an assessment of;

composition quality of the multiple candidate croppings, the assessment of the composition quality performed using a first combination of the multiple different saliency maps;

degrees of preservation of content appearing in the scene for the multiple candidate croppings, the assessment of the degrees of preservation of content performed using a second combination of the multiple different saliency maps; and

boundary simplicity of the multiple candidate croppings, the assessment of the boundary simplicity performed using a third combination of the multiple different saliency maps.

2. A method as described in claim 1, wherein the first combination of saliency maps includes:

a dense saliency map that combines a saliency map computed from red, green, blue (RGB) information of the image and single-channel saliency maps computed for each color channel of the RGB information; and

a saliency edge map computed using the dense saliency map and a gradient magnitude map that indicates, for each pixel of the image, an amount the image is changing at the pixel relative to other pixels.

3. A method as described in claim 1, wherein the second combination of saliency maps includes:

a base saliency map that is computed from RGB information of the image;

a dense saliency map that combines the base saliency map and single-channel saliency maps computed for each color channel of the RGB information; and

a row-normalized gradient map that is computed by applying a threshold to pixel values of a gradient magnitude map and then normalizing the pixel values of the gradient magnitude map by row, the gradient magnitude map configured to indicate, for each pixel of the image, an amount the image is changing at the pixel relative to other pixels.

4. A method as described in claim 1, wherein the third combination of saliency maps includes edge saliency maps computed for each of:

a left edge boundary region of the cropping;

a right edge boundary region of the cropping;

a top edge boundary region of the cropping; and

a bottom edge boundary region of the cropping.

5. A method as described in claim 1, wherein generating the one or more suggested image croppings includes clustering the multiple candidate croppings into clusters of similar candidate croppings based on the assessment.

6. A method as described in claim 5, further comprising selecting the one or more suggested image croppings based on being ranked highest in a respective cluster.

7. A method as described in claim 5, further comprising selecting the one or more suggested image croppings from different clusters.

8. A method as described in claim 5, wherein the candidate croppings in one cluster are different from the candidate croppings in another cluster by at least a threshold amount.

9. A method as described in claim 5, wherein the candidate croppings in a given cluster are different, one candidate cropping from another, by less than a threshold amount.

10. A system comprising:

one or more modules implemented at least partially in hardware, the one or more modules configured to perform operations comprising;

displaying, via a user interface, one or more suggested croppings of a scene, the suggested croppings being selected from a plurality of candidate croppings of the scene and selected for suggestion based on selection of the suggested cropping s from multiple clusters of the candidate croppings, each of the clusters having different candidate croppings from the candidate croppings in the other clusters by at least a threshold amount; and

receiving, via the user interface, a selection of one of the one or more suggested croppings to apply a selected cropping of the scene.

11. A system as described in claim 10, wherein the operations further comprise:

ranking each of the candidate croppings with regard to multiple different visual characteristics;

calculating an average ranking for each of the candidate croppings based on the rankings of the candidate croppings in each of the different visual characteristics; and

using the average rankings to cluster the candidate croppings.

12. A system as described in claim 10, wherein the operations further comprise selecting the suggested croppings from the multiple clusters of candidate croppings.

13. A system as described in claim 12, wherein the suggested croppings are selected from the multiple clusters of candidate croppings based on being ranked highest in a respective cluster.

14. A system as described in claim 10, wherein the candidate croppings in a given cluster are different, one candidate cropping from another, by less than the threshold amount.

15. A method implemented by a computing device, the method comprising:

computing scores for candidate image croppings of a scene, the scores being indicative of visual characteristics ascertained from multiple different types of saliency maps of the scene; and

selecting a highest-ranked image cropping from different clusters of the candidate image croppings that are clustered based on the computed scores.

16. A method as described in claim 15, further comprising ranking the candidate image croppings within a given cluster.

17. A method as described in claim 15, wherein the candidate image croppings in one cluster are different from the candidate image croppings in other clusters by at least a threshold amount.

18. A method as described in claim 15, wherein the candidate image cropping s in a given cluster are different, one candidate image cropping from another, by less than a threshold amount.

19. A method as described in claim 15, wherein the scores comprise multiple component scores that are computed for each candidate image cropping and are indicative of a different visual characteristic established for visually pleasing croppings.

20. A method as described in claim 19, wherein the component scores include at least one of:

a composition quality score;

a content preservation score; and

a boundary simplicity score.