System and method for content aware hybrid cropping and seam carving of images

• US 8,963,960 B2
• Filed: 05/20/2009
• Issued: 02/24/2015
• Est. Priority Date: 05/20/2009
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Gilra, Anant
• Primary Examiner(s): WANG, JIN CHENG
• Application Number:
  US12/469,486
• Publication Number:
  US 20130127915A1
• Time in Patent Office: 2,106 Days
• Field of Search: 345629-667, 382/162, 382/195, 382/276
• US Class Current: 345/660
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 3/04 Context-preserving transformations, e.g. by using an importance map panospheric to cylindrical image transformations G06T3/12
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G09G 2340/0407 Resolution change, inclusive of the use of different resolutions for different screen areas
• CPC Class Codes: G09G 2340/045 Zooming at least part of an image, i.e. enlarging it or shrinking it
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H04N 1/3872 Repositioning or masking
• CPC Class Codes: H04N 1/3875 combined with enlarging or reducing enlarging or reducing per se H04N1/393
• CPC Class Codes: H04N 1/393 Enlarging or reducing
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

A system and method for performing content aware cropping/expansion may be applied to resize an image or to resize a selected object therein. An image object may be selected using an approximate bounding box of the object. The system may receive input indicating a lowest priority edge or corner of the image or object to be resized (e.g., using a drag operation). Respective energy values for some pixels of the image and/or of the object to be resized may be weighted based on their distance from the lowest priority edge/corner and/or on a cropping or expansion graph, and relative costs may be determined for seams of the image dependent on the energy values. Low cost seams may be removed or replicated in different portions of the image and/or the object to modify the image. The selected object may be resized using interpolated scaling and patched over the modified image.

Claims

1. A method, comprising:

using a computer to perform;

accessing data representing an image;

receiving an input defining a bounding box of a portion of the image, the bounding box having one or more edges and corners;

receiving an input involving selection of the one or more edges or corners of the bounding box thereby indicating an amount by which the portion of the image defined by the bounding box is to be resized;

receiving an input indicating a relative priority of a particular edge or corner of the bounding box with respect to other edges or corners of the bounding box;

resizing the portion of the image based on the relative priority indicated by the input for the particular edge or corner with respect to the other edges or corners of the bounding box; and

generating a modified image comprising the resized portion of the image based on the relative priority.

2. The method of claim 1, wherein said resizing comprises removing one or more low cost seams of the portion of the image or replicating the one or more low cost seams of the portion of the image.

3. The method of claim 1,

wherein the receiving of the input indicating a relative priority comprises receiving input identifying the particular edge or corner as a user-specified lowest priority edge or corner;

and

further comprising determining respective energy values for pixels of the image by for at least some of the energy values, applying a weighting to each energy value dependent on a distance of a respective said pixel to the user-specified lowest priority edge or corner such that lower weightings are applied to the pixels close to the user-specified lowest priority edge or corner than to the pixels farther away from the user-specified lowest priority edge or corner.

4. The method of claim 1,

wherein the receiving of the input indicating the relative priority comprises receiving input identifying a cropping or expansion graph that maps the particular said edge or corner of the portion of the image to a respective said priority;

wherein the cropping or extension graph further maps two or more said portions of the image to respective said priorities; and

further comprising determining energy values for pixels of the image by applying a weighting to each energy value dependent on the cropping or expansion graph such that lower weightings are applied to pixels in portions of the image mapped to low priorities than to pixels in portions of the image mapped to high priorities.

5. The method of claim 1, further comprising:

using the computer to perform;

dividing the image into three regions in one dimension corresponding to a dimension of said resizing, wherein the three regions comprise a center region comprising the portion of the image, and regions on each side of the center region;

resizing the regions on each side of the center region along one or more low cost seams of the regions to produce resized regions on each side of the center region;

wherein said resizing the a portion of the image comprises resizing the center region; and

wherein the modified image further comprises the resized regions on each side of the center region.

6. The method of claim 5,

wherein said resizing the portion of the image comprises removing the one or more low cost seams of the center region;

and

wherein said resizing the regions on each side of the center region comprises replicating the one or more low cost seams of the regions on each side of the center region.

7. The method of claim 5,

wherein said resizing the portion of the image comprises replicating the one or more low cost seams of the center region;

and

wherein said resizing the regions on each side of the center region comprises removing the one or more low cost seams of the regions on each side of the center region.

8. The method of claim 5,

wherein the portion of the image comprises a portion of the image less than the entire center region;

and

further comprising using the computer to perform;

scaling the portion of the image; and

patching the scaled a portion of the image over the resized portion of the image in the resized center region of the modified image to produce a refined modified image.

9. The method of claim 5, further comprising:

using the computer to perform;

receiving input indicating a second amount by which the resized portion of the image is to be further resized in a second dimension and input indicating a relative priority of another particular said edge or corner of the resized portion of the image with respect to the other edges or corners of the resized the portion of the image;

determining relative costs of a plurality of seams of the modified image dependent on respective energy values for pixels of the modified image, wherein at least some of the energy values for pixels of the modified image are dependent on the input indicating the relative priority of the particular edge or corner of the resized portion of the image with respect to the other edges or corners of the resized portion of the image;

dividing the modified image into three regions in the second dimension, the three regions comprising;

a center region of the modified image comprising the resized a portion of the image, and regions of the modified image on each side of the center region of the modified image;

resizing the center region of the modified image by the second amount along one or more low cost seams of the center region of the modified image;

resizing the regions of the modified image on each side of the center region of the modified image in the second dimension along one or more low cost seams of the regions of the modified image to produce resized regions of the modified image on each side of the center region of the modified image;

wherein the at least some of the energy values for pixels of the modified image comprise energy values for pixels of the regions of the modified image on each side of the center region of the modified image; and

generating a further modified image comprising the resized center region of the modified image and the resized regions of the modified image on each side of the center region of the modified image; and

storing data representing the further modified image.

10. The method of claim 1, further comprising:

using the computer to perform;

receiving input indicating a direction in which the portion of the image is to be resized or a registration point for said resizing.

11. The method of claim 1, wherein the portion of the image comprises the entire image.

12. The method of claim 1, further comprising:

generating a version of the image having a lower resolution than the image;

determining respective energy values for pixels of the lower resolution version of the image, wherein at least some of the energy values are dependent on the input indicating a relative priority;

determining relative costs of a plurality of seams of the lower resolution version of the image dependent on the respective energy values for pixels of the lower resolution version of the image;

resizing the portion of the lower resolution version of the image by the indicated amount along one or more low cost seams of the portion of the lower resolution version of the image; and

generating a preview image comprising the resized portion of the lower resolution version of the image.

13. A tangible computer-readable storage medium storing program instructions computer-executable to implement:

accessing data representing an image;

receiving an input defining a bounding box of a portion of the image, the bounding box having one or more edges and corners;

receiving input that includes selection of the one or more edges or corners of the bounding box thereby indicating an amount by which the portion of the image is to be resized;

receiving input indicating a relative priority of a particular edge or corner of the bounding box with respect to other edges or corners of the bounding box of the portion of the image;

determining respective energy values for pixels of the image, at least some of the energy values dependent on the input indicating the relative priority of the particular edge or corner of the portion of the image with respect to the other edges or corners of the portion of the image;

determining relative costs of a plurality of seams of the image dependent on the respective energy values based on the relative priority indicated by the input for the particular edge or corner with respect to the other edges or corners of the bounding box;

resizing at least the portion of the image by the indicated amount along one or more low cost seams of the portion of the image, said resizing including removing the one or more low cost seams of the portion of the image or replicating the one or more low cost seams of the portion of the image; and

generating a modified image comprising the resized portion of the image.

14. The tangible computer-readable storage medium of claim 13,

wherein said receiving input indicating the relative priority comprises receiving input identifying the particular edge or corner as a user-specified lowest priority edge or corner;

and

wherein said determining respective energy values comprises, for the at least some of the energy values, applying a weighting to each said energy value dependent on a distance of the respective pixel to the user-specified lowest priority edge or corner such that lower weightings are applied to pixels close to the user-specified lowest priority edge or corner than to pixels farther from the user-specified lowest priority edge or corner.

15. The tangible computer-readable storage medium of claim 13,

wherein said receiving input indicating a relative priority comprises receiving input identifying a cropping or expansion graph that maps the particular said edge or corner of the portion of the image to a respective priority;

wherein the cropping or extension graph further maps two or more portions of the image to respective priorities; and

wherein said determining respective energy values comprises, for the at least some of the energy values, applying a weighting to each energy value dependent on the cropping or expansion graph such that lower weightings are applied to pixels in portions of the image mapped to low priorities than to pixels in portions of the image mapped to high priorities.

16. The tangible computer-readable storage medium of claim 13, wherein the program instructions are further computer-executable to implement:

dividing the image into three regions in one dimension corresponding to a dimension of said resizing, wherein the three regions comprise a center region comprising the portion of the image, and regions on each side of the center region;

resizing the regions on each side of the center region along one or more low cost seams of the regions to produce resized regions on each side of the center region;

wherein said resizing the portion of the image comprises resizing the center region;

wherein the at least some of the energy values comprise the respective energy values for pixels of the regions on each side of the center region; and

wherein the modified image further comprises the resized regions on each side of the center region.

17. The tangible computer-readable storage medium of claim 13, wherein the portion of the image comprises a portion of the image less than the entire center region, and wherein the program instructions are further computer-executable to implement:

scaling the portion of the image; and

patching the scaled portion of the image over the resized portion of the image in the resized center region of the modified image to produce a refined modified image.

18. The tangible computer-readable storage medium of claim 13,

wherein the portion of the image comprises an entirety of image;

and

wherein the at least some of the energy values comprises the respective energy values for the pixels of the entirety of the image.

19. A system, comprising:

one or more processors; and

a memory coupled to the one or more processors and storing program instructions executable by the one or more processors to implement;

accessing data representing an image;

receiving an input defining a bounding box of a portion of the image, the bounding box having one or more edges and corners;

receiving an input involving selection of the one or more edges or corners of the bounding box thereby indicating an amount by which the portion of the image is to be resized;

receiving an input indicating a relative priority of a particular edge or corner of the bounding box with respect to other edges or corners of the bounding box;

resizing the portion of the image based on the relative priority indicated by the input for the particular edge or corner with respect to the other edges or corners of the bounding box; and

generating a modified image comprising the resized portion of the image based on the relative priority.

20. The system of claim 19,

wherein said receiving input indicating a relative priority comprises receiving input identifying the particular edge or corner as a user-specified lowest priority edge or corner;

and

further comprising determining respective energy values by applying a weighting to each energy value dependent on a distance of a respective pixel to the user-specified lowest priority edge or corner such that lower weightings are applied to pixels close to the user-specified lowest priority edge or corner than to pixels farther from the user-specified lowest priority edge or corner.

21. The system of claim 19,

wherein said receiving input indicating a relative priority comprises receiving input identifying a cropping or expansion graph that maps the particular edge or corner of the portion of the image to a respective priority;

wherein the cropping or extension graph further maps two or more portions of the image to respective priorities; and

further comprising determining respective energy values by applying a weighting to each energy value dependent on the cropping or expansion graph such that lower weightings are applied to pixels in portions of the image mapped to low priorities than to pixels in portions of the image mapped to high priorities.

22. The system of claim 19, wherein the program instructions are further computer-executable to implement:

dividing the image into three regions in one dimension corresponding to a dimension of said resizing, wherein the three regions comprise a center region comprising the portion of the image, and regions on each side of the center region;

resizing the regions on each side of the center region along one or more low cost seams of the regions to produce resized regions on each side of the center region;

wherein said resizing the portion of the image comprises resizing the center region;

wherein at least some energy values comprise respective energy values for pixels of the regions on each side of the center region; and

wherein the modified image further comprises the resized regions on each side of the center region.

23. The system of claim 19, wherein the portion of the image comprises a portion of the image less than an entirety of the center region, and wherein the program instructions are further computer-executable to implement:

scaling the portion of the image; and

patching the scaled portion of the image over the resized portion of the image in the resized center region of the modified image to produce a refined modified image.

24. The system of claim 19, wherein the portion of the image comprises an entirety of the image.