System and method for image composition using non-destructive editing model and fast gradient solver

• US 8,380,005 B1
• Filed: 02/02/2009
• Issued: 02/19/2013
• Est. Priority Date: 02/02/2009
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Jonsson, Michael D.
• Primary Examiner(s): CHANG, JON CARLTON
• Application Number:
  US12/364,378
• Time in Patent Office: 1,478 Days
• Field of Search: 382/282, 382/283, 382/284, 382/299
• US Class Current: 382/282
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text

Abstract

Systems and methods for interactive image compositing may integrate image searching using visual search criteria (e.g., color, composition, tonal distribution, or lighting effects), non-destructive image compositing, and high-performance rendering using a fast gradient solver to provide an efficient workflow for users. An image compositing application, executable by a CPU and/or GPU, may employ a sketch-and-refine approach, whereby a user draws a rough perimeter around an object of interest in an image and (following an initial application of the fast gradient solver to a lower-resolution version of the image) receives a preview of how it would fit into a composition. The fast gradient solver may differentiate between the object and its background by applying artificial intelligence techniques and/or dependent on user-supplied hints. In a second stage, the fast gradient solver may refine the solution for a final rendering of the composite image by operating on successively higher-resolution versions of the image.

Claims

1. A system, comprising:

one or more processors; and

a memory coupled to the one or more processors and storing program instructions executable by the one or more processors to implement an interactive image compositing application configured to;

access data in persistent storage representing an image in response to selection of the image;

display a copy of the image superimposed over another image;

extract an object of interest from the copy of the image;

merge the extracted object and the other image to generate and display a preview of a composite image in which the extracted object appears to be inserted in the other image; and

subsequent to generating and displaying the preview, render a refined version of the composite image;

wherein to render the refined version of the composite image, the application is configured to operate on a representation of a portion of the copy of the image that comprises the object of interest that is at a higher resolution than another representation of the portion of the copy of the image on which the application operates to generate and display the preview of the composite image.

2. The system of claim 1, wherein to extract the object of interest from the copy of the image, the interactive image compositing application is configured to extract the object of interest from the portion of the copy of the image, wherein the portion of the copy of the image is identified by a rough perimeter drawn around the object of interest using an element of a graphical user interface of the interactive image compositing application.

3. The system of claim 1, wherein to extract the object of interest from the copy of the image, the interactive image compositing application is further configured to apply a fast gradient solver of the interactive image compositing application to the other representation of the portion of the copy of the image to distinguish pixels of the object of interest as pixels of a foreground object and to remove any other pixels within the portion of the copy of the image identified by the rough perimeter as background pixels.

4. The system of claim 3,

wherein the portion of the copy of the image is identified by a rough perimeter drawn around the object of interest using an element of a graphical user interface of the interactive image compositing application;

and

wherein said distinguishing pixels of the object of interest is dependent on one or more of;

a distance of the pixels from pixels along the rough perimeter, color values of the pixels, intensity values of the pixels, or gradient values of the pixels.

5. The system of claim 3, wherein said distinguishing pixels of the object of interest is dependent on one or more pixels marked as being pixels of the object of interest or as background pixels using an element of the graphical user interface of the interactive image compositing application.

6. The system of claim 1,

wherein to generate and display the preview of the composite image, the interactive image compositing application is configured to apply a fast gradient solver of the interactive image compositing application to the other representation of the portion of the copy of the image;

and

wherein the other representation is at a lower resolution than the copy of the image.

7. The system of claim 1, wherein to render the refined version of the composite image, the interactive image compositing application is configured to apply a fast gradient solver of the interactive image compositing application to the representation of the portion of the copy of the image that is at the higher resolution.

8. The system of claim 1,

wherein to extract the selected image element from the copy of the image, the interactive image compositing application is further configured to generate an image pyramid comprising two or more levels, each level of the image pyramid comprising data representing a respective version of the portion of the copy of the image having a different image resolution;

and

wherein the image pyramid comprises data representing the representation of the portion of the copy of the image that is at the higher resolution, and data representing the other representation of the portion of the copy of the image.

9. The system of claim 1, wherein in merging the extracted image element and the other image to generate and display the preview, the interactive image compositing application maintains the extracted object and the other image as separately editable images, and does not replace any portion of the data representing the other image with data representing the extracted object.

10. The system of claim 1, wherein the interactive image compositing application is further configured to, prior to said accessing data:

performing a search for images similar to a reference image with respect to at least one visual search criterion;

returning and displaying images meeting the visual search criterion; and

storing one or more images meeting the visual search criterion in the persistent storage.

11. The system of claim 10, wherein said performing a search comprises searching one or more of:

a local image database, a remote image database, a collection of images in the public domain, or a commercial image database.

12. The system of claim 10, wherein said performing a search comprises performing the search over a user-specified search domain.

13. The system of claim 10, wherein the at least one visual search criterion comprises a color, a pattern, a color distribution, an image composition, a distribution of image elements, a tonal distribution, or a lighting effect.

14. The system of claim 1, wherein the interactive image compositing application is further configured to:

modify the extracted object in response to user input to modify the extracted object; and

merge the modified extracted object and the other image to generate and display a preview of a composite image in which the modified extracted object, rather than the object of interest, appears to be inserted in the other image.

15. The system of claim 14, wherein to modify the extracted object, the interactive image compositing application is configured to modify data representing the extracted object without modifying data representing the image or the other image.

16. The system of claim 1, wherein the interactive image compositing application is further configured to store the refined version of the composite image in the persistent storage.

17. The system of claim 1, wherein the one or more processors comprise at least one of:

a general-purpose central processing unit (CPU) or a graphics processing unit (GPU).

18. A computer-readable storage medium, storing program instructions computer-executable to implement an interactive image compositing application configured to:

access data in persistent storage representing an image in response to selection of the image;

display a copy of the image superimposed over another image;

extract an object of interest from the copy of the image;

merge the extracted object and the other image to generate and display a preview of a composite image in which the extracted object appears to be inserted in the other image; and

subsequent to generating and displaying the preview, render a refined version of the composite image;

wherein to render the refined version of the composite image, the application is configured to operate on a representation of a portion of the copy of the image that comprises the object of interest that is at a higher resolution than another representation of the portion of the copy of the image on which the application operates to generate and display the preview of the composite image.

19. The storage medium of claim 18, wherein to extract the object of interest from the copy of the image, the interactive image compositing application is configured to extract the object of interest from the portion of the copy of the image, wherein the portion of the copy of the image is identified by a rough perimeter drawn around the object of interest using an element of a graphical user interface of the interactive image compositing application.

20. The storage medium of claim 18, wherein to extract the object of interest from the copy of the image, the interactive image compositing application is further configured to apply a fast gradient solver of the interactive image compositing application to the other representation of the portion of the copy of the image to distinguish pixels of the object of interest as pixels of a foreground object and to remove any other pixels within the portion of the copy of the image identified by the rough perimeter as background pixels.

21. The storage medium of claim 20,

wherein the portion of the copy of the image is identified by a rough perimeter drawn around the object of interest using an element of a graphical user interface of the interactive image compositing application;

and

wherein said distinguishing pixels of the object of interest is dependent on one or more of;

a distance of the pixels from pixels along the rough perimeter, color values of the pixels, intensity values of the pixels, gradient values of the pixels, or one or more pixels marked as being pixels of the object of interest or as background pixels using an element of the graphical user interface of the interactive image compositing application.

22. The storage medium of claim 18,

wherein to generate and display the preview of the composite image, the interactive image compositing application is configured to apply a fast gradient solver of the interactive image compositing application to the other representation of the portion of the copy of the image, wherein the other representation is at a lower resolution than the copy of the image;

and

wherein to render the refined version of the composite image, the interactive image compositing application is configured to apply the fast gradient solver of the interactive image compositing application to the representation of the portion of the copy of the image that is at the higher resolution.

23. The storage medium of claim 18, wherein in merging the extracted image element and the other image to generate and display the preview, the interactive image compositing application maintains the extracted object and the other image as separately editable images, and does not replace any portion of the data representing the other image with data representing the extracted object.

24. The storage medium of claim 18, wherein the interactive image compositing application is further configured to, prior to said accessing data:

performing a search for images similar to a reference image with respect to at least one visual search criterion;

returning and displaying images meeting the visual search criterion; and

storing one or more images meeting the visual search criterion in the persistent storage.

25. The storage medium of claim 24, wherein said performing a search comprises performing the search over a user-specified search domain, and wherein the search domain comprises one or more of:

a local image database, a remote image database, a collection of images in the public domain, or a commercial image database.

26. The storage medium of claim 24, wherein the at least one visual search criterion comprises a color, a pattern, a color distribution, an image composition, a distribution of image elements, a tonal distribution, or a lighting effect.

27. The storage medium of claim 18, wherein the interactive image compositing application is further configured to:

modify the extracted object in response to user input to modify the extracted object without modifying data representing the image or the other image; and

merge the modified extracted object and the other image to generate and display a preview of a composite image in which the modified extracted object, rather than the object of interest, appears to be inserted in the other image.

28. The storage medium of claim 18, wherein the interactive image compositing application is further configured to store the refined version of the composite image in the persistent storage.

29. A computer-implemented method, comprising:

accessing data in persistent storage representing an image in response to selection of the image;

displaying a copy of the image superimposed over another image;

extracting an object of interest from the copy of the image;

merging the extracted object and the other image to generate and display a preview of a composite image in which the extracted object appears to be inserted in the other image; and

subsequent to generating and displaying the preview, rendering a refined version of the composite image;

wherein said rendering the refined version of the composite image comprises operating on a representation of a portion of the copy of the image that comprises the object of interest that is at a higher resolution than another representation of the portion of the copy of the image that is operated on in said generating and displaying the preview of the composite image.

30. The method of claim 29, wherein said extracting the object of interest from the copy of the image comprises extracting the object of interest from the portion of the copy of the image, wherein the portion of the copy of the image is identified by a rough perimeter drawn around the object of interest using an element of a graphical user interface of an interactive image compositing application.

31. The method of claim 29, wherein said extracting the object of interest from the copy of the image comprises applying a fast gradient solver of an interactive image compositing application to the other representation of the portion of the copy of the image to distinguish pixels of the object of interest as pixels of a foreground object and to remove any other pixels within the portion of the copy of the image identified by the rough perimeter as background pixels.

32. The method of claim 31,

wherein the portion of the copy of the image is identified by a rough perimeter drawn around the object of interest using an element of a graphical user interface of an interactive image compositing application;

and

wherein said distinguishing pixels of the object of interest is dependent on one or more of;

a distance of the pixels from pixels along the rough perimeter, color values of the pixels, intensity values of the pixels, gradient values of the pixels, or one or more pixels marked as being pixels of the object of interest or as background pixels using an element of the graphical user interface of the interactive compositing image application.

33. The method of claim 29,

wherein said generating and displaying the preview of the composite image comprises applying a fast gradient solver of an interactive image compositing application to the other representation of the portion of the copy of the image, wherein the other representation is at a lower resolution than the copy of the image;

and

wherein said rendering the refined version of the composite image comprises applying the fast gradient solver of the interactive image compositing application to the representation of the portion of the copy of the image that is at the higher resolution.

34. The method of claim 29, wherein said merging comprises maintaining the extracted object and the other image as separately editable images, and wherein said merging does not include replacing any portion of the data representing the other image with data representing the extracted object.

35. The method of claim 29, further comprising:

performing a search for images similar to a reference image with respect to at least one visual search criterion;

returning and displaying images meeting the visual search criterion; and

storing one or more images meeting the visual search criterion in the persistent storage.

36. The method of claim 35, wherein said performing a search comprises performing the search over a user-specified search domain, and wherein the search domain comprises one or more of:

a local image database, a remote image database, a collection of images in the public domain, or a commercial image database.

37. The method of claim 35, wherein the at least one visual search criterion comprises a color, a pattern, a color distribution, an image composition, a distribution of image elements, a tonal distribution, or a lighting effect.

38. The method of claim 29, further comprising:

modifying the extracted object in response to user input to modify the extracted object without modifying data representing the image or the other image; and

merging the modified extracted object and the other image to generate and display a preview of a composite image in which the modified extracted object, rather than the object of interest, appears to be inserted in the other image.

39. The method of claim 29, further comprising storing the refined version of the composite image in the persistent storage.

40. A computer-implemented method, comprising:

executing instructions on a specific apparatus so that binary digital electronic signals representing an image are accessed in persistent storage in response to selection of the image;

executing instructions on said specific apparatus so that binary digital electronic signals representing a copy of the image are displayed superimposed over another image;

executing instructions on said specific apparatus so that binary digital electronic signals representing an object of interest are extracted from the binary digital electronic signals representing the copy of the image;

executing instructions on said specific apparatus so that binary digital electronic signals representing the extracted object of interest are merged with binary digital electronic signals representing the other image to generate and display a preview of a composite image in which the extracted object appears to be inserted in the other image;

subsequent to generating and displaying the preview, executing instructions on said specific apparatus so that binary digital electronic signals representing a refined version of the composite image are rendered; and

storing the binary digital electronic signals representing the refined version of the composite image in a memory location of said specific apparatus for later use;

wherein rendering the refined version of the composite image comprises operating on a representation of a portion of the copy of the image that comprises the object of interest that is at a higher resolution than another representation of the portion of the copy of the image that is operated on in generating and displaying the preview of the composite image.