AUTOMATIC OBJECT REPLACEMENT IN AN IMAGE

• US 20200134834A1
• Filed: 10/31/2018
• Published: 04/30/2020
• Est. Priority Date: 10/31/2018
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Inc.
• Inventors: Pao, I-Ming, Kong, Sarah Aye, Erickson, Alan Lee, Sunkavalli, Kalyan, Byun, Hyunghwan
• Granted Patent: US 11,042,990 B2
• Time in Patent Office: Days
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06T 11/40 Filling a planar surface by adding surface attributes, e.g. colour or texture
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation

Abstract

Systems and techniques for automatic object replacement in an image include receiving an original image and a preferred image. The original image is automatically segmented into an original image foreground region and an original image object region. The preferred image is automatically segmented into a preferred image foreground region and a preferred image object region. A composite image is automatically composed by replacing the original image object region with the preferred image object region such that the composite image includes the original image foreground region and the preferred image object region. An attribute of the composite image is automatically adjusted.

Claims

1. A computer-implemented method for automatic object replacement in an image, the method comprising:

receiving an original image and a sky image;

automatically segmenting the original image into an original image foreground region and an original image sky region;

automatically segmenting the sky image into a sky image foreground region and a sky image sky region;

automatically composing a composite image by replacing the original image sky region with the sky image sky region such that the composite image includes the original image foreground region and the sky image sky region; and

automatically adjusting an attribute of the composite image.

2. The computer-implemented method as in claim 1, wherein:

automatically segmenting the original image comprises;

generating a probability map for the original image that identifies a probability for each pixel in the original image of whether the pixel is part of the original image foreground region or the original image sky region, and

segmenting each pixel in the original image into the original image foreground region and the original image sky region using the probability map for the original image; and

automatically segmenting the sky image comprises;

generating a probability map for the sky image that identifies a probability for each pixel in the sky image of whether the pixel is part of the sky image foreground region or the sky image sky region, and

segmenting each pixel in the sky image into the sky image foreground region and the sky image sky region using the probability map for the sky image.

3. The computer-implemented method as in claim 2, wherein:

generating the probability map for the original image comprises generating the probability map for the original image using a convolutional neural network (CNN); and

generating the probability map for the sky image comprises generating the probability map for the sky image using the CNN.

4. The computer-implemented method as in claim 1, wherein automatically composing the composite image comprises:

detecting a maximum tight bounding box in the sky image sky region in which all pixels in the maximum tight bounding box are sky pixels;

determining if the maximum tight bounding box covers the original image sky region;

replacing the original image sky region with the maximum tight bounding box if the maximum tight bounding box covers the original image sky region; and

upscaling the maximum tight bounding box to cover the original image sky region if the maximum tight bounding box does not cover the original image sky region and replacing the original image sky region with the upscaled maximum tight bounding box.

5. The computer-implemented method as in claim 1, wherein automatically adjusting the attribute of the composite image comprises automatically adjusting a color of the original image foreground region of the composite image.

6. The computer-implemented method as in claim 1, wherein automatically adjusting the attribute of the composite image comprises:

generating color statistics for the original image foreground region and the sky image foreground region;

generating a transfer function using the color statistics; and

adjusting the color of the composite image by applying the transfer function to the original image foreground region of the composite image.

7. The computer-implemented method as in claim 1, wherein automatically adjusting the attribute of the composite image comprises:

generating color statistics for the sky image sky region and the original image foreground region;

searching a database of sky regions using the color statistics of the sky image sky region to find a matching sky region having a style matching the sky image sky region;

generating a transfer function using the color statistics of the original image foreground region and color statistics of a foreground region of the matching sky region from the database of sky regions; and

adjusting the color of the composite image by applying the transfer function to the original image foreground region of the composite image.

8. The computer-implemented method as in claim 1, further comprising storing the original image foreground region as a first layer and the sky image sky region as a second layer, wherein the second layer is separate from the first layer and the second layer is user-moveable relative to the first layer.

9. A system for automatic object replacement in an image, the system comprising:

at least one memory including instructions; and

at least one processor that is operably coupled to the at least one memory and that is arranged and configured to execute instructions that, when executed, cause the at least one processor to implement an application, the application comprising;

a convolutional neural network (CNN) that is arranged and configured to;

receive an original image and a sky image,

segment the original image into an original image foreground region and an original image sky region by generating a probability map for the original image that identifies a probability for each pixel in the original image of whether the pixel is part of the original image foreground region or the original image sky region, and

segment the sky image into a sky image foreground region and a sky image sky region by generating a probability map for the sky image that identifies a probability for each pixel in the sky image of whether the pixel is part of the sky image foreground region or the sky image sky region; and

a composition module that is arranged and configured to;

detect a maximum tight bounding box in the sky image sky region in which all pixels in the maximum tight bounding box are sky pixels,

determine if the maximum tight bounding box covers the original image sky region,

replace the original image sky region with the maximum tight bounding box if the maximum tight bounding box covers the original image sky region to form a composite image, and

upscale the maximum tight bounding box to cover the original image sky region if the maximum tight bounding box does not cover the original image sky region and replace the original image sky region with the upscaled maximum tight bounding box to form the composite image, wherein the composite image includes the original image foreground region and the sky image sky region.

10. The system of claim 9, wherein the application further comprises a color adjustment module that is arranged and configured to adjust a color of the composite image.

11. The system of claim 10, wherein the color adjustment module adjusts the color of the composite image by adjusting a color of the original image foreground region of the composite image.

12. The system of claim 10, wherein the color adjustment module adjusts the color of the composite image by:

generating color statistics for the original image foreground region and the sky image foreground region;

generating a transfer function using the color statistics; and

adjusting the color of the composite image by applying the transfer function to the original image foreground region of the composite image.

13. The system of claim 10, wherein the color adjustment module adjusts the color of the composite image by:

generating color statistics for the sky image sky region and the original image foreground region;

searching a database of sky regions using the color statistics of the sky image sky region to find a matching sky region having a style matching the sky image sky region;

generating a transfer function using the color statistics of the original image foreground region and color statistics of a foreground region of the matching sky region from the database of sky regions; and

adjusting the color of the composite image by applying the transfer function to the original image foreground region of the composite image.

14. The system of claim 9, wherein the application stores the original image foreground region as a first layer in the at least one memory and stores the sky image sky region as a second layer in the at least one memory, wherein the second layer is separate from the first layer.

15. The system of claim 14, wherein the application further comprises a graphical user interface (GUI), wherein the GUI includes:

a canvas that is configured to display the first layer and the second layer as the composite image, wherein the second layer is user-moveable relative to the first layer.

16. The system of claim 15, wherein the GUI further includes a layer area that is configured to display the first layer with a layer mask to mask out the original image sky region such that only the original image foreground region is displayed.

17. A computer program product for automatic object replacement in an image, the computer program product being tangibly embodied on a non-transitory computer-readable storage medium and comprising instructions that, when executed by at least one computing device, are configured to cause the at least one computing device to:

receive an original image and a preferred image using a graphical user interface (GUI);

automatically segment the original image into an original image foreground region and an original image object region;

automatically segment the preferred image into a preferred image foreground region and a preferred image object region;

automatically compose a composite image by replacing the original image object region with the preferred image object region such that the composite image includes the original image foreground region and the preferred image object region;

automatically adjust an attribute of the composite image; and

display the composite image on the GUI.

18. The computer program product of claim 17, wherein:

the instructions that, when executed, cause the at least one computing device to automatically segment the original image comprise instructions that, when executed, cause the at least one computing device to;

generate a probability map for the original image that identifies a probability for each pixel in the original image of whether the pixel is part of the original image foreground region or the original image object region, and

segment each pixel in the original image into the original image foreground region and the original image object region using the probability map for the original image; and

the instructions that, when executed, cause the at least one computing device to automatically segment the preferred image comprise instructions that, when executed, cause the at least one computing device to;

generate a probability map for the preferred image that identifies a probability for each pixel in the preferred image of whether the pixel is part of the preferred image foreground region or the preferred image object region, and

segment each pixel in the preferred image into the preferred image foreground region and the preferred image object region using the probability map for the preferred image.

19. The computer program product of claim 17, wherein the instructions that, when executed, cause the at least one computing device to automatically compose the composite image comprise instructions that, when executed, cause the at least one computing device to:

detect a maximum tight bounding box in the preferred image object region in which all pixels in the maximum tight bounding box are object pixels;

determine if the maximum tight bounding box covers the original image object region;

replace the original image object region with the maximum tight bounding box if the maximum tight bounding box covers the original image object region; and

upscale the maximum tight bounding box to cover the original image object region if the maximum tight bounding box does not cover the original image object region and replacing the original image object region with the upscaled maximum tight bounding box.

20. The computer program product of claim 17, further comprising instructions that, when executed, cause the at least one computing device to:

store the original image foreground region as a first layer and store the preferred image object region as a second layer in the at least one memory, wherein the second layer is separate from the first layer, wherein

the GUI includes a canvas that is configured to display the first layer and the second layer as the composite image, wherein the second layer is user-moveable relative to the first layer.