Generating a modified image with additional content provided for a region thereof

• US 9,330,476 B2
• Filed: 05/21/2009
• Issued: 05/03/2016
• Est. Priority Date: 05/21/2009
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Shechtman, Eli, Goldman, Dan
• Primary Examiner(s): TSAI, TSUNG YIN
• Application Number:
  US12/454,666
• Publication Number:
  US 20100296748A1
• Time in Patent Office: 2,539 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation

Abstract

An image is displayed in a computer system. The image includes contents having a feature visible therein. The contents have a region thereof defined to be provided with additional content in generating a modified image. An input is received comprising a semantic mark to be placed on the image. The semantic mark indicates an inside-region part inside the region and an outside-region part outside the region. The additional content for the region is determined using a patch-based optimization algorithm applied to the image. The patch-based optimization algorithm (i) identifies the additional content for the inside-region part based on the outside-region part and not on an area of the image that the semantic mark does not indicate, and (ii) identifies the additional content for a remainder of the region without being restricted to the outside-region part. The modified image having the additional content in the region is stored.

Claims

1. A computer-implemented method comprising:

displaying, by a computer system, an image having a region to which content is to be provided in generating a modified image and an outside region, wherein the region has a first sub-region and a second sub-region, wherein the outside region at least partially surrounds the region;

receiving an input comprising a semantic mark to be placed on the image, the semantic mark indicating the first sub-region and the outside region;

determining, by the computer system applying a patch-based optimization algorithm to the image, first content to be added to the first sub-region and second content to be added to the second sub-region adjacent to the first sub-region, wherein determining the first content and the second content comprises;

restricting, based on the semantic mark indicating the first sub-region and the outside region, the patch-based optimization algorithm to using the outside region for determining the first content such that an area of the image other than the outside region is not used to identify the first content,

identifying, using the patch-based optimization algorithm, the first content based on the outside region to which the patch-based optimization algorithm is restricted and not based on the area of the image that the semantic mark does not indicate, and

identifying the second content for the second sub-region without being restricted to the outside region; and

generating the modified image by modifying the image to include the identified first content in the first sub-region and the identified second content in the second sub-region.

2. The method of claim 1, wherein the first sub-region comprises a hole in the image without the content and wherein the modified image is generated in a hole-filling process that includes the patch-based optimization algorithm.

3. The method of claim 2, wherein the hole is filled such that a feature from the outside region extends into the first sub-region that lacks the content prior to performing the hole-filling process.

4. The method of claim 1, wherein the image is a photograph depicting a physical object, and wherein the region is defined corresponding to a structure of the physical object that is missing from the image.

5. The method of claim 1, wherein the semantic mark is made using at least one of a line tool, an arc tool, a brush tool, an area-selection tool, or combinations thereof.

6. The method of claim 1, wherein the input specifies that a first part of the semantic mark is placed inside the region to indicate the first sub-region and a second part of the semantic mark is placed outside the region to indicate the outside region.

7. The method of claim 1, wherein an identifying characteristic is assigned to the semantic mark, the identifying characteristic distinguishing the semantic mark from an additional semantic mark in the image such that the patch-based optimization algorithm the semantic mark separately from the additional semantic mark.

8. The method of claim 7, wherein the semantic mark and the additional semantic mark cross each other and the patch-based optimization algorithm the semantic mark separately from the additional semantic mark.

9. The method of claim 1, further comprising receiving an additional input that defines an additional semantic constraint for the patch-based optimization algorithm.

10. The method of claim 9, wherein the additional semantic constraint includes at least one of:

a first search-space restriction defined for a coherence aspect of the patch-based optimization algorithm, the first search-space restriction excluding at least a first area of the image from being used in the content to be added to the first sub-region; and

a second search-space restriction defined for a completeness aspect of the patch-based optimization algorithm, the second search-space restriction requiring the content to be complete with regard to a second area of the image indicated by the second search-space restriction.

11. The method of claim 1, further comprising receiving an additional input comprising an additional semantic mark that indicates an additional region from an additional image separate from the image, wherein determining the first content further comprises:

restricting, based on the additional semantic mark indicating the additional region, the patch-based optimization algorithm to using the additional region for determining the first content such that an area of the additional image other than the additional region is not used to identify the first content, and

identifying, using the patch-based optimization algorithm, the first content based on the outside region to which the patch-based optimization algorithm is restricted and not based on the area of the additional image that the additional semantic mark does not indicate.

12. A computer system comprising:

a processing device; and

a non-transitory computer-readable medium, wherein the processing device is configured to execute instructions stored on the non-transitory computer-readable medium and thereby perform operations comprising;

displaying an image having a region to which content is to be provided in generating a modified image and an outside region, wherein the region has a first sub-region and a second sub-region, wherein the outside region at least partially surrounds the region;

receiving an input comprising a semantic mark to be placed on the image, the semantic mark indicating the first sub-region and the outside region;

determining, by applying a patch-based optimization algorithm to the image, first content to be added to the first sub-region and second content to be added to the second sub-region adjacent to the first sub-region, wherein determining the first content and the second content comprises;

restricting, based on the semantic mark indicating the first sub-region and the outside region, the patch-based optimization algorithm to using the outside region for determining the first content such that an area of the image other than the outside region is not used to identify the first content,

identifying, using the patch-based optimization algorithm, the first content based on the outside region to which the patch-based optimization algorithm is restricted and not based on the area of the image that the semantic mark does not indicate, and

identifying the second content for the second sub-region without being restricted to the outside region; and

generating the modified image by modifying the image to include the identified first content in the first sub-region and the identified second content in the second sub-region.

13. The computer system of claim 12, wherein the first sub-region comprises a hole in the image without the content and wherein processing device is configured for generating the modified image by executing a hole-filling process that includes the patch-based optimization algorithm.

14. The computer system of claim 13, wherein the hole is filled such that a feature from the outside region extends into the first sub-region that lacks the content prior to the hole-filling process being executed.

15. The computer system of claim 12, wherein the image is a photograph depicting a physical object, and wherein the region is defined corresponding to a structure of the physical object that is missing from the image.

16. The computer system of claim 12, wherein the input specifies that a first part of the semantic mark is placed inside the region to indicate the first sub-region and a second part of the semantic mark is placed outside the region to indicate the outside region.

17. A non-transitory computer-readable medium storing program instructions that are executable by a processing device, the program instructions comprising:

program instructions for displaying an image having a region to which content is to be provided in generating a modified image and an outside region, wherein the region has a first sub-region and a second sub-region, wherein the outside region at least partially surrounds the region;

program instructions for receiving an input comprising a semantic mark to be placed on the image, the semantic mark indicating the first sub-region and the outside region;

program instructions for determining, by applying a patch-based optimization algorithm to the image, first content to be added to the first sub-region and second content to be added to the second sub-region adjacent to the first sub-region, wherein determining the first content and the second content comprises;

restricting, based on the semantic mark indicating the first sub-region and the outside region, the patch-based optimization algorithm to using the outside region for determining the first content such that an area of the image other than the outside region is not used to identify the first content,

identifying, using the patch-based optimization algorithm, the first content based on the outside region to which the patch-based optimization algorithm is restricted and not based on the area of the image that the semantic mark does not indicate, and

identifying the second content for the second sub-region without being restricted to the outside region; and

program instructions for generating the modified image by modifying the image to include the identified first content in the first sub-region and the identified second content in the second sub-region.

18. The non-transitory computer-readable medium of claim 17, wherein an identifying characteristic is assigned to the semantic mark, the identifying characteristic distinguishing the semantic mark from an additional semantic mark in the image such that the patch-based optimization algorithm the semantic mark separately from the additional semantic mark.

19. The non-transitory computer-readable medium of claim 18, wherein the semantic mark and the additional semantic mark cross each other and program instructions further comprise program instructions for executing the patch-based optimization algorithm such that the patch-based optimization algorithm the semantic mark separately from the additional semantic mark.

20. The non-transitory computer-readable medium of claim 17, further comprising program instructions for receiving an additional input that defines an additional semantic constraint for the patch-based optimization algorithm, wherein the additional semantic constraint includes at least one of:

a first search-space restriction defined for a coherence aspect of the patch-based optimization algorithm, the first search-space restriction excluding at least a first area of the image from being used in the content to be added to the first sub-region; and

a second search-space restriction defined for a completeness aspect of the patch-based optimization algorithm, the second search-space restriction requiring the content to be complete with regard to a second area of the image indicated by the second search-space restriction.