Generating a modified image with additional content provided for a region thereof

• US 20100296748A1
• Filed: 05/21/2009
• Published: 11/25/2010
• Est. Priority Date: 05/21/2009
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Shechtman, Eli, Goldman, Dan
• Granted Patent: US 9,330,476 B2
• Time in Patent Office: Days
• US Class Current: 382/254
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation

Abstract

An image is displayed in a computer system. The image includes contents having a feature visible therein. The contents have a region thereof defined to be provided with additional content in generating a modified image. An input is received comprising a semantic mark to be placed on the image. The semantic mark indicates an inside-region part inside the region and an outside-region part outside the region. The additional content for the region is determined using a patch-based optimization algorithm applied to the image. The patch-based optimization algorithm (i) identifies the additional content for the inside-region part based on the outside-region part and not on an area of the image that the semantic mark does not indicate, and (ii) identifies the additional content for a remainder of the region without being restricted to the outside-region part. The modified image having the additional content in the region is stored.

Claims

1. A computer-implemented method for generating a modified image, the method comprising:

displaying an image in a computer system, the image comprising contents that have a feature visible therein, the contents having a region thereof defined to be provided with additional content in generating a modified image;

receiving an input comprising a semantic mark to be placed on the image, the semantic mark indicating an inside-region part inside the region and an outside-region part outside the region;

determining the additional content for the region using a patch-based optimization algorithm applied to the image, the patch-based optimization algorithm (i) identifying the additional content for the inside-region part based on the outside-region part and not on an area of the image that the semantic mark does not indicate, and (ii) identifying the additional content for a remainder of the region without being restricted to the outside-region part; and

storing the modified image having the additional content in the region.

2. The method of claim 1, wherein the region comprises a hole in the image without the contents and wherein the modified image is generated in a hole-filling process.

3. The method of claim 2, wherein the hole is filled such that the feature extends into the region that previously did not contain the contents.

4. The method of claim 1, wherein the image is a photograph and the feature is a physical object, and wherein the region is defined corresponding to a structure of the physical object that is missing in the feature.

5. The method of claim 1, wherein the semantic mark is made using at least one of:

a line tool, an arc tool, a brush tool, an area-selection tool, and combinations thereof.

6. The method of claim 1, wherein a part of the semantic mark is placed inside the region to indicate the inside-region part, and another part of the semantic mark is placed outside the region to indicate the outside-region part.

7. The method of claim 1, wherein an identifying characteristic is assigned to the semantic mark, the identifying characteristic distinguishing the semantic mark from at least one other semantic mark in the image such that the patch-based optimization algorithm processes the semantic marks separately.

8. The method of claim 7, wherein the semantic marks cross each other and the patch-based optimization algorithm processes the semantic marks separately.

9. The method of claim 1, further comprising receiving another input that defines an additional semantic constraint for the patch-based optimization process.

10. The method of claim 9, wherein the additional constraint includes at least one of:

a first search-space restriction defined for a coherence aspect of the patch-based optimization algorithm, the first search-space restriction excluding at least a first area of the image from being used in the additional content; and

a second search-space restriction defined for a completeness aspect of the patch-based optimization algorithm, the second search-space restriction requiring the additional content to be complete with regard to a second area of the image indicated by the second search-space restriction.

11. The method of claim 1, wherein the outside-region part is located in another image separate from the image having the region.

17. The computer program product of claim 1, wherein the input control associates an identifying characteristic with the semantic mark, the identifying characteristic distinguishing the semantic mark from at least one other semantic mark in the image such that the patch-based optimization algorithm processes the semantic marks separately.

12. A computer program product tangibly embodied in a tangible program carrier and comprising instructions that when executed by a processor perform a method comprising:

displaying an image in a computer system, the image comprising contents that have a feature visible therein, the contents having a region thereof defined to be provided with additional content in generating a modified image;

receiving an input comprising a semantic mark to be placed on the image, the semantic mark indicating an inside-region part inside the region and an outside-region part outside the region;

determining the additional content for the region using a patch-based optimization algorithm applied to the image, the patch-based optimization algorithm (i) identifying the additional content for the inside-region part based on the outside-region part and not on an area of the image that the semantic mark does not indicate, and (ii) identifying the additional content for a remainder of the region without being restricted to the outside-region part; and

storing the modified image having the additional content in the region.

13. A computer program product tangibly embodied in a computer-readable storage medium, the computer program product including instructions that, when executed, generate on a display device a graphical user interface comprising:

an image display area displaying an image in a computer system, the image comprising contents that have a feature visible therein, the contents having a region thereof defined to be provided with additional content in generating a modified image; and

an input control for receiving an input comprising a semantic mark to be placed on the image, the semantic mark indicating an inside-region part inside the region and an outside-region part outside the region;

wherein the additional content for the region is determined using a patch-based optimization algorithm applied to the image, the patch-based optimization algorithm (i) identifying the additional content for the inside-region part based on the outside-region part and not on an area of the image that the semantic mark does not indicate, and (ii) identifying the additional content for a remainder of the region without being restricted to the outside-region part, and the modified image having the additional content in the region is stored.

14. The computer program product of claim 13, wherein the region comprises a hole in the image without the contents and wherein the modified image is generated in a hole-filling process.

15. The computer program product of claim 13, wherein the image is a photograph and the feature is a physical object, and wherein the region is defined corresponding to a structure of the physical object that is missing in the feature.

16. The computer program product of claim 13, wherein the input control comprises at least one of:

a line tool, an arc tool, a brush tool, an area-selection tool, and combinations thereof.

18. A system comprising:

a display device displaying an image, the image comprising contents that have a feature visible therein, the contents having a region thereof defined to be provided with additional content in generating a modified image;

an input device for receiving an input comprising a semantic mark to be placed on the image that indicates at least part of the feature, the semantic mark crossing a border of the region such that an inside-region part and an outside-region part of the semantic mark are formed; and

an image editor component determining the additional content for the region using a patch-based optimization algorithm applied to the image, the patch-based optimization algorithm identifying (i) the additional content for the inside-region part based on the outside-region part and not on an area of the image that the semantic mark does not indicate, and (ii) the additional content for a remainder of the region without being restricted to the outside-region part.

19. The system of claim 18, wherein the input device comprises at least one of:

a line tool, an arc tool, a brush tool, an area-selection tool, and combinations thereof.

20. The system of claim 18, wherein the input device associates an identifying characteristic with the semantic mark, the identifying characteristic distinguishing the semantic mark from at least one other semantic mark in the image such that the patch-based optimization algorithm processes the semantic marks separately.