Patch-based synthesis techniques

• US 8,861,868 B2
• Filed: 08/02/2012
• Issued: 10/14/2014
• Est. Priority Date: 08/29/2011
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Shechtman, Elya, Goldman, Daniel R., Darabi, Aliakbar
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): WOLDEMARIAM, AKLILU K
• Application Number:
  US13/565,552
• Publication Number:
  US 20130051685A1
• Time in Patent Office: 803 Days
• Field of Search: 382/218, 382/243, 382/250, 382/284
• US Class Current: 382/218
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text

Abstract

Methods, apparatus, and computer-readable storage media for patch-based image synthesis. A patch matching technique provides an extended patch search space that encompasses geometric and photometric transformations, as well as gradient domain features. The geometric transformations may include translation, rotation, reflection, scale, and non-uniform scale. The photometric transformations may include gain and bias. Patch-based image synthesis techniques may leverage the extended patch search space in various image synthesis applications. The patch-based image synthesis techniques may also integrate image gradients into the patch representation and replace conventional color averaging with a technique that performs voting for colors and gradients and then solves a screened Poisson equation based on values for colors and gradients when blending patch(es) with a target image.

Claims

1. A method, comprising:

synthesizing, by one or more computing devices, a region in a target image from image data obtained from one or more source images, said synthesizing comprises;

identifying transformed source patches in the one or more source images that best match target patches in the target image, said identifying comprises;

for each target patch, applying one or more transformations from a set of supported transformations to each of a plurality of candidate source patches in the one or more source images, the set of supported transformations including one or more geometric or photometric transformations; and

comparing each transformed source patch to the respective target patch to identify a transformed source patch that best matches the respective target patch; and

blending image data, using color gradient domain blending techniques, according to each identified geometrically or photometrically transformed source patch from the one or more source images into the target image.

2. The method as recited in claim 1, wherein the one or more geometric transformations include one or more of a reflection transformation or a non-uniform scale transformation.

3. The method as recited in claim 2, wherein the one or more geometric transformations further include one or more of a uniform scale transformation, a translation transformation, or a rotation transformation.

4. The method as recited in claim 1, wherein the one or more photometric transformations include one or more of a gain transformation to adjust for brightness differences between the one or more source images and the target image or a bias transformation to adjust for contrast differences between the one or more source images and the target image.

5. The method as recited m claim 1, wherein the set of supported transformations includes one or more geometric transformations and one or more photometric transformations.

6. The method as recited in claim 1, further comprising iteratively performing said identifying and said blending.

7. The method as recited in claim 1, further comprising iteratively performing said identifying and said blending at each of a plurality of scales from a coarsest scale to a finest scale.

8. The method as recited in claim 1, wherein said identifying further comprises comparing gradient values determined at each transformed source patch to gradient values at the respective target patch in the target image.

9. The method as recited in claim 1, wherein said color gradient domain blending techniques comprises a voting technique that averages color and gradient values from the identified transformed source patches, the method further comprising applying a screened Poisson technique to the averaged color and gradient values to integrate the averaged color and gradient values into the target image.

10. The method as recited in claim 1, wherein the one or more source images include the target image, and wherein the plurality of candidate source patches include patches in the target image that are located outside the region being synthesized.

11. The method as recited in claim 1, wherein there are two or more source images, wherein the plurality of candidate source patches includes patches in each of the two or more source images, and wherein at least one transformed source patch is identified in each of at least two of the two or more source images.

12. A system, comprising:

at least one processor; and

a memory comprising program instructions, the program instructions executable by the at least one processor to synthesize a region in a target image from image data obtained from one or more source images, to synthesize the region, the program instructions executable by the at least one processor to implement operations comprising;

identifying transformed source patches in the one or more source images that best match target patches in the target image, to identify a source patch that best matches a particular target patch, the program instructions executable by the at least one processor to implement operations comprising;

applying one or more transformations from a set of supported transformations to each of a plurality of candidate source patches in the one or more source images, the set of supported transformations including one or more geometric or photometric transformations; and

comparing each transformed source patch to the particular target patch to identify a transformed source patch that best matches the respective target patch; and

blending image data, using color gradient domain blending techniques, according to each identified geometrically or photometrically transformed source patch from the one or more source images into the target image.

13. The system as recited in claim 12, wherein the one or more geometric transformations include one or more of a reflection transformation or a non-uniform scale transformation.

14. The system as recited in claim 12, wherein the one or more photometric transformations include one or more of a gain transformation to adjust for brightness differences between the one or more source images and the target image or a bias transformation to adjust for contrast differences between the one or more source images and the target image.

15. The system as recited in claim 12, wherein the program instructions are executable by the at least one processor to iteratively perform said identifying and said blending at each of a plurality of scales from a coarsest scale to a finest scale.

16. The system as recited in claim 12, wherein, to identify the source patches, the program instructions are further executable by the at least one processor to compare gradient values determined at each transformed source patch to gradient values at the respective target patch in the target image.

17. The system as recited in claim 12, wherein, the color gradient domain blending techniques comprise a voting technique that averages color and gradient values from the identified transformed source patches, and wherein the program instructions are further executable by the at least one processor to apply a screened Poisson technique to the averaged color and gradient values to integrate the color and gradient values into the target image.

18. The system as recited in claim 12, wherein there are two or more source images, wherein the plurality of candidate source patches includes patches in each of the two or more source images, and wherein at least one transformed source patch is identified in each of at least two of the two or more source images.

19. A non-transitory computer-readable storage medium storing program instructions, the program instructions computer-executable to implement operations comprising:

synthesizing a region in a target image from image data obtained from one or more source images, in said synthesizing, the program instructions computer-executable to implement operations comprising;

identifying transformed source patches in the one or more source images that best match target patches in the target image, in said identifying, the program instructions computer-executable to implement operations comprising;

for each target patch, applying one or more transformations from a set of supported transformations to each of a plurality of candidate source patches in the one or more source images, the set of supported transformations including one or more geometric or photometric transformations; and

comparing each transformed source patch to the respective target patch to identify a transformed source patch that best matches the respective target patch; and

blending image data, using color gradient domain blending techniques, according to each identified geometrically or photometrically transformed source patch from the one or more source images into the target image.

20. The non-transitory computer-readable storage medium as recited in claim 19, wherein the one or more geometric transformations include one or more of a reflection transformation or a non-uniform scale transformation.

21. The non-transitory computer-readable storage medium as recited in claim 19, wherein the one or more photometric transformations include one or more of a gain transformation to adjust for brightness differences between the one or more source images and the target image or a bias transformation to adjust for contrast differences between the one or more source images and the target image.

22. The non-transitory computer-readable storage medium as recited in claim 19, wherein the program instructions are computer-executable to implement iteratively performing said identifying and said blending at each of a plurality of scales from a coarsest scale to a finest scale.

23. The non-transitory computer-readable storage medium as recited in claim 19, wherein, in said identifying, the program instructions are further computer-executable to implement comparing gradient values determined at each transformed source patch to gradient values at the respective target patch in the target image.

24. The non-transitory computer-readable storage medium as recited in claim 19, wherein the color gradient domain blending techniques comprise a voting technique that averages color and gradient values from the identified transformed source patches, and wherein the program instructions are further computer-executable to implement applying a screened Poisson technique to the averaged color and gradient values to integrate the color and gradient values into the target image.

25. The non-transitory computer-readable storage medium as recited in claim 19, wherein there are two or more source images, wherein the plurality of candidate source patches includes patches in each of the two or more source images, and wherein at least one transformed source patch is identified in each of at least two of the two or more source images.