Patch-based synthesis techniques using color and color gradient voting

• US 9,317,773 B2
• Filed: 09/05/2014
• Issued: 04/19/2016
• Est. Priority Date: 08/29/2011
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Shechtman, Elya, Goldman, Daniel R., Darabi, Aliakbar
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): WOLDEMARIAM, AKLILU K
• Application Number:
  US14/478,932
• Publication Number:
  US 20140376807A1
• Time in Patent Office: 592 Days
• Field of Search: 382/100, 382/165, 382/181, 382/218, 382/219, 382/305, 345/419, 345/422, 345/423, 345/426, 345/473, 345/629, 345/582, 345/588
• US Class Current: 1/1
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text

Abstract

Methods, apparatus, and computer-readable storage media for patch-based image synthesis using color and color gradient voting. A patch matching technique provides an extended patch search space that encompasses geometric and photometric transformations, as well as color and color gradient domain features. The photometric transformations may include gain and bias. The patch-based image synthesis techniques may also integrate image color and color gradients into the patch representation and replace conventional color averaging with a technique that performs voting for colors and color gradients and then solves a screened Poisson equation based on values for colors and color gradients when blending patch(es) with a target image.

Claims

1. A method, comprising:

synthesizing, by one or more computing devices, a region in a target image from image data obtained from one or more source images, the synthesizing comprising;

identifying a target patch in the target image to be synthesized;

identifying one or more source patches in the one or more source images as corresponding to the target patch;

applying one or more transformations to each of the source patches;

comparing each of the transformed source patches to the target patch and selecting one of the transformed source patches as best matching the target patch based on one or more criteria, the one or more criteria comprising at least color gradient information determined from the transformed source patches;

and

blending the selected transformed source patch into the target image using a voting technique that averages color and color gradient values from the transformed source patch to produce a final image.

2. A method as described in claim 1, wherein subsequent to using the voting technique, solving a screened Poisson equation based on the color and color gradient values.

3. A method as described in claim 1, wherein the one or more transformations further comprise geometric or photometric transformations.

4. A method as described in claim 3, wherein the geometric or photometric transformations are limited to predefined ranges that vary based on a type of the synthesis or prior information received.

5. A method as described in claim 1, wherein the identifying the one or more source patches comprises determining one or more correspondences between the source images and the target image.

6. A method as described in claim 1, wherein prior to identifying one or more source patches, down sampling the target image, computing intensity color gradients of the down sampled target image, and estimating colors in the target patch.

18. A method as described in claim 1, wherein the one or more transformations comprise transformations based on gain and bias parameters.

7. A system, comprising:

at least one processor; and

a memory comprising program instructions, the program instructions executable by the at least one processor to synthesize a region in a target image from image data obtained from one or more source images, the synthesizing comprising;

identifying a target patch in the target image to be synthesized;

identifying one or more source patches in the one or more source images that best match the target patch;

applying one or more transformations to each of the source patches;

comparing each of the transformed source patches to the target patch and selecting one of the transformed source patches as best matching the target patch based on one or more criteria, the one or more criteria comprising at least color gradient information;

and

blending the selected transformed source patch into the target image using a voting technique that averages color and color gradient values from the transformed source patch to produce a final image.

8. A system as described in claim 7, wherein subsequent to using the voting technique, solving a screened Poisson equation based on the color and color gradient values.

9. A system as described in claim 7, wherein the one or more transformations further comprise geometric or other photometric transformations.

10. A system as described in claim 9, wherein the geometric or photometric transformations are limited to predefined ranges that vary based on a type of the synthesis or prior information received.

11. A system as described in claim 7, wherein prior to identifying one or more source patches, down sampling the target image, computing intensity color gradients of the down sampled target image, and estimating colors in the target patch.

19. A system as described in claim 7, wherein the one or more transformations comprise transformations based on gain and bias parameters.

12. A non-transitory computer-readable storage medium storing program instructions, the program instructions computer-executable to implement operations comprising synthesizing a target region in a target image by performing operations comprising:

identifying a target patch in the target image to be synthesized;

identifying one or more source patches in the one or more source images that best match the target patch;

applying one or more transformations to each of the source patches;

comparing each of the transformed source patches to the target patch and selecting one of the transformed source patches as best matching the target patch based on one or more criteria, the one or more criteria comprising at least color gradient information; and

blending the selected transformed source patch into the target image using a voting technique that averages color and color gradient values from the transformed source patch to produce a final image.

13. A non-transitory computer-readable storage medium as described in claim 12, wherein the transformation of the source patches comprise photometric and geometric transformations.

14. A non-transitory computer-readable storage medium as described in claim 13, wherein the photometric and geometric transformations are limited to predefined ranges.

15. A non-transitory computer-readable storage medium as described in claim 12, wherein prior to the synthesis, down sampling the target image, computing intensity color gradients of the down sampled target image, and estimating colors in the target region.

16. A non-transitory computer-readable storage medium as described in claim 12, wherein the one or more source regions are determined based on one or more correspondences determined between the one or more source images and the target image.

17. A non-transitory computer-readable storage medium as described in claim 12, wherein the blending comprises solving a screened Poisson equation using the averaged color and color gradient values.

20. A non-transitory computer-readable storage medium as described in claim 12, wherein the one or more transformed source regions are transformed using at least gain and bias parameters.