Automatic image inpainting using local patch statistics

• US 10,636,119 B1
• Filed: 04/17/2019
• Issued: 04/28/2020
• Est. Priority Date: 03/02/2017
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Duan, Kun, Gong, Yunchao, Hu, Nan
• Primary Examiner(s): Chin, Michelle
• Application Number:
  US16/387,110
• Time in Patent Office: 377 Days
• Field of Search: None
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal

Abstract

Disclosed are systems, methods, and computer-readable storage media to object replacement in an image. The method includes receiving a user input identifying an object depicted in an image. The method further includes removing the object from the image by replacing a first region of the image with one or more patch matches identified from a second region of the image. Each of the one or more patch matches comprises two identical image patches. The second region includes a portion of the image that is outside of the first region and that surrounds the first region. The replacing of the first region with the one or more patch matches identified from the second region yields a modified image without the object.

Claims

1. A system comprising:

processor; and

memory coupled to the processor and storing instructions that, when executed by the processor, cause the system to perform operations comprising;

receiving user input identifying an object depicted in an image; and

removing the object from the image by replacing a first region of the image with one or more patch matches identified from a second region of the image, each of the one or more patch matches comprising two identical image patches, the second region including a portion of the image that is outside of the first region and that surrounds the first region, the replacing of the first region with the one or more patch matches identified from the second region yielding a modified image without the object.

2. The system of claim 1, wherein the operations further comprise:

determining second region of the image based on the user input.

3. The system of claim 2, wherein the determining the second region includes dynamically computing dimensions of the second region based on the user input.

4. The system of claim 1, wherein the operations further comprise:

identifying the one or more patch matches within the second region, each patch match comprising two identical image patches, each image patch comprising one or more pixels of the image.

5. The system of claim 4, wherein the operations further comprise padding the image by a predefined padding size prior to identifying the one or more patch matches.

6. The system of claim 1, wherein the replacing of the first region of the image comprises inpainting the first region with the one or more patch matches.

7. The system of claim 1, wherein the operations further comprise applying one or more blending techniques to the modified image to blend the replaced region with a remainder of the modified image.

8. The system of claim 1, wherein the operations further comprise:

computing patch offsets for each of the one or more patch matches, each patch offset including a distance between the two identical image patches; and

determining patch offset statistics based on the computed patch offsets.

9. The system of claim 8, wherein the replacing of the first region comprises combining a stack of two or more shifted images based on the patch offset statistics.

10. The system of claim 8, wherein the determining patch offset statistics based on the computed patch offsets comprises generating a two-dimensional histogram that corresponds to a distribution of patch offsets in the second region.

11. The system of claim 10, wherein the replacing of the first region comprises:

applying a pixel-level graph cut algorithm to the first region, the applying of the pixel-level graph cut algorithm including assigning a label to each pixel in the first region that corresponds to a patch offset in the histogram.

12. The system of claim 1, wherein the operations further comprise:

enlarging the second region by a predefined padding size.

13. The system of claim 1, wherein:

the operations further comprise computing a binary mask for the first region yielding a masked region; and

the replacing of the first region includes filling the masked region.

14. The system of claim 1, wherein the operations further comprise:

identifying one or more strong edges in the modified image;

generating an edge map based on the identified one or more strong edges;

generating a binary mask for strong edge pixels by dilating the edge map;

applying blur techniques to the modified image to yield a blurred image; and

blending the modified image with the blurred image using the binary mask.

15. The system of claim 1, wherein the operations further comprise:

scaling the first region to a predetermined size, the scaling of the first region yielding a scaled region, wherein the replacing of the first region comprises inpainting the scaled region.

16. A method comprising:

receiving user input identifying an object depicted in an image; and

removing, by one or more hardware processors of a machine, the object from the image by replacing a first region of the image with one or more patch matches identified from a second region of the image, each of the one or more patch matches comprising two identical image patches, the second region including a portion of the image that is outside of the first region and that surrounds the first region, the replacing of the first region with the one or more patch matches identified from the second region yielding a modified image without the object.

17. The method of claim 16, further comprising:

dynamically computing dimensions of the second region based on the user input; and

identifying the one or more patch matches within the second region, each patch match comprising two identical image patches, each image patch comprising one or more pixels of the image.

18. The method of claim 16, further comprising:

applying one or more blending techniques to the modified image to blend the replaced region with a remainder of the modified image.

19. The method of claim 16, further comprising:

padding the image by a predefined padding size prior to identifying the one or more patch matches; and

scaling the first region to a predetermined size, the scaling of the first region yielding a scaled region, wherein the replacing of the first region comprises inpainting the scaled region.

20. A non-transitory computer-readable medium storing instructions that, when executed by a computer system, cause the computer system to perform operations comprising:

receiving user input identifying an object depicted in an image; and

removing, by one or more hardware processors of a machine, the object from the image by replacing a first region of the image with one or more patch matches identified from a second region of the image, each of the one or more patch matches comprising two identical image patches, the second region including a portion of the image that is outside of the first region and that surrounds the first region, the replacing of the first region with the one or more patch matches identified from the second region yielding a modified image without the object.