Method for sliced inpainting

• US 10,535,123 B2
• Filed: 12/14/2017
• Issued: 01/14/2020
• Est. Priority Date: 11/27/2006
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Kokemohr, Nils
• Primary Examiner(s): Li, Ruiping
• Application Number:
  US15/842,804
• Publication Number:
  US 20180108119A1
• Time in Patent Office: 761 Days
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20192 Edge enhancement; Edge preservation
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/90 Determination of colour characteristics

Abstract

A method for replacing image data in a destination region that is divided into sub-pieces along one or more cutting paths, which start and end at two different points on the border, and finding replacement data for the sub-pieces. The cutting paths may be determined as a function of the type of image structured at the start and the end points. The cutting paths may also be determined as a function of the area of the sub-pieces and the lengths of the cutting paths. Optionally, the destination region may be determined by a spot detection algorithm. Further optionally, the spot detection algorithm may comprise calculation of a high pass filter, or detection of areas of luminosity and border-to-volume ratios. A method for moving an image element within an image is also provided.

Claims

1. A computer-implemented method comprising:

identifying a plurality of areas adjoining borders of an image, wherein the plurality of areas are created by a misalignment of the borders of the image with respect to a horizontal axis and a vertical axis;

finding a respective set of replacement image data corresponding to each area of the plurality of areas, wherein each respective set of replacement image data is obtained from a respective source region of the image or of a different image;

adapting pixels of each respective set of replacement image data to visually fit the corresponding area of the plurality of areas; and

after the adapting, producing an updated image by replacing each of the areas of the plurality of areas with the corresponding respective set of replacement image data.

2. The computer-implemented method of claim 1, further comprising modifying the image to cause the misalignment of the borders of the image with respect to the horizontal axis and the vertical axis.

3. The computer-implemented method of claim 2, wherein modifying the image includes rotating the image to cause the borders of the image to become misaligned with the horizontal axis and the vertical axis.

4. The computer-implemented method of claim 2, wherein modifying the image includes modifying pixels of the image to correct perspective distortion in the image caused by an image capture device that captured the image.

5. The computer-implemented method of claim 2, wherein modifying the image includes a modifying pixels of the image to correct distortion in the image caused by lens defects of an image capture device that captured the image.

6. The computer-implemented method of claim 1, further comprising determining margin pixels at a margin of the image, wherein the margin pixels have pixel values based on one or more pixels closest to the margin pixels, wherein each of the plurality of areas includes one or more of the margin pixels that form a border of the respective area.

7. The computer-implemented method of claim 1, wherein producing the updated image includes overwriting undefined pixels in the plurality of areas with the respective sets of replacement image data.

8. The computer-implemented method of claim 1, wherein adapting the pixels includes iteratively applying a kernel to particular pixels enclosed by border pixels determined by differences between pixel values of margins of the respective source region and the corresponding area.

9. The computer-implemented method of claim 1, wherein finding the respective set of replacement image data includes:

dividing one or more areas of the plurality of areas into a plurality of sub-pieces along one or more cutting paths, each cutting path starting and ending at two respective points on a border of the one or more areas; and

finding a respective subset of replacement image data corresponding to each respective sub-piece of the plurality of sub-pieces, wherein the respective subset of replacement image data is obtained from the respective source region of one of;

the image or the different image.

10. The computer-implemented method of claim 9, wherein adapting pixels of each respective set of replacement image data includes adapting pixels of the respective subsets of replacement image data to visually fit the corresponding sub-pieces of the plurality of sub-pieces, and wherein producing the updated image includes replacing the respective sub-pieces of the image with the corresponding subsets of replacement image data.

11. The computer-implemented method of claim 9, wherein the adapting includes interpolating pixel values along the one or more cutting paths between a start point and an end point of each of the one or more cutting paths to determine one or more pixel values along the one or more cutting paths in the one or more areas.

12. The computer-implemented method of claim 9, further comprising expanding one or more of the sub-pieces to include additional pixels and blending colors of the additional pixels with pixels in the one or more of the sub-pieces.

13. A system comprising:

a storage device; and

at least one processor operative to access the storage device and configured to perform operations comprising;

detecting a plurality of areas along borders of a rectangular image, wherein the plurality of areas are created by a misalignment of the borders of the rectangular image with respect to a horizontal axis and a vertical axis;

finding a respective set of replacement image data corresponding to each area of the plurality of areas, wherein each respective set of replacement image data is obtained from a respective source region of the rectangular image or of a different image;

adapting pixels of each respective set of replacement image data to fit a corresponding area of the plurality of areas, wherein the adapting is based upon differences between pixel values of a margin of the respective source region and a margin of the corresponding area; and

after the adapting, producing an updated image by replacing each of the areas of the plurality of areas with a corresponding respective set of replacement image data.

14. The system of claim 13 wherein the at least one processor is configured to perform further operations comprising modifying the rectangular image to cause the misalignment of the borders of the rectangular image with respect to the horizontal axis and the vertical axis.

15. The system of claim 14 wherein the operation of modifying the rectangular image includes rotating the rectangular image to causes the borders of the rectangular image to become misaligned with the horizontal axis and the vertical axis.

16. The system of claim 14 wherein the operation of modifying the rectangular image includes modifying pixels of the rectangular image to correct distortion in the rectangular image caused by an image capture device that captured the image.

17. The system of claim 13 wherein the operation of producing the updated image includes overwriting undefined pixels in the plurality of areas with the respective sets of replacement image data.

18. The system of claim 13 wherein the operation of finding the respective set of replacement image data includes:

dividing one or more areas of the plurality of areas into a plurality of sub-pieces along one or more cutting paths, the one or more cutting paths starting and ending at two respective points on a border of the one or more areas; and

finding a respective subset of replacement image data corresponding to each respective sub-piece of the plurality of sub-pieces, wherein the respective subset of replacement image data is obtained from the respective source region of one of;

the rectangular image or the different image.

19. A non-transitory computer readable medium having stored thereon program instructions that, when executed by a processor, cause the processor to perform operations including:

modifying an image to cause misalignment of borders of the image with respect to a horizontal axis and a vertical axis;

identifying a plurality of border areas adjoining the borders of the image, wherein the plurality of border areas have pixels with undefined pixel values with respect to pixel values of the image, wherein the plurality of border areas are created by the misalignment of the borders of the image;

finding a respective set of replacement image data corresponding to each border area of the plurality of border areas, wherein each respective set of replacement image data is obtained from a respective source region of the image or of a different image;

adapting pixels of each respective set of replacement image data to visually fit the corresponding border area of the plurality of border areas; and

after the adapting, producing an updated image by replacing each of the border areas of the plurality of border areas with the corresponding respective set of replacement image data.

20. The non-transitory computer readable medium of claim 19 wherein modifying the image includes at least one of:

rotating the image to cause the borders of the image to become misaligned with the horizontal axis and the vertical axis;

or

modifying pixels of the image to correct distortion in the image caused by an image capture device that captured the image.