IMAGE COMPLETION BASED ON PATCH OFFSET STATISTICS

• US 20140369622A1
• Filed: 06/05/2014
• Published: 12/18/2014
• Est. Priority Date: 06/13/2013
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: He, Kaiming, Sun, Jian
• Application Number:
  US14/297,530
• Publication Number:
  US 20140369622A1
• Time in Patent Office: Days
• US Class Current: 382/266
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal

Abstract

An image completion system receives an input image that includes an unknown region to be filled. Upon receiving the image, the image completion system examines a known region of the image other than the unknown region and matches a plurality of patches that are obtained from the known region. The image completion system determines a plurality of offsets associated with the matching and computes statistics associated with these offsets. Based on a subset of the offsets, the image completion system locates features in the known region that are used to fill the unknown region and corresponding offsets based on an energy function and an optimization algorithm. Upon locating the features, the image completion system fills the unknown region based on the located features and the corresponding offsets.

Claims

1. A method comprising:

under control of one or more processors configured with executable instructions;

matching a plurality of patches in a known region of an image to obtain a plurality of offsets, matching the plurality of patches comprising precluding each patch of the plurality of patches from matching to a nearby patch that is located within an offset threshold of each patch;

computing statistics associated with the plurality of offsets; and

completing an unknown region of the image by filling the unknown region based on the computed statistics.

2. The method as recited in claim 1, wherein the unknown region comprises a region that is blocked by an object, a region that is blurred or a region that is missing from the image.

3. The method as recited in claim 1, wherein an offset of the plurality of offsets represents a relative distance between a patch of the plurality of patches and a corresponding matched patch.

4. The method as recited in claim 1, matching the plurality of patches comprising:

for each respective patch of the plurality of patches,

determining a patch that is most similar to the respective patch based on a similarity measure; and

determining a respective offset between the respective patch and the determined patch.

5. The method as recited in claim 4, the similarity measure comprising:

a measure of a sum of squared differences between pixel values of the respective patch and pixel values of the determined patch;

or

a measure of a sum of absolute differences between the pixel values of the respective patch and the pixel values of the determined patch.

6. The method as recited in claim 1, matching the plurality of patches comprising:

determining degrees of similarity between each patch and other patches of the plurality of patches; and

selecting a patch having a highest degree of similarity from the other patches for each patch.

7. The method as recited in claim 1, computing the statistics associated with the plurality of offsets comprising counting a respective number of times that each offset of the plurality of offsets is obtained from the matching.

8. The method as recited in claim 7, computing the statistics associated with the plurality of offsets further comprising selecting offsets for which numbers of times are among first N highest ones of the counted numbers of times, wherein N is an integer greater than zero.

9. The method as recited in claim 8, completing the unknown region of the image further comprising optimizing a function having the selected offsets as parameters of the function, wherein offsets of the plurality of offsets that are not selected are excluded from consideration in optimizing the function.

10. The method as recited in claim 1, the statistics associated with the plurality of offsets being sparsely distributed.

11. The method as recited in claim 1, further comprising receiving an indication of the known region from a user, the known region comprising a pattern or feature that is expected to be present in the unknown region.

12. One or more computer-readable media storing executable instructions that, when executed by one or more processors, cause the one or more processors to perform acts comprising:

matching patches in a region other than an unknown region of an image to obtain a plurality of offsets, each patch being matched to a patch that is at an offset greater than an offset threshold;

counting a respective number of occurrences that each offset of the plurality of offsets is obtained from the matching;

selecting a plurality of dominant offsets from the plurality of offsets based on the respective number of times of each offset, a total number of the plurality of dominant offsets being less than a total number of the plurality of offsets;

determining respective offsets from the plurality of dominant offsets for pixels in the unknown region based on a function that comprises the plurality of dominant offsets as parameters to be optimized; and

completing the pixels in the unknown region using respective pixels in the other region that are separated from the pixels in the unknown region by the respective offsets.

13. The one or more computer-readable media as recited in claim 12, the function comprising a smoothness portion that penalizes incoherent seams.

14. The one or more computer-readable media as recited in claim 12, further comprising receiving an indication of the unknown region in the image from a user.

15. The one or more computer-readable media as recited in claim 12, further comprising:

prior to matching the patches in the other region, down-sampling the image to a predetermined resolution; and

in response to completing the unknown region, up-sampling the down-sampled image to an original resolution.

16. The one or more computer-readable media as recited in claim 15, further comprising:

multiplying the respective offsets by a scale corresponding to a scale change from the down-sampled image to the up-sampled image;

re-determining offsets for pixels that are within a predetermined distance from a boundary between the unknown region and the other region, the re-determining configured to correct misalignments; and

completing the unknown region in the up-sampled image based on the re-determined offsets.

17. A system comprising:

one or more processors;

memory storing executable instructions that, when executed by the one or more processors, configure the one or more processors to perform acts comprising;

receiving an indication of an unknown region of an image to be completed;

matching a plurality of patches in a known region of the image to obtain a plurality of offsets;

computing a statistic associated with the plurality of offsets;

selecting a subset of the plurality of offsets based on the computed statistic;

determining respective offsets from the subset of the plurality of offsets for pixels in the unknown region based on a function that comprises the subset of the plurality of offsets as parameters to be optimized; and

completing the pixels in the unknown region using respective pixels in the other region that are separated from the pixels in the unknown region by the respective offsets.

18. The system as recited in claim 17, the plurality of offsets being sparsely distributed.

19. The system as recited in claim 17, the statistic comprising a histogram representing a number of occurrences of each offset of the plurality of offsets that is obtained from the matching.

20. The system as recited in claim 19, selecting the subset of the plurality of offsets comprising selecting first N highest ones among the number of occurrences of each offset of the plurality of offsets, wherein N is an integer greater than zero.