Image Completion Including Automatic Cropping

• US 20130243320A1
• Filed: 03/15/2012
• Published: 09/19/2013
• Est. Priority Date: 03/15/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Kang, Sing Bing, Drucker, Steven M., Kopf, Johannes Peter, Kienzle, Wolf
• Granted Patent: US 9,881,354 B2
• Time in Patent Office: Days
• US Class Current: 382/170
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal

Abstract

Described is a technology by which an image such as a stitched panorama is automatically cropped based upon predicted quality data with respect to filling missing pixels. The image may be completed, including by completing only those missing pixels that remain after cropping. Predicting quality data may be based on using restricted search spaces corresponding to the missing pixels. The crop is computed based upon the quality data, in which the crop is biased towards including original pixels and excluding predicted low quality pixels. Missing pixels are completed by using restricted search spaces to find replacement values for the missing pixels, and may use histogram matching for texture synthesis.

Claims

1. A method comprising, processing an image to perform image completion, including predicting which missing regions that are not original regions in the image are likely to be completed with low quality, computing a crop based upon the predicting, in which the crop is biased towards including original regions and predicted high quality regions and excluding predicted low quality regions pixels, and completing remaining missing pixels within the crop.

2. The method of claim 1 wherein the regions comprise pixels, and wherein the predicting includes restricting a source location for missing pixels based upon texture features.

3. The method of claim 2 further comprising using histogram matching for texture synthesis.

4. The method of claim 1 wherein the regions comprise pixels, and wherein the predicting includes computing a distance to each missing pixel from a corresponding source location.

5. The method of claim 1 wherein computing the crop comprises using an energy function that penalizes the inclusion of low quality pixels and penalizes the exclusion of original pixels.

6. The method of claim 5 further comprising, computing a per-pixel energy, and determining whether to adjust a portion of the crop based upon the combined energy of pixels corresponding to the portion of the crop.

7. The method of claim 1 further comprising, replacing at least some low quality pixels with part of a decorative frame around at least part of the crop.

8. The method of claim 1 further comprising, training a prediction mechanism that predicts the predicted quality of the missing pixels.

9. The method of claim 1 wherein predicting which missing pixels are likely to be completed with low quality comprises inputting a feature vector into a trained prediction mechanism.

10. The method of claim 1 wherein completing remaining missing pixels comprises finding a closest appearing known patch for each synthesized patch.

11. A system comprising, a prediction mechanism, the prediction mechanism configured to output predicted quality data for missing pixels associated with an image, and a cropping mechanism configured to process the quality data to generate a crop that is biased towards excluding low quality pixels.

12. The system of claim 11 wherein the prediction mechanism is configured to output the predicted quality data for a pixel by inputting a feature vector corresponding to a restricted search space that is based at least in part on a location of the pixel.

13. The system of claim 12 wherein the feature vector is based at least in part on one or more color-related histograms, edge density data, an edge orientation-related histogram, a contour-related histogram or a straight line length-related histogram, or any combination of one or more color-related histograms, edge density data, an edge orientation-related histogram, a contour-related histogram or a straight line length-related histogram.

14. The system of claim 13 wherein at least one of the histograms corresponds to entropy data, mean data or standard deviation data, or any combination of entropy data, mean data or standard deviation data.

15. The system of claim 11 further comprising a completion mechanism configured to complete missing pixels, including for each missing pixel, the completion mechanism configured to use histogram matching for texture synthesis, in which the histograms correspond to a restricted search space computed for the missing pixel.

16. The system if claim 11 wherein the cropping mechanism is further configured to bias cropping based upon including original pixels that are not missing in the image.

17. One or more computer-readable media having computer-executable instructions, which when executed perform steps, comprising, processing an image to complete missing regions, including predicting the quality of a whole region or sub-parts of a region to be filled in.

18. The one or more computer-readable media of claim 17 having further computer-executable instructions comprising computing a crop shape based upon the predicting, in which the crop shape is biased toward including original and predicted high quality regions and excluding predicted low quality regions.

19. The one or more computer-readable media of claim 17 wherein the regions comprise pixels, and wherein predicting is based upon restricting a source location for each filled-in pixel based upon texture features.

20. The one or more computer-readable media of claim 17 wherein the regions comprise pixels, and wherein predicting is based upon restricting a source location for each filled-in pixel based upon histogram matching for texture synthesis.