Object-based image inpainting

• US 7,755,645 B2
• Filed: 03/29/2007
• Issued: 07/13/2010
• Est. Priority Date: 03/29/2007
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Li, Shipeng, Wu, Feng, Liu, Dong, Sun, Xiaoyan
• Primary Examiner(s): Caschera; Antonio A
• Application Number:
  US11/615,537
• Publication Number:
  US 20080238942A1
• Time in Patent Office: 1,202 Days
• Field of Search: None
• US Class Current: 345/634
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal

Abstract

Systems and methods provide object-based image inpainting. The object-based inpainting can recover part of a missing object or can synthesize visual objects that are entirely missing. In one implementation, an exemplary system generates a target visual object to be inpainted into an image, by geometrically matching key features with a model visual object that can be selected from the same or another image. Borders of the target object are refined through pixel-based contour deformation and textures for the target object are obtained either by pixel interpolation or by patch-based modeling of pre-existing texture in the image. Graph-cut segmentation rectifies image overlap as the target object is embedded in the image. Poisson editing filters visual seams, and edge filling completes visual gaps around the border of the target object.

Claims

1. A computer-executable method, comprising:

selecting by a computing device, a first visual object to be a model for synthesizing a part of an image via object-based inpainting;

generating a second visual object from the first visual object;

wherein generating the second visual object includes geometrically matching features of the second visual object to features of the first visual object, wherein geometrically matching features includes identifying and matching a main axis of symmetry in the first and second visual objects, wherein the main axis of symmetry is identified by;

i) calculating a weight center of a visual object;

ii) searching a plurality of lines that pass through the weight center that separate the visual object into two partitions; and

iii) selecting a line of the plurality of lines as the main axis of symmetry based on a length of the line of the plurality of lines and a relative position of the line of the plurality of lines; and

inpainting the second visual object into the part of the image.

2. The computer-executable method as recited in claim 1, wherein the inpainting includes rotating and scaling features of the first visual object to establish an orientation and a size of the second visual object.

3. The computer-executable method as recited in claim 1, further comprising generating inner structure elements of the second visual object, wherein inner structure elements of the first visual object are reproduced in the second visual object according to contours of the second visual object with respect to corresponding symmetry axes of the second visual object.

4. The computer-executable method as recited in claim 1, wherein generating the second visual object includes refining a border of the second visual object via contour deformation.

5. The computer-executable method as recited in claim 4, wherein refining the border includes segmenting the border of the second visual object into two segments divided by the main symmetry axis, wherein each segment with a fraction of an entire contour is generated from a counterpart segment that has a counterpart contour in the first visual object.

6. The computer-executable method as recited in claim 1, wherein generating the second visual object includes generating texture for different areas of the second visual object, wherein a texture from the image is adopted as a model for generating texture patches used for generating the texture.

7. The computer-executable method as recited in claim 6, wherein a confidence map guides the order of generating the texture patches for the different areas of the second visual object and an availability map limits a range of candidate texture patches for generating the texture.

8. The computer-executable method as recited in claim 6, further comprising comparing a similarity of textures in the image using a sum of squared difference (SSD) of pixel values between overlapped known regions of two candidate texture patches to find the texture to be adopted as the model.

9. The computer-executable method as recited in claim 8, further comprising attenuating the SSD between two pixels by an attenuation factor when either of the pixels belong to a border of an object.

10. The computer-executable method as recited in claim 6, wherein when an area of the second visual object is small enough below a threshold value, then generating the texture includes filling in the area by interpolating pixels of the area with neighboring pixels adjacent to the area.

11. The computer-executable method as recited in claim 1, further comprising performing a graph-cut segmentation to eliminate overlaps between the second visual object and a border of the part of the image being inpainted.

12. The computer-executable method as recited in claim 11, further comprising Poisson filtering to eliminate visual seams caused by the graph-cut segmentation.

13. The computer-executable method as recited in claim 1, further comprising edge filling by propagating a structure or a texture in gaps between the second visual object and a border of the part of the image being inpainted.

14. A system, comprising:

a coarse object construction engine for generating via a computing device, an object to embed in a part of an image to be inpainted via geometrical matching with features of a source model object; and

a texture generator to model texture patches to apply to the object with respect to contours and symmetry axes of the object, wherein a symmetry axis of the object is identified by;

i) calculating a weight center of an object;

ii) searching a plurality of lines that pass through the weight center that separate the object into two partitions; and

iii) selecting a line of the plurality of lines as the main axis of symmetry based on a length of the line of the plurality of lines and a relative position of the line of the plurality of lines.

15. The system as recited in claim 14, further comprising a border refinement engine to process a border of the object via pixel-based contour deformation.

16. The system as recited in claim 14, further comprising an environment refiner, to fit the object into the image using graph-cut segmentation, edge filling, and Poisson filtering.

17. The system as recited in claim 14, further comprising an inner structure generator to synthesize non-border visual structure within the contours of the object.

18. A computing device comprising:

a processor; and

a memory coupled to the processor, the memory comprising computer-program instructions executable by the processor for;

selecting a first visual object to be a model for synthesizing a part of an image via object-based inpainting;

generating a second visual object from the first visual object;

wherein generating the second visual object includes geometrically matching features of the second visual object to features of the first visual object, wherein geometrically matching features includes identifying and matching a main axis of symmetry in the first and second visual objects, wherein the main axis of symmetry is identified by;

i) calculating a weight center of a visual object;

ii) searching a plurality of lines that pass through the weight center that separate the visual object into two partitions; and

iii) selecting a line of the plurality of lines as the main axis of symmetry based on a length of the line of the plurality of lines and a relative position of the line of the plurality of lines; and

inpainting the second visual object into the part of the image.