Creation of Rectangular Images from Input Images

• US 20150131924A1
• Filed: 11/13/2013
• Published: 05/14/2015
• Est. Priority Date: 11/13/2013
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: He, Kaiming, Sun, Jian, Chang, Huiwen
• Application Number:
  US14/079,310
• Publication Number:
  US 20150131924A1
• Time in Patent Office: Days
• US Class Current: 382/284
• CPC Class Codes: G06T 2210/22 Cropping
• CPC Class Codes: G06T 3/047 Fisheye or wide-angle transformations
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2624 for obtaining an image which is composed of whole input images, e.g. splitscreen
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

Stitched images generated from combinations of multiple separate images mostly have irregular boundaries. Users generally prefer rectangular boundaries. Techniques for warping an image with irregular boundaries to give the image rectangular boundaries are disclosed herein. Preliminary warping of the image into the rectangle provides a rectangular shape on which to overlay a mesh. The image is reverted to its original shape with irregular boundaries and the mesh is warped accordingly. Global optimization is applied to the image by finding an energy minimum, or reduced energy below a threshold, for a function that gives the image a rectangular shape while preserving shapes and preserving straight lines. The mesh is warped according to the solution of the function and the image is stretched and/or compressed along with the mesh. This approach generates results that are qualitatively more visually attractive than other contemporary techniques.

Claims

1. A method comprising:

a) receiving a plurality of input images;

b) stitching the plurality of input images together to create a stitched image that has irregular boundaries;

c) expanding the stitched image to occupy a target rectangle to create a stretched rectangular image;

d) overlaying a mesh on the stretched rectangular image;

e) warping the mesh back to a shape of the stitched image to create a warped mesh;

f) overlaying the warped mesh on the stitched image;

g) creating, by a hardware processor, an output mesh by changing positions of vertexes of the warped mesh in a way that reduced the value of an energy function which includes a shape preservation function, a straight-line preservation function, and a boundary constraint function; and

h) warping the stitched image according to the output mesh.

2. The method of claim 1, wherein input images comprise at least three images of a panoramic scene.

3. The method of claim 1, wherein the stitching comprises at least one of a graph cuts algorithm or gradient-domain composting.

4. The method of claim 1, wherein the target rectangle comprises a bounding box that frames the stitched image.

5. The method of claim 1, wherein expanding the stitched image comprises:

selecting a connected sequence of missing pixels between an edge of the stitched image and the target rectangle;

adding a sequence of pixels within the stitched image that shifts a portion of the stitched image toward the inside edge of the target rectangle to fill all or some of the sequence of missing pixels; and

repeating the selecting and the adding until there are no missing pixels within the target rectangle.

6. The method of claim 1, wherein the mesh is a square mesh or a triangular mesh.

7. The method of claim 1, wherein the warping the mesh comprises repositioning grid vertexes of the mesh using a displacement field representing displacement that is created in the stitched image by expansion of the stitched image to occupy the target rectangle.

8. The method of claim 1, wherein the shape preservation function comprises a similarity transformation that reduces distortion in relatively more important regions of the stitched image while increasing distortion in relatively less important regions of the stitched image, importance of regions of the stitched image based at least in part on an importance map.

9. The method of claim 1, wherein the straight-line preserving function comprises:

detecting straight lines;

cutting the straight lines where the straight lines cross a line of the mesh to create a plurality of line segments;

calculating orientation vectors for the plurality of line segments;

dividing possible orientations into a number of bins;

grouping the line segments into the bins according to the respective orientation vectors;

calculating an average rotation angle of the line segments in the respective bins; and

rotating the line segments in a same bin by the average rotation angle.

10. The method of claim 1, wherein the boundary constraint function stretches vertexes on the outer boundary of the warped mesh to the target rectangle.

11. The method of claim 1, wherein the energy function comprises a weighted sum of the shape preservation function, the straight line preservation function, and the boundary constraint function.

12. The method of claim 1, wherein reducing the output of the energy function comprises an alternating algorithm that first fixes a target rotation angle of the straight line preserving function and solves for the output mesh then fixes the output mesh and solves for the target rotation angle.

13. The method of claim 1, further comprising:

following the warping the stitched image, calculating a mean vertical scaling factor and a mean horizontal scaling factor from changes in vertical and horizontal dimensions of cells of the output mesh compared to cells in the warped mesh;

scaling the vertical dimension of the target rectangle by a reciprocal of the mean vertical scaling factor and scaling the horizontal dimension of the target rectangle by a reciprocal of the mean horizontal scaling factor to create an updated target rectangle; and

repeating acts d-h using the updated target rectangle.

14. The method of claim 1, further comprising:

downsampling the stitched image from an original size to smaller size;

upsampling the output mesh from the smaller size to the original size to create an original size output mesh; and

warping the original size stitched image according to the original size output mesh.

15. The method of claim 1, further comprising:

introducing a transparent region at a concave boundary of the stitched image;

performing c-h with the transparent region treated the same as other pixels in the stitched image; and

filling the transparent region using image completion.

16. Computer storage media storing information that, when accessed by a computing device, instructs the computing devices to perform the acts of:

local warping an input image by seam carving to create a first rectangular image;

placing a mesh on the first rectangular image;

warping the mesh to a shape of the input image to create a warped mesh;

placing the warped mesh on the input image;

global warping of the input image by transforming the warped mesh to an output mesh to create a second rectangular image.

17. The media of claim 16, wherein the global warping comprises warping that preserves the appearance of shapes in the input image and preserves straight lines in the input image.

18. A system comprising;

one or more processing units;

a local warping module, in communication with the one or more processing units, configured to warp an irregular-shaped input image into a first rectangular image;

a mesh alignment module, in communication with the one or more processing units, configured to;

overlay a regular mesh on the rectangular image;

warp the mesh to a shape of the irregular-shaped input image making a warped mesh; and

overlay the warped mesh on the irregular-shaped input image;

a global warping module, in communication with the one or more processing units, configured to warp the irregular-shaped input image into a second rectangular image by transforming the warped mesh into an output mesh such that an output of an energy function describing locations of vertexes in the output mesh is reduced.

19. The system of claim 18, wherein the energy function preserves the appearance of shapes and the appearance of straight lines in the irregular-shaped input image.

20. The system of claim 18, further comprising an image capture device, coupled to the one or more processing units, configured to capture a plurality of images that are combined to form the irregular-shaped input image.