Multi-view image registration with application to mosaicing and lens distortion correction
First Claim
1. Method for multi-view image registration comprising the steps of:
- a) receiving a plurality of images;
b) establishing an ideal coordinate system;
c) processing the received images to register the images to the ideal coordinate system using none of the plurality of images as a reference image, including the steps of;
c1) selecting at least two images from the plurality of images;
c2) computing a plurality of motion parameters to warp the at least two images to the ideal coordinate system as at least two warped images respectively; and
c3) minimizing an objective function that characterizes differences between the at least two warped images in the ideal coordinate system by iteratively deriving the motion parameters in a coarse-to-fine manner over a set of image pyramids to produce the further set of parameters which define the parametric transformation.
1 Assignment
0 Petitions
Accused Products
Abstract
An embodiment of the invention is a system and process for true multi-image alignment that does not rely on the measurements of a reference image being distortion free. For instance, lens distortion is a common imaging phenomenon. When lens distortion is present, none of the images can be assumed to be ideal. In an embodiment of the invention, all the images are modeled as intensity measurements represented in their respective coordinate systems, each of which is related to a reference coordinate system through an interior camera transformation and an exterior view transformation. Motion parameters determined in accordance with an embodiment of the invention dictate the position of the input frames within the reference frame. A reference coordinate system is used, but not a reference image. Motion parameters are computed to warp all input images to a virtual image mosaic in the reference coordinate system of the reference frame. Each pixel in the virtual image mosaic may be predicted by intensities at corresponding pixel positions from more than one image. The error measure, which is the sum of the variances of predicted pixel intensities at each pixel location summed over the virtual image mosaic, is minimized. The embodiment of the invention advantageously maximally uses information present in all images.
143 Citations
27 Claims
-
1. Method for multi-view image registration comprising the steps of:
-
a) receiving a plurality of images;
b) establishing an ideal coordinate system;
c) processing the received images to register the images to the ideal coordinate system using none of the plurality of images as a reference image, including the steps of;
c1) selecting at least two images from the plurality of images;
c2) computing a plurality of motion parameters to warp the at least two images to the ideal coordinate system as at least two warped images respectively; and
c3) minimizing an objective function that characterizes differences between the at least two warped images in the ideal coordinate system by iteratively deriving the motion parameters in a coarse-to-fine manner over a set of image pyramids to produce the further set of parameters which define the parametric transformation. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9)
selecting one image from the at least two images; deriving a first set of parameters which define a first parametric transformation that translates the one image to a position in the ideal coordinate system;
iteratively deriving at least one further set of parameters that define at least one respective further parametric transformation defining the warping of the one image.
-
-
4. The method of claim 1, wherein:
-
step c2 includes the steps of;
defining a set of parametric motion models each with a particular respective functional form and number of parameters;
deriving a parametric motion description using an initial selection from said set of models by estimating the corresponding parameters; and
iteratively deriving the parameters for each respective further parametric motion description using further selections from said set of models the derivation of the parameters for each further parametric motion description using the parameters derived for each previous parametric motion description to provide an initial estimate of the parameters for the further parametric motion description.
-
-
5. The method of claim 4, wherein the step of deriving the parametric motion description using the initial selection from the set of models uses a first portion of each image, said first portion having a predetermined size, and the step of iteratively deriving parameters for each respective further parametric motion description uses a further portion of each image, said further portions having a size greater than or equal to the size of the portion used in the previous iteration.
-
6. The method of claim 1, wherein:
-
the step c2) includes the step of;
deriving a parametric motion description including a number of initial parameters; and
the step c3) includes the steps of;
iteratively deriving respective further parametric motion descriptions each further parametric motion description being defined by an equal or greater number of further motion parameters than are included in any previously derived parametric motion description, using the previously derived parametric motion descriptions to provide an initial estimate of the further motion parameters; and
deriving image distortion parameters using the further motion parameters derived in a final iteration.
-
-
7. The method of claim 6, wherein the step of deriving the initial motion parameters uses a first portion of each image, said first portion having a predetermined size, and each subsequent iteration uses a further portion of each image, said further portions having a size greater than or equal to the size of the portion used in a previous iteration.
-
8. The method of claim 3, wherein:
-
the step of deriving the first set of parameters includes the step of deriving global translation parameters; and
the step of iteratively deriving at least one further set of parameters includes the steps of;
deriving a set of affine parameters using the global translation parameters; and
deriving projective parameters and global lens distortion parameters using the set of affine parameters.
-
-
9. The method of claim 8, wherein the step of deriving the global translation parameters uses approximately the inner ⅓
- rectangle of each image;
the step of deriving the set of affine parameters uses approximately the inner ⅔
rectangle of each image; and
the step of deriving the projective parameters and the global lens distortion parameters uses approximately each entire image.
- rectangle of each image;
-
10. A computer-readable medium having stored thereon a plurality of instructions, the plurality of instructions including instructions which, when executed by a processor, cause the processor to perform the steps of:
-
a) receiving, in digital form, a plurality of images;
b) establishing an ideal coordinate system;
c) processing the received images to register the images to the ideal coordinate system using none of the plurality of images as a reference image, including the steps of;
c1) selecting at least two images from the plurality of images;
c2) computing a plurality of motion parameters to warp the at least two images to the ideal coordinate system as at least two warped images respectively; and
c3) minimizing an objective function that characterizes differences between the at least two warped images in the ideal coordinate system by iteratively deriving the motion parameters in a coarse-to-fine manner over a set of image pyramids to produce the further set of parameters which define the parametric transformation. - View Dependent Claims (11, 12, 13, 14)
in the step c2), the plurality of instructions cause the processor to perform the step of deriving a parametric motion description including a number of initial parameters; and
in step c3), the plurality of instructions cause the processor to perform the step of iteratively deriving respective further parametric motion descriptions each including an equal or greater number of motion parameters than are included in any previously derived parametric motion description, using previously derived parametric motion descriptions.
-
-
12. The computer-readable medium of claim 11, wherein in step c3) the plurality of instructions cause the processor to further perform the step of
deriving image distortion parameters using the further set of parameters derived in a final iteration. -
13. The computer-readable medium of claim 11, wherein the plurality of instructions cause the processor to perform the step of deriving the initial motion parameters uses a first portion of each image, said first portion having a predetermined size, and each subsequent iteration uses a further portion of each image, each of said further portions having a size greater than or equal to the size of the portion used in the previous iteration.
-
14. The computer-readable medium of claim 10, wherein:
-
the plurality of instructions which cause the processor to perform the step c2) cause the processor to perform the steps of;
solving for global translation parameters; and
solving for a set of affine parameters using the global translation parameters; and
plurality of instructions which cause the processor to perform step c3) cause the processor to perform the step of;
solving for projective parameters and global lens distortion parameters using the set of affine parameters.
-
-
15. Apparatus for multi-view image registration comprising:
-
input means for receiving a plurality of images;
means for establishing an ideal coordinate system;
processing means for processing the received images to register the images to the ideal coordinate system using none of the plurality of images as a reference image, comprising;
means for selecting at least two images from the plurality of images;
means for computing a plurality of motion parameters to warp the at least two images to the ideal coordinate system as at least two warped images respectively; and
means for minimizing an objective function that characterizes differences between the at least two warped images in the ideal coordinate system by iteratively deriving the motion parameters in a coarse-to-fine manner over a set of image pyramids to produce the further set of parameters which define the parametric transformation. - View Dependent Claims (16, 17, 18, 19)
means for selecting one image from the at least two images;
means for deriving a first set of parameters which define a first parametric transformation that translates the one image to a position in the ideal coordinate system; and
means for deriving at least one further set of parameters which define at least one respective further parametric transformation defining a warping of the one image.
-
-
17. The apparatus of claim 15, wherein:
-
the means for computing the plurality of motion parameters includes means for deriving a parametric motion description comprising a number of initial parameters; and
the means for iteratively deriving at least one further set of parameters includes means for iteratively deriving respective further parametric motion descriptions each motion description including an equal or greater numbers of motion parameters than are included in any previously derived motion description, using the previously derived parametric motion descriptions.
-
-
18. The apparatus of claim 17, further comprising:
means for deriving image distortion parameters using the motion parameters derived in a final iteration.
-
19. The apparatus of claim 17, wherein:
-
the means for deriving the parametric motion description comprising the number of initial parameters uses a first portion of each image, said first portion having a predetermined size, and the means for iteratively deriving the respective further parametric motion descriptions uses a respective further portion of each image for each iteration, said further portions each having a size greater than or equal to the size of the portion used in the previous iteration.
-
-
20. A method for generating a two-dimensional output mosaic image comprising the steps of:
-
a) receiving a plurality of images;
b) generating a set of alignment parameters which map the pixels of the plurality of images to a single coordinate system, including the steps of;
b1) selecting at least two images from the plurality of images;
b2) computing a plurality of motion parameters to warp the at least two images to the ideal coordinate system as at least two warped images respectively; and
b3) minimizing an objective function that characterizes differences between the at least two warped images in the ideal coordinate system by iteratively deriving the motion parameters in a coarse-to-fine manner over a set of image pyramids to produce the further set of parameters which define the parametric transformation; and
c) performing two dimensional blending, responsive to the alignment parameters, to combine the plurality of images into the output mosaic image. - View Dependent Claims (21, 22, 23, 24)
performing frame-to-frame alignment; and
performing frame-to-mosaic alignment.
-
-
22. The method of claim 20 wherein the step of two dimensional blending comprises of:
-
a) selecting for each pixel position in the mosaic image, one of the plurality of images from which a corresponding pixel will be taken;
b) creating for each selected one of the plurality of warped images a working mosaic image comprising the selected warped image in its entirety, together with surrounding regions composed of other warped images as determined in step a);
c) creating Gaussian and Laplacian pyramids for each of said working mosaics;
d) copying pixels from each said working mosaic pyramid into an output mosaic pyramid as determined in step a); and
e) reconstructing the output mosaic by collapsing said output mosaic pyramid.
-
-
23. The method of claim 22 wherein the process of selecting one of the plurality of images from which the corresponding pixel value will be taken comprises steps of:
-
a) computing, based on alignment parameters relative to the output mosaic coordinate system, mapped locations corresponding to respective central portions of each image in the plurality of images; and
b) creating a Voronoi tessellation in the output mosaic coordinate system in which the said mapped locations of the centers of the respective plurality of images are sites from which distance is determined.
-
-
24. The method of claim 20, further comprising steps of estimating lens distortion and correcting each of the plurality of images to compensate for the estimated lens distortion.
-
25. Apparatus for generating a two dimensional output mosaic image comprising:
-
means for receiving a plurality of images;
means for generating a set of alignment parameters which map the pixels of the plurality of images to an ideal coordinate system to form a respective plurality of warped images including;
means for selecting at least two images from the plurality of images;
means for computing a plurality of motion parameters to warp the at least two images to the ideal coordinate system as at least two warped images respectively; and
means for minimizing an objective function that characterizes differences between the at least two warped images in the ideal coordinate system by iteratively deriving the motion parameters in a coarse-to-fine manner over a set of image pyramids to produce the further set of parameters which define the parametric transformation; and
means for performing two dimensional blending, responsive to the further set of parameters, to combine the plurality of warped images into the output mosaic image. - View Dependent Claims (26)
means for selecting, for each pixel position in the output mosaic image, one of the plurality of images from which a corresponding pixel value will be taken;
means for creating for each selected one of the plurality of warped images a working mosaic image comprising the selected warped image in its entirety, together with surrounding regions composed of other warped images;
means for creating Gaussian and Laplacian pyramids for each of said working mosaics;
means for copying pixels from each working mosaic pyramid into an output mosaic pyramid as determined by the means for selecting; and
means for reconstructing the output mosaic by collapsing said output mosaic pyramid.
-
-
27. A method for estimating lens distortion and correcting each of the plurality of images to compensate for the estimated lens distortion comprising the steps of:
-
a) receiving a plurality of images containing lens distortion;
b) selecting a parametric motion description to align said plurality of images to a single coordinate system in the absence of lens distortion;
c) estimating said parametric motion descriptions for each of said images while simultaneously estimating lens distortion parameters; and
d) warping each of said images using the estimated lens distortion parameters to correct for estimated lens distortion.
-
Specification