Generalized panoramic mosaic

• US 6,532,036 B1
• Filed: 09/15/1999
• Issued: 03/11/2003
• Est. Priority Date: 01/30/1997
• Status: Expired due to Term

• Current Assignee: Yissum Research Development Company of the Hebrew University of Jerusalem Ltd. (Hebrew University of Jerusalem)
• Original Assignee: Yissum Research Development Company of the Hebrew University of Jerusalem Ltd. (Hebrew University of Jerusalem)
• Inventors: Rousso, Benny, Peleg, Shmuel
• Primary Examiner(s): Kelley, Chris
• Assistant Examiner(s): Senfi, Behrooz
• Application Number:
  US09/355,048
• Time in Patent Office: 1,273 Days
• Field of Search: 348/36, 348/37, 348/588, 348/575, 382/276, 382/282, 382/284, 382/287, 382/288, 382/289, 382/294, 345/436, 345/437
• US Class Current: 348/36
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images

Abstract

A method for combining a sequence of two-dimensional images of a scene to construct a panoramic mosaic of the scene has a sequence of images acquired by a camera with relative motion to the scene. The relative motion gives rise to an optical flow between the images. The images are warped so that the optical flow becomes substantially parallel to a direction in which the mosaic is constructed and pasted so that the sequence of the images is continuous for the scene to construct the panoramic mosaic of the scene. For this, the images are projected onto a three-dimensional cylinder having a major axis that approximates the path of an optical center of the camera. The combination of the images is achieved by translating the projected two-dimensional images substantially along the cylindrical surface of the three-dimensional cylinder.

Claims

1. A method of combining a sequence of two-dimensional images of a scene for obtaining a panoramic mosaic of said scene, said sequence of two-dimensional images being acquired by a moving camera in relative motion with respect to said scene, said camera motion giving rise to an optical flow between the images, the method comprising the steps of:

(a) selecting for each image of said sequence of two-dimensional images at least one family of lines perpendicular to said optical flow and defining at least one strip having a front edge, through which said optical flow enters, and a back edge, through which said optical flow exits, so that each strip is substantially perpendicular to said optical flow;

(b) warping said strips to create non-rectangular strips, so that said optical flow becomes substantially parallel to a direction in which said mosaic is constructed; and

(c) pasting said non-rectangular strips so that said sequence of two-dimensional images are continuous for said scene,

whereby said panoramic mosaic of the scene is constructed.

2. The method of claim 1 wherein step (b) of warping the strips includes transforming said strips into strips having straight edges before the strips are combined together.

3. The method according to claim 1, wherein two-dimensional images are related by an affine transformation or by a planar-projective transformation.

4. The method of claim 1 wherein said images are projected onto a three-dimensional cylinder whose major axis approximates the path of camera optical center, the combination of the strips being achieved by translating the projected two-dimensional images substantially along the cylindrical surface of the three-dimensional cylinder.

5. The method of claim 4 wherein every two subsequent images define their own cylinder whose major axis substantially passes through the optical center of the camera, and the cylinders are concatenated substantially along the image sequence.

6. The method according to claim 1, wherein a transformation is applied to the panoramic mosaic depending on a desired viewpoint.

7. The method according to claim 1, wherein the sequence of images is augmented by sets of interpolated images intermediate to the images of the sequence of images, and wherein the strips are augmented with strips defined on the interpolated images.

8. A memory containing a file representing a panoramic mosaic of a scene generated by the method of claim 1.

9. A computer program product comprising a computer useable medium having computer readable program code embodied therein of combining a sequence of two-dimensional images of a scene for obtaining a panoramic mosaic of the scene, said sequence of two-dimensional images being acquired by a moving camera in relative motion with respect to the scene, said camera motion giving rise to optical flow between the images, the computer program product comprising computer readable program code for causing the computer to perform method steps of claim 1.

10. The method of claim 1 wherein step (b) of warping the strips includes transforming said strips into strips having their front and back edges fitting an edge of an adjacent strip.

11. The method of claim 1 wherein step (b) of warping the strips includes rescaling said strips to rectangular strips so that their front edge be substantially aligned with the back edge of an adjacent strip.

12. The method of claim 1 wherein at least one strip of said strips is a circular strip, the front edge of said circular strip being an arc of length l₁ having a radius r₁, the back edge of said circular strip being an arc of length l₂ having a radius r₂, and the rescaling includes multiplying l₁ by r₁/r₂.

13. The method of claim 1 wherein step (a) includes:

determining a shape of said at least one strip;

determining a size and a width of said at least one strip;

defining a position of said at least one strip on a two-dimensional image; and

cutting said at least one strip from the two-dimensional image.

14. Use of the method of claim 1 for handling at least one case from the group consisting of image parallax, camera zoom, forward camera motion, and general camera motion combining translation and rotation.

15. In a method for combining a sequence of two-dimensional images of a scene to construct a panoramic mosaic of said scene, said sequence of images being acquired by a camera for relative motion with respect to said scene, said relative motion giving rise to an optical flow between said images, the improvements comprising the steps of:

(a) warping said images so that said optical flow becomes substantially parallel to a direction in which said mosaic is constructed; and

(b) pasting said warped images so that said sequence of said images is continuous for said scene,

whereby to construct said panoramic mosaic of said scene,

wherein said images are projected onto a three-dimensional cylinder whose major axis approximates the path of camera optical center, the combination of the images being achieved by translating the projected two-dimensional images substantially along the cylindrical surface of the three-dimensional cylinder, and

wherein every two subsequent images define their own cylinder whose major axis substantially passes through the optical center of the camera, and the cylinders are concatenated substantially along the image sequence.

16. A system for combining a sequence of two-dimensional images of a scene for obtaining a panoramic mosaic of said scene, said sequence of two-dimensional images being acquired by a moving camera in relative motion with respect to said scene, said camera motion giving rise to an optical flow between the images, the system comprising:

(a) a selector for selecting each image of said sequence of two-dimensional images at least one family of lines perpendicular to said optical flow and defining at least one strip having a front edge, through which said optical flow enters, and a back edge, through which said optical flow exits, so that each strip is substantially perpendicular to said optical flow;

(b) a warper for warping said strips to create non-rectangular strips, so that said optical flow becomes substantially parallel to a direction in which said mosaic is constructed; and

(c) a paster for pasting said non-rectangular strips so that said sequence of two-dimensional images are continuous for said scene,

whereby said panoramic mosaic of the scene is constructed.