Generalized panoramic mosaic

• US 20030076406A1
• Filed: 09/16/2002
• Published: 04/24/2003
• Est. Priority Date: 01/30/1997
• Status: Active Grant

• Current Assignee: Yissum Research Development Company of the Hebrew University of Jerusalem Ltd. (Hebrew University of Jerusalem)
• Original Assignee: Yissum Research Development Company of the Hebrew University of Jerusalem Ltd. (Hebrew University of Jerusalem)
• Inventors: Rousso, Benny, Peleg, Shmuel
• Granted Patent: US 7,006,124 B2
• Time in Patent Office: Days
• US Class Current: 348/36
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images

Abstract

Video mosaicing is commonly used to increase the visual field of view by pasting together many video frames. The invention provides for image mosaicing for general camera motion, including forward camera motion and zoom. After computing the motion between the images in a sequence, strips are selected from individual frames such that the strips are approximately perpendicular to the optical flow. The strips are warped such that the optical flow becomes parallel, and are pasted to a panoramic mosaic. The warping transformation on the strips, which results in making the optical flow to be parallel, can be modeled by an oblique projection of the image onto a cylindrical surface whose central axis is the trajectory of the camera. In addition, this invention uses view interpolation to generate dense intermediate views between original video frames, such that these intermediate views are used to overcome effects of motion parallax when creating panoramic mosaics.

Claims

1. A method for combining a sequence of two-dimensional images of a scene to obtain a panoramic mosaic of said scene, said sequence of two-dimensional images being acquired by a moving camera in relative motion with respect to said scene, said camera having an optical center, the camera motion giving rise to optical flow between the images, the method comprising the step of wrapping the images;

pasting the images into the panoramic image,

such that the optical flow becomes substantially parallel to the direction in which the mosaic is constructed.

4. The method according to claim 1, wherein the strips are transformed by warping into strips having edges of arbitrary shape before the strips are pasted together.

5. The method according to claim 1, wherein the strips are transformed by warping into strips having straight edges before the strips are combined together.

8. The method according to claim 1, wherein the two-dimensional images are related by an affine transformation or by a planar-projective transformation.

10. The method according to claim 1, wherein said images are projected onto a three-dimensional cylinder whose major axis approximates the path of the camera centers of said images, the combination of the strips is achieved by translating the projected two-dimensional images substantially along the cylindrical surface of the three-dimensional cylinder.

12. The method according to claim 10, wherein every two subsequent images define their own cylinder whose major axis substantially passes through the centers of the cameras of said images, and the cylinders are concatenated substantially along the image sequence.

14. The method according to claim 1, wherein a transformation is applied to the panoramic mosaic depending on a desired viewpoint.

16. The method according to claim 1, wherein the sequence of images is augmented by sets of interpolated images intermediate to the images of the sequence of images, and wherein the strips are augmented with strips defined on the interpolated images.

20. A memory containing a file representing a panoramic mosaic of a scene generated by the method of claim 1.

2. A method for combining a sequence of two-dimensional images of a scene to obtain a panoramic mosaic of said scene, said sequence of two-dimensional images being acquired by a moving camera in relative motion with respect to said scene, said camera having an optical center, the camera motion giving rise to optical flow between the images, the method comprising the steps of:

(a) selecting for each image of said sequence at least one strip such that each strip is substantially perpendicular to said optical flow;

said strips having a front edge and a back edge with the optical flow entering a strip through the front edge and exiting the strip through the back edge; and

(b) pasting together said strips from adjacent to construct a panoramic mosaic.

3. The method according to claim 2, further comprising the step of:

(a′

) wrapping the front edge of a strip defined on a two-dimensional so that it is substantially aligned with the back edge of a strip defined on an adjacent two-dimensional image.

6. The method according to claim 2, wherein the strips are transformed by warping into strips having edges of arbitrary shape before the strips are pasted together.

7. The method according to claim 2, wherein the strips are transformed by warping into strips having straight edges before the strips are combined together.

9. The method according to claim 2, wherein the two-dimensional images are related by an affine transformation or by a planar-projective transformation.

11. The method according to claim 2, wherein said images are projected onto a three-dimensional cylinder whose major axis approximates the path of the camera centers of said images, the combination of the strips is achieved by translating the projected two-dimensional images substantially along the cylindrical surface of the three-dimensional cylinder.

13. The method according to claim 11, wherein every two subsequent images define their own cylinder whose major axis substantially passes through the centers of the cameras of said images, and the cylinders are concatenated substantially along the image sequence

15. The method according to claim 2, wherein a transformation is applied to the panoramic mosaic depending on a desired viewpoint.

17. The method according to claim 2, wherein the sequence of images is augmented by sets of interpolated images intermediate to the images of the sequence of images, and wherein the strips are augmented with strips defined on the interpolated images.

21. A memory containing a file representing a panoramic mosaic of a scene generated by the method of claim 2.

18. A system for combining a sequence of two-dimensional images of a scene to obtain a panoramic mosaic of said scene, said sequence of two-dimensional images being acquired by a moving camera in relative motion with respect to said scene, said camera having an optical center, the camera motion giving rise to optical flow between the images, the system comprising:

wrapper for wrapping the images;

paster for pasting the images into the panoramic image, such that the optical flow becomes substantially parallel to the direction in which the mosaic is constructed.

19. A system for combining a sequence of two-dimensional images of a scene to obtain a panoramic mosaic of said scene, said sequence of two-dimensional images being acquired by a moving camera in relative motion with respect to said scene, said camera having an optical center, the camera motion giving rise to optical flow between the images, the system comprising:

(a) selector for selecting for each image of said sequence at least one strip such that each strip is substantially perpendicular to said optical flow;

said strips having a front edge and a back edge with the optical flow entering a strip through the front edge and exiting the strip through the back edge; and

(b) paster for pasting together said strips from adjacent images in such a way that the front edge of a strip defined on an image is substantially aligned with the back edge of a strip defined on an adjacent image.