Panoramic video

• US 6,788,333 B1
• Filed: 07/07/2000
• Issued: 09/07/2004
• Est. Priority Date: 07/07/2000
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Uyttendaele, Matthew T., Szeliski, Richard S.
• Primary Examiner(s): Kelley, Chris
• Assistant Examiner(s): PARSONS, CHARLES E
• Application Number:
  US09/611,646
• Time in Patent Office: 1,523 Days
• Field of Search: 348/42, 348/47, 348/48, 348/36, 348/38
• US Class Current: 348/36
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A system and process for generating a panoramic video. Essentially, the panoramic video is created by first acquiring multiple videos of the scene being depicted. Preferably, these videos collectively depict a full 360 degree view of the surrounding scene and are captured using a multiple camera rig. The acquisition phase also includes a calibration procedure that provides information about the camera rig used to capture the videos that is used in the next phase for creating the panoramic video. This next phase, which is referred to as the authoring phase, involves mosaicing or stitching individual frames of the videos, which were captured at approximately the same moment in time, to form each frame of the panoramic video. A series of texture maps are then constructed for each frame of the panoramic video. Each texture map coincides with a portion of a prescribed environment model of the scene. The texture map representations of each frame of the panoramic video are encoded so as to facilitate their transfer and viewing. This can include compressing the panoramic video frames. Such a procedure is useful in applications where the panoramic video is to be transferred over a network, such as the Internet.

Claims

1. A system for generating a panoramic video of a surrounding scene, comprising:

a camera rig comprising,

multiple video cameras disposed so as to view different portions of the scene, but wherein the field of view of each camera overlaps that of each adjacent camera,

a surface onto which the cameras are mounted, wherein the optical axes of each camera are approximately coplanar and directed approximately radially outward from an axis extending normal to the mounting surface from a central point thereof, and wherein the mounting surface is rotatable a full 360 degrees in at least one direction about a center of rotation approximately coinciding with said central point of the mounting surface;

a general purpose computing device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

input multiple videos of the scene, wherein each of said videos was captured using a separate one of the cameras of the camera rig,

mosaic individual frames of the videos which were captured at approximately the same moment in time to form each frame of the panoramic video,

construct a series of texture maps for each frame of the panoramic video, each of which coincides with a portion of a prescribed environment model of the scene, and

calibrate the camera rig prior to executing the program module for inputting multiple videos of the scene, wherein the calibration module provides at least an estimate of the focal length, radial distortion, and rotation matrix associated with each camera which are employed in the execution of the mosaicing module to assist in the creation of each frame of the panoramic video, and wherein the calibration module requires a calibration video captured by one of the cameras and calibration images captured by each of the cameras in order to estimate the focal length, radial distortion, and rotation matrix associated with each camera, and wherein the calibration video and calibration images are obtained in a pre-calibration procedure comprising,

selecting one of the video cameras of the camera rig,

setting the selected video camera to record mode,

rotating the mounting surface of the camera rig 360 degrees,

deactivating the record mode on the selected video camera,

storing the video captured by the selected camera during the rotation of the mounting surface and designating it as the calibration video,

capturing a single frame with each of the video cameras while holding the camera rig stationary, and

storing each of the single frames as the calibration image associated with the video camera used to capture it.

2. A system for generating a panoramic video of a surrounding scene, comprising:

a camera rig comprising,

multiple video cameras disposed so as to view different portions of the scene, but wherein the field of view of each camera overlaps that of each adjacent camera,

a surface onto which the cameras are mounted, wherein the optical axes of each camera are approximately coplanar and directed approximately radially outward from ad axis extending normal to the mounting surface from a central point thereof, and wherein the mounting surface is rotatable a full 360 decrees in at least one direction about a center of rotation approximately coinciding with said central point of the mounting surface;

a general purpose computing device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

input multiple videos of the scene, wherein each of said videos was captured using a separate one of the cameras of the camera rig,

mosaic individual frames of the videos which were captured at approximately the same moment in time to form each frame of the panoramic video,

construct a series of texture maps for each frame of the panoramic video, each of which coincides with a portion of a prescribed environment model of the scene, and

calibrate the camera rig prior to executing the program module for inputting multiple videos of the scene, wherein the calibration module provides at least an estimate of the focal length, radial distortion, and rotation matrix associated with each camera which are employed in the execution of the mosaicing module to assist in the creation of each frame of the panoramic video, and wherein the calibration module requires a calibration video captured by one of the cameras and calibration images captured by each of the cameras in order to estimate the focal length, radial distortion, and rotation matrix associated with each camera, and wherein the calibration video and calibration images are obtained in a pre-calibration procedure comprising,

setting all the video cameras of the camera rig to record mode,

rotating the mounting surface of the camera rig 360 degrees,

deactivating the record mode on the video cameras,

storing the videos produced by each camera,

selecting the video captured by one of the cameras during the rotation of the mounting surface and designating it as the calibration video,

extracting a single frame with each of the stored videos that was captured at approximately the same time, and

storing each of the single frames as the calibration image associated with the video camera used to capture it.

3. A system for generating a panoramic video of a surrounding scene, comprising:

a camera rig comprising,

multiple video cameras disposed so as to view different portions of the scene, but wherein the field of view of each camera overlaps that of each adjacent camera,

a surface onto which the cameras are mounted, wherein the optical axes of each camera are approximately coplanar and directed approximately radially outward from an axis extending normal to the mounting surface from a central point thereof;

a general purpose computing device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

input multiple videos of the scene, wherein each of said videos was captured using a separate one of the cameras of the camera rig,

mosaic individual frames of the videos which were captured at approximately the same moment in time to form each frame of the panoramic video,

construct a series of texture maps for each frame of the panoramic video, each of which coincides with a portion of a prescribed environment model of the scene, and

calibrate the camera rig prior to executing the program module for inputting multiple videos of the scene, wherein the calibration module provides at least an estimate of the focal length, radial distortion, and rotation matrix associated with each camera which are employed in the execution of the mosaicing module to assist in the creation of each frame of the panoramic video, and wherein the calibration module requires a calibration video captured by one of the cameras and calibration images captured by each of the cameras in order to estimate the focal length, radial distortion, and rotation matrix associated with each camera wherein the calibration video and calibration images are obtained in a pre-calibration procedure, and wherein the calibration module comprises sub-modules for,

inputting the calibration video and calibration images,

extracting just enough of the frames of the calibration video needed to depict every part of the surrounding scene captured in the video with a prescribed overlap between frames,

estimating the focal length and radial distortion associated with the video camera used to capture the calibration video, and mosaicing the extracted frames of the calibration video to form a panoramic image,

mosaicing the calibration images into said panoramic image and estimating the rotation matrix associated with the camera employed to capture each calibration image,

deleting the extracted frames of the calibration video from the panoramic image, thereby leaving just the calibration images, said remaining mosaiced calibration images forming a calibration panoramic image,

saving the calibration panoramic image, estimated focal length, estimated radial distortion, and estimated rotation matrices in a calibration file, wherein the estimated focal length and radial distortion is assumed to apply to all the video cameras of the camera rig.

4. The system of claim 3, wherein the program module for calibrating the camera rig further comprises a sub-module for centering the extracted frames of the calibration video so as to exhibit a common pixel location for the center of each frame, wherein the centering sub-module is executed prior to the execution of the sub-module for estimating the focal length and radial distortion.

5. The system of claim 3, wherein the program module for calibrating the camera rig further comprises sub-modules for:

adjusting the panoramic image formed from the extracted calibration video frames and the calibration images to reduce any accumulated registration errors caused by the execution of the mosaicing sub-modules, said adjusting being accomplished by varying the focal length and rotation matrices to minimize mis-registration between the overlapping images making up the panoramic image; and

saving the revised estimates for the focal length and rotational matrices resulting in said minimum mis-registration in the calibration file in lieu of the original estimates thereof;

wherein

the adjusting sub-module is executed prior to the execution of the sub-module for deleting the extracted frames of the calibration video.

6. The system of claim 3, wherein the program module for calibrating the camera rig further comprises sub-modules for:

deghosting the calibration panoramic to reduce any double imaging or blurring associated with localized mis-registrations by computing the amount of local mis-registration and then locally warping each image making up the calibration panoramic image;

computing a correction field that characterizes the local warping performed in each of said images; and

saving the correction field as an additional element of the calibration file.