Panoramic video

• US 7,583,288 B2
• Filed: 06/15/2004
• Issued: 09/01/2009
• Est. Priority Date: 07/07/2000
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Uyttendaele, Matthew, Szeliski, Richard
• Primary Examiner(s): VO, TUNG T
• Application Number:
  US10/869,020
• Publication Number:
  US 20040233274A1
• Time in Patent Office: 1,904 Days
• Field of Search: 348/42, 348/46, 348/92, 348/143, 348/722, 348/36, 348/240.16, 348/240.01, 345/419, 345/634, 382/294, 382/154, 382/152, 382/284
• US Class Current: 348/36
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A system and process for generating a panoramic video. Essentially, the panoramic video is created by first acquiring multiple videos of the scene being depicted. Preferably, these videos collectively depict a full 360 degree view of the surrounding scene and are captured using a multiple camera rig. The acquisition phase also includes a calibration procedure that provides information about the camera rig used to capture the videos that is used in the next phase for creating the panoramic video. This next phase, which is referred to as the authoring phase, involves mosaicing or stitching individual frames of the videos, which were captured at approximately the same moment in time, to form each frame of the panoramic video. A series of texture maps are then constructed for each frame of the panoramic video. Each texture map coincides with a portion of a prescribed environment model of the scene. The texture map representations of each frame of the panoramic video are encoded so as to facilitate their transfer and viewing. This can include compressing the panoramic video frames Such a procedure is useful in applications where the panoramic video is to be transferred over a network, such as the Internet.

Claims

1. A computer-implemented process for generating a panoramic video of a scene, comprising:

using a computer to perform the following process actions,

acquiring multiple videos of the scene, wherein each of said videos was captured using a separate camera, and wherein said acquiring comprises,

employing a camera rig that is capable of recording the videos of the scene, said camera rig comprising multiple video cameras disposed so as to view different portions of the scene which collectively cover a 360 degree view of the scene,

setting all the cameras of the camera rig to record mode,

deactivating the record mode on all the cameras once sufficient video has been captured by each to ultimately produce the desired length panoramic video, and

determining the relative frame offsets between all the cameras, wherein the frame offset between any pair of cameras is defined as the number of frames before or after a particular frame number of the video captured by one of the cameras and the frame number of a frame of the video captured by the other camera at approximately the same moment in time;

mosaicing individual frames of the videos which were captured at approximately the same moment in time to form each frame of the panoramic video; and

constructing a series of texture maps for each frame of the panoramic video, each of which coincides with a portion of a prescribed environment model of the scene.

2. The process of claim 1, wherein the process action of determining the relative frame offsets, comprises the process actions of:

recording a synchronizing event after all the cameras of the camera rig have been set to record mode;

identifying the frame number in each video coinciding with the beginning of the synchronization event; and

computing the relative frame offsets among all the cameras based on the frame numbers associated with the beginning of the synchronization event in each video.

3. The process of claim 1, wherein the process action of determining the relative frame offsets, comprises the process actions of:

employing cameras in the camera rig that have a synchronization feature wherein all the cameras can be set to record mode simultaneously such that corresponding frame numbers in the captured videos correspond to frames captured at the same moment in time

designating the frame offsets to be zero.

4. The process of claim 1, wherein the process action of mosaicing individual frames of the videos which were captured at approximately the same moment in time to form each frame of the panoramic video, comprises the actions of:

(a) selecting one of the videos and choosing a starting frame from the selected video;

(b) identifying the corresponding frame in each of the non-selected videos that was captured closest in time to the chosen starting frame;

(c) mosaicing the chosen starting frame and the identified corresponding frames of the videos to form the first frame of the panoramic video

(d) selecting a frame in one of the videos that was captured immediately after a frame used to form the last-produced frame of the panoramic video;

(e) selecting the corresponding frame in each of the other videos that was captured closest in time to the previously selected frame;

(f) mosaicing the selected frames to form a next frame of the panoramic video; and

(g) repeating process actions (d) through (f) until either the desired length panoramic video is generated or the last-produced frame of the panoramic video was formed using in part the last frame of one of the videos.

5. The process of claim 4, further comprising the process actions of:

employing a camera rig that is capable of recording the videos of the scene, said camera rig comprising multiple video cameras disposed so as to view different portions of the scene which collectively cover a 360 degree view of the scene;

calibrating the camera rig to obtain at least an estimate of the focal length, radial distortion, and rotation matrix associated with each camera, and wherein

the process action of mosaicing the chosen starting frame and the identified corresponding frames to form the first frame of the panoramic video comprises a process action of employing at least the estimated focal length, radial distortion, and rotation matrix associated with each camera to assist in stitching the frames together.

6. The process of claim 4, further comprising the process actions of:

following performing the action of mosaicing the chosen starting frame and the identified corresponding frames of the videos to form the first frame of the panoramic video, computing a deghosting correction field that characterizes the local warping needed in each of said video frames to reduce any double imaging or blurring associated with localized mis-registrations; and

employing the computed deghosting correction field to reduce any double imaging or blurring in the first frame of the panoramic video.

7. The process of claim 4, further comprising the process action of, following performing the action of mosaicing the chosen starting frame and the identified corresponding frames of the videos to form the first frame of the panoramic video, adjusting the exposure of the first panoramic video frame to reduce any exposure mismatch between the video frames used to form said first frame.

8. The process of claim 4, further comprising the process actions of:

employing a camera rig that is capable of recording the videos of the scene, said camera rig comprising multiple video cameras disposed so as to view different portions of the scene which collectively cover a 360 degree view of the scene;

calibrating the camera rig to obtain at least an estimate of the focal length, radial distortion, and rotation matrix associated with each camera, and wherein

the process action of mosaicing the selected frames to form the next frame of the panoramic video comprises a process action of employing at least the estimated focal length, radial distortion, and rotation matrix associated with each camera to assist in stitching the selected frames together.

9. The process of claim 4, further comprising the process actions of:

following performing the action of mosaicing the selected frames to form said next frame of the panoramic video, computing a deghosting correction field that characterizes the local warping needed in each of the selected frames to reduce any double imaging or blurring associated with localized mis-registrations; and

employing the computed deghosting correction field to reduce any double imaging or blurring in that frame of the panoramic video.

10. The process of claim 4, further comprising the process action of, following performing the action of mosaicing the selected frames to form said next frame of the panoramic video, adjusting the exposure of that panoramic video frame to reduce any exposure mismatch between the selected frames used to form the panoramic video frame.

11. The process of claim 1, further comprising the action of encoding the frames of the panoramic video so as to facilitate their transfer and viewing.

12. The process of claim 11, wherein the process action of encoding the frames of the panoramic video comprises an action of creating at least one data file comprising the series of texture maps associated with the frames of the panoramic video.

13. The process of claim 12, wherein the process action of creating at least one data file comprising the series of texture maps associated with the frames of the panoramic video, comprises an action of compressing the at least one data file.

14. The process of claim 12, wherein the process action of encoding the frames of the panoramic video further comprises an action of creating a master data file comprising pointers to said data files associated with the frames of the panoramic video and a definition of the prescribed environment model used to construct the texture maps for each frame of the panoramic video.

15. The process of claim 14, wherein the process action of creating the master data file further comprises an action of defining navigational boundaries associated with the frames of the panoramic video, wherein said navigational boundaries refer to what part of the environment model the texture maps associated with each frame of the panoramic video characterize.

16. The process of claim 14, wherein the process action of creating the master data file further comprises an action of including a pointer to a data file comprising at least one static texture map that characterizes portions of the environment model that are considered unchanging and not characterized by any of said texture maps associated with the frames of the panoramic video.

17. The process of claim 1, further comprising a process action of incorporating an audio track into the panoramic video.

18. A system for generating a panoramic video of a surrounding scene, comprising:

a camera rig comprising multiple video cameras disposed so as to view different portions of the scene, but wherein the field of view of each camera overlaps that of each adjacent camera;

a general purpose computing device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

input multiple videos of the scene, wherein each of said videos was captured using a separate one of the cameras of the camera rig,

determine the relative frame offsets between all the cameras, wherein the frame offset between any pair of cameras is defined as the number of frames before or after a particular frame number of the video captured by one of the cameras and the frame number of a frame of the video captured by the other camera at approximately the same moment in time,

mosaic individual frames of the videos which were captured at approximately the same moment in time, as determined using the relative frame offsets between the cameras, to form each frame of the panoramic video,

locally warp each mosaiced video frame of each panoramic video frame to reduce any ghosting or blurring in the frame associated with localized mis-registrations, and

construct a series of texture maps for each frame of the panoramic video, each of which coincides with a portion of a prescribed environment model of the scene.

19. The system of claim 18, wherein the camera rig further comprises a surface onto which the cameras are mounted, and wherein the optical axes of each camera are approximately coplanar and directed approximately radially outward from an axis extending normal to the mounting surface from a central point thereof.

20. The system of claim 18, wherein a sufficient number of cameras are employed in the camera rig to provide a full 360 degree coverage of the surrounding scene including the overlap in the field of view between each pair of adjacent cameras.

21. The system of claim 20, wherein as few cameras are employed in the camera rig as possible while still providing the full 360 degree coverage of the surrounding scene including the overlap in the field of view between each pair of adjacent cameras.

22. The system of claim 21, wherein each camera comprises a wide angle lens so as to increase its field of view, thereby allowing the use of fewer cameras than would be possible employing a standard, non-wide angle lens.

23. The system of claim 18, wherein the overlap in the field of view of each camera in relation to each camera adjacent thereto is at least approximately 20 percent.

24. The system of claim 18, wherein each of the cameras is set so as to record in a progressive scan mode at a frame rate of approximately 30 frames per second.

25. The system of claim 18, wherein the cameras are all adjusted to have approximately the same setting for zoom, focus, exposure, shutter speed, as well as being white balanced in approximately the same way and having any image stabilization feature deactivated.

26. The system of claim 19, further comprising a program module for calibrating the camera rig which is executed prior to executing the program module for inputting multiple videos of the scene, wherein the calibration module provides at least an estimate of the focal length, radial distortion, and rotation matrix associated with each camera, which are employed in the execution of the mosaicing module to assist in the creation of each frame of the panoramic video.

27. The system of claim 26, wherein the program module for calibrating the camera rig requires a calibration video captured by one of the cameras and calibration images captured by each of the cameras in order to estimate the focal length, radial distortion, and rotation matrix associated with each camera, and wherein the calibration video and calibration images are obtained in a pre-calibration procedure.

28. The system of claim 27, wherein the mounting surface is rotatable a full 360 degrees in at least one direction about a center of rotation approximately coinciding with said central point of the mounting surface.

29. The system of claim 26, wherein the program module for calibrating the camera rig is repeated whenever the camera rig is reconfigured or the camera settings are adjusted.

30. The system of claim 18, further comprising a program module for encoding the frames of the panoramic video by creating at least one data file comprising the series of texture maps associated with the frames of the panoramic video.

31. The system of claim 30, wherein the program module for encoding the frames of the panoramic video, comprises a sub-module for adding an audio component to each data file associated with each frame of the panoramic video to create an audio track for the panoramic video.

32. The system of claim 31, wherein the sub-module for adding an audio component to each data file comprises a sub-module for adding an audio component representing the sound occurring in the scene at the time the video frames used to construct the frame of the panoramic video associated with the data file were captured.

33. The system of claim 32, wherein the audio component representing the sound occurring in the scene is captured using just one of the cameras of the camera rig.

34. The system of claim 32, wherein the audio component representing the sound occurring in the scene is a compilation of the sound captured by more than one of the cameras of the camera rig.

35. The system of claim 30, wherein the program module for encoding the frames of the panoramic video, comprises a sub-module for adding an audio component to each data file, wherein the audio component comprises multiple audio data sets each of which represents sounds of the environment which would be heard by a person viewing a portion of the scene captured in the panoramic video.

36. The system of claim 35, wherein each audio data set is derived from a sound recording made by a separate one of the cameras of the camera rig, and wherein each audio data set represents sounds of the environment which would be heard by a person viewing a portion of the scene defined by the field of view of the camera recording the sounds.

37. The system of claim 36, wherein the sub-module for adding an audio component to each data file comprises a sub-module for assigning the audio data set associated with a particular portion of the scene to the series of texture maps associated that portion of the scene.

38. A computer-readable storage memory medium having computer-executable instructions stored thereon for generating a panoramic video of a scene, said computer-executable instructions comprising:

(a) acquiring multiple videos of the scene, wherein each of said videos was captured using a separate camera;

(b) selecting one of the videos and choosing a starting frame from the selected video;

(c) identifying the corresponding frame in each of the non-selected videos that was captured closest in time to the chosen starting frame, said identifying comprising determining the relative frame offsets between all the cameras, wherein the frame offset between any pair of cameras is defined as the number of frames before or after a particular frame number of the video captured by one of the cameras and the frame number of a frame of the video captured by the other camera at approximately the same moment in time;

(d) mosaicing the chosen starting frame and the identified corresponding frames to form the first frame of the panoramic video;

(e) locally warping each of the mosaiced video frames to reduce any ghosting or blurring in the first frame of the panoramic video associated with localized mis-registrations;

(f) constructing texture maps for the first frame of the panoramic video each of which coincides with a portion of a prescribed environment model of the scene;

(g) selecting a frame in one of the videos that was captured immediately after a frame used to form the last-produced frame of the panoramic video;

(h) selecting the corresponding frame in each of the other videos that was captured closest in time to the previously selected frame

(i) mosaicing the selected frames to form a next frame of the panoramic video;

(j) constructing texture maps from the last-formed frame of the panoramic video, each of which coincides with a portion of the prescribed environment model of the scene; and

(k) repeating instructions (g) through (j) until either the desired length panoramic video is generated or the last-produced frame of the panoramic video was formed using in part the last frame of one of the videos.

39. The computer-readable medium of claim 38, further comprising executing, following the execution of the program module for mosaicing the chosen starting frame and the identified corresponding frames of the videos to form the first frame of the panoramic video, an instruction for adjusting the exposure of the first panoramic video frame to reduce any exposure mismatch between the video frames used to form the first frame.

40. The computer-readable medium of claim 38, further comprising executing, following the execution of the program module for mosaicing the selected frames to form said next frame of the panoramic video, an instruction for locally warping each of said video frames to reduce any ghosting or blurring in that frame of the panoramic video associated with localized mis-registrations.

41. The computer-readable medium of claim 38, further comprising executing, following the execution of the program module for mosaicing the selected frames to form said next frame of the panoramic video, an instruction for adjusting the exposure of that panoramic video frame to reduce any exposure mismatch between the video frames used to form it.

42. The computer-readable medium of claim 38, further comprising an instruction for encoding the frames of the panoramic video so as to facilitate their transfer and viewing.

43. The computer-readable medium of claim 42, wherein the instruction for encoding the frames of the panoramic video comprises a sub-module for creating a master data file comprising at least one data element comprising the series of texture maps for each frame of the panoramic video, and a data element defining the prescribed environment model used to construct the texture maps for each frame of the panoramic video.

44. The computer-readable medium of claim 43, wherein the instruction sub-module for creating the master data file further comprises a sub-module for including a data element defining navigational boundaries associated with the frames of the panoramic video, wherein said navigational boundaries refer to what part of the environment model the texture maps associated with each frame of the panoramic video characterize.

45. The computer-readable medium of claim 43, wherein the instruction sub-module for creating the master data file further comprises a sub-module for including data elements comprising at least one static texture map that characterizes portions of the environment model that are considered unchanging and not characterized by any of said texture maps associated with the frames of the panoramic video.