Interactive viewpoint video employing viewpoints forming an array

• US 7,286,143 B2
• Filed: 03/31/2005
• Issued: 10/23/2007
• Est. Priority Date: 06/28/2004
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Uyttendaele, Matthew, Kang, Sing Bing, Szeliski, Richard, Winder, Simon, Zitnick, Charles III
• Primary Examiner(s): Chauhan; Ulka J.
• Assistant Examiner(s): WOODS, ERIC V
• Application Number:
  US11/097,542
• Publication Number:
  US 20050286759A1
• Time in Patent Office: 936 Days
• Field of Search: 345419-429, 345/619, 345/629, 345638-639, 345/634, 382298-302
• US Class Current: 345/629
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: G06T 7/596 from three or more stereo images
• CPC Class Codes: H04N 5/2627 for providing spin image effect, 3D stop motion effect or temporal freeze effect 2D image animation in general G06T13/80

Abstract

A system and process for generating, and then rendering and displaying, an interactive viewpoint video in which a user can watch a dynamic scene while manipulating (freezing, slowing down, or reversing) time and changing the viewpoint at will. In general, the interactive viewpoint video is generated using a small number of cameras to capture multiple video streams. A multi-view 3D reconstruction and matting technique is employed to create a layered representation of the video frames that enables both efficient compression and interactive playback of the captured dynamic scene, while at the same time allowing for real-time rendering.

Claims

1. A computer-implemented process for generating an interactive viewpoint video, comprising using a computer to perform the following process actions:

inputting three or more synchronized video streams each depicting a portion of the same scene captured from different viewpoints, wherein said viewpoints form an array;

inputting calibration data defining geometric and photometric parameters associated with each video stream; and

for each group of contemporaneous frames from the synchronized video streams,

generating a 3D reconstruction of the scene,

using the reconstruction to compute a disparity map for each frame in the group of contemporaneous frames, and

for each frame in the group of contemporaneous frames,

identifying areas of significant depth discontinuities based on its disparity map,

identifying background pixel information and foreground pixel information for pixels in said areas of significant depth discontinuities,

generating a main layer comprising pixel information associated with areas in the frame that do not exhibit depth discontinuities exceeding a prescribed threshold and the background pixel information from areas having depth discontinuities above the threshold,

generating a boundary layer comprising the foreground pixel information associated with areas having depth discontinuities that exceed the threshold, to produce a layered representation for the frame under consideration, and

storing the main layer and boundary layer in a computer readable medium that has a physical form.

2. The process of claim 1, further comprising an action of using the reconstruction to compute correspondences across all camera views.

3. The process of claim 1, further comprising a process action of compressing the layered representations generated for the frames of the interactive viewpoint video to facilitate transfer and/or storage of the video.

4. The process of claim 1, further comprising a process action of generating an interactive viewpoint video file comprising the layered representations generated from frames of the inputted video streams and said calibration data.

5. The process of claim 1, wherein the process action of generating a 3D reconstruction of the scene comprises an action of employing a segmentation-based reconstruction technique.

6. The process of claim 1, wherein the process action of generating the main layer comprises an action of establishing the color and depth of each pixel in the layer, and wherein the process action of generating the boundary layer comprises an action of establishing the color, depth and opacity of each pixel in the layer.

7. The process of claim 1, wherein the process action of generating the boundary layer comprises an action of dilating the layer to encompass a prescribed number of pixels adjoining the pixels exhibiting depth discontinuities that exceed the threshold.

8. A system for generating an interactive viewpoint video, comprising:

a video capture sub-system comprising,

three or more video cameras for capturing multiple video streams each depicting a portion of the same scene captured from different viewpoints which form an array,

synchronization equipment for synchronizing the video streams to create a sequence of groups of contemporaneously captured video frames each depicting a portion of the same scene,

one or more general purpose computing devices;

a first computer program having program modules executable by at least one of said one or more general purpose computing devices, said modules comprising, a camera calibration module for computing geometric and photometric parameters associated with each video stream; and

a second computer program having program modules executable by at least one of said one or more general purpose computing devices, said modules comprising,

a 3D reconstruction module which generates a 3D reconstruction of the scene depicted in each group of contemporaneous frames from the synchronized video streams, and which uses the reconstruction to compute a disparity map for each frame in the group of contemporaneous frames,

a matting module which, for each frame in each group of contemporaneous frames, identifies areas of significant depth discontinuities based on the frame'"'"'s disparity map, and

a layered representation module which, for each frame in each group of contemporaneous frames,

identifies background pixel information and foreground pixel information for pixels in said areas of significant depth discontinuities,

generates a main layer comprising pixel information associated with areas in the frame that do not exhibit depth discontinuities exceeding a prescribed threshold and the background pixel information from pixels in areas having depth discontinuities exceeding the threshold, and

generates a boundary layer comprising the foreground pixel information associated with areas having depth discontinuities that exceed the threshold, to produce a layered representation for the frame under consideration.

9. The system of claim 8, wherein a field of view of each camera overlaps the field of view of any adjacent camera by a prescribed amount.

10. The system of claim 8, wherein the array of camera viewpoints is a rectangular array having one of (i) regularly-spaced viewpoints and (ii) irregularly-spaced viewpoints.

11. The system of claim 8, wherein the array of camera viewpoints is a triangular array having one of (i) regularly-spaced viewpoints and (ii) irregularly-spaced viewpoints.

12. The system of claim 8, wherein the video capture sub-system further comprises storage equipment for storing the video streams prior to processing.

13. The system of claim 8, wherein the second computer program further comprises a compression program module for compressing the layered representations generated for the frames of the interactive viewpoint video to facilitate transfer and/or storage of the video.

14. The system of claim 8, wherein the second computer program further comprises an interactive viewpoint video file generation program module for creating a file comprising the layered representations generated from frames of the inputted video streams and the output of the calibration module.

15. A computer-implemented process for rendering and displaying an interactive viewpoint video from data comprising layered representations of video frames generated from sequential groups of contemporaneously input video frames each depicting a portion of the same scene and exhibiting different viewpoints which form an array, and comprising calibration data comprising geometric parameters and photometric parameters associated with the capture of each video frame, said process comprising using a computer to perform the following process actions for each frame of the interactive viewpoint video to be rendered:

identifying a current user-specified viewpoint;

identifying the frame or frames from a group of contemporaneously captured frames corresponding with a current temporal portion of the video being rendered that are needed to render the scene depicted therein from the identified viewpoint;

inputting the layered representations of the identified video frame or frames; and

rendering and displaying the frame of the interactive viewpoint video from the viewpoint currently specified by the user using the inputted layered frame representations;

wherein,

background pixel information and foreground pixel information have been identified for pixels in areas of each input frame that have significant depth discontinuities, and

the layer representation of each input frame comprises,

a main layer comprising pixel information associated with areas in the frame that do not exhibit depth discontinuities exceeding a prescribed threshold and the background pixel information from areas of depth discontinuities above the threshold, and

a boundary layer comprising the foreground pixel information associated with areas having depth discontinuities that exceed the threshold.

16. The process of claim 15, wherein the video frame data is compressed, and wherein process action of inputting the layered representations of the identified video frame or frames, comprises an action of decoding the portion of the video frame data necessary to obtain the layered representations of the identified video frame or frames.

17. The process of claim 15, wherein the process action of identifying the frame or frames from a group of contemporaneously captured frames corresponding with a current temporal portion of the video being rendered that are needed to render the scene depicted therein from the identified viewpoint, comprises the actions of:

using the calibration data to determine the viewpoints associated with each of the video frames from which the layer representations were generated;

whenever the identified viewpoint coincides with a viewpoint of one of the video frames from which the layer representations were generated, identifying that frame as the only frame needed to render the scene; and

whenever the identified viewpoint falls between the viewpoints of the video frames from which the layer representations were generated, identifying a prescribed number of frames whose viewpoints neighbor the identified viewpoint as the frames needed to render the scene.

18. The process of claim 15, wherein the process action of rendering and displaying the frame of the interactive viewpoint video, comprises an action of generating an interactive viewpoint video frame from a prescribed number of input frames of the group of contemporaneously captured frames corresponding with a current temporal portion of the video being rendered whose viewpoints neighbor the identified viewpoint, using the calibration data associated therewith, whenever the identified viewpoint falls between the viewpoints associated with said three input frames.

19. The process of claim 18, wherein the process action of generating an interactive viewpoint video frame from the prescribed number of input frames of the group of contemporaneously captured frames corresponding with a current temporal portion of the video being rendered, comprises the actions of:

for each of the input frames in turn,

projecting the main layer of the input frame under consideration into a view corresponding to the current user-specified viewpoint, and

projecting the boundary layer of the input frame under consideration into the view corresponding to the current user-specified viewpoint;

blending the two resulting sets of projected layers to create a finalized frame of the interactive viewpoint video.

20. The process of claim 19, wherein the process action of blending the resulting sets of projected layers comprises an action of blending the projected layers such that the weight each is given is in direct proportion to how close the viewpoint associated with the input layer used to create the projected layer is to the current user-specified viewpoint.