System and process for generating 3D video textures using video-based rendering techniques

• US 6,611,268 B1
• Filed: 08/22/2000
• Issued: 08/26/2003
• Est. Priority Date: 05/30/2000
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Szeliski, Richard S., Salesin, David, Schödl, Arno
• Primary Examiner(s): Padmanabhan, Mano
• Application Number:
  US09/643,635
• Time in Patent Office: 1,099 Days
• Field of Search: 345/419, 345/421, 345/422, 345/420, 345/473-475, 345/629-633, 345/640, 345/683, 348/154, 348/155, 348/699-701, 382/173, 382/236, 708/203
• US Class Current: 345/473
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/215 Motion-based segmentation

Abstract

A system and process for generating a 3D video animation of an object referred to as a 3D Video Texture is presented. The 3D Video Texture is constructed by first simultaneously videotaping an object from two or more different cameras positioned at different locations. Video from, one of the cameras is used to extract, analyze and synthesize a video sprite of the object of interest. In addition, the first, contemporaneous, frames captured by at least two of the cameras are used to estimate a 3D depth map of the scene. The background of the scene contained within the depth map is then masked out, and a clear shot of the scene background taken before filming of the object began, leaving just the object. To generate each new frame in the 3D video animation, the extracted region making up a “frame” of the video sprite is mapped onto the previously generated 3D surface. The-resulting image is rendered from a novel viewpoint, and then combined with a flat image of the background which has been warped to the correct location. In cases where it is anticipated that the subject could move frequently, the foregoing part of the procedure associated with estimating a 3D depth map of the scene and extracting the 3D surface representation of the object is performed for each subsequent set of contemporaneous frames captured by at least two of the cameras.

Claims

1. A computer-implemented process for generating a 3D video animation of an object, comprising using a computer to perform the following process actions.

(a) simultaneously capturing images of a scene containing the object from two or more different cameras positioned at different locations, wherein at least one of the cameras is a video camera which captures sequential images of the object to generate an input video clip;

(b) estimating a 3D depth map of the scene containing the object using a first, contemporaneous, image captured by at least two of the cameras;

(c) extracting the region of the 3D depth map representing the object;

(d) extracting a corresponding region of each of a selected number of frames of the input video clip which depicts the object;

(e) computing a value indicative of the similarity between the extracted region of each of the selected frames and every corresponding extracted region of the other selected frames;

(f) for each extracted region, analyzing the similarity values associated therewith to identify potentially acceptable transitions between it and corresponding extracted regions of the other selected frames;

(g) synthesizing a new video sequence from the group of corresponding extracted regions by using the identified transitions to generate an order in which the extracted regions associated with these transitions are to be played;

(h) mapping onto the extracted region of the 3D depth map, the extracted region of the new video sequence that was specified as first to be played to produce a 3D image of the object;

(i) rendering the 3D image of the object from a prescribed novel viewpoint;

(j) combining the 3D image of the object with a background image to create a frame of the 3D video animation; and

(k) repeating process actions (h) through (j) for each subsequent extracted region of the new video sequence in the order specified in the synthesizing process action, to create each subsequent frame of the 3D video animation.

2. A computer-implemented process for generating a 3D video animation of an object, comprising using a computer to perform the following process actions:

(a) simultaneously capturing images of a scene containing the object from two or more different cameras positioned at different locations, wherein at least one of the cameras is a video camera which captures sequential images of the object to generate an input video clip;

(b) estimating a 3D depth map of the scene containing the object using contemporaneous images captured by at least two of the cameras;

(c) extracting the region of the 3D depth map representing the object;

(d) extracting a corresponding region of each of a selected number of frames of the input video clip which depicts the object;

(e) computing a value indicative of the similarity between the extracted region of each of the selected frames and every corresponding extracted region of the other selected frames;

(f) for each extracted region, analyzing the similarity values associated therewith to identify potentially acceptable transitions between, it and corresponding extracted regions of the other selected frames;

(g) synthesizing a new video sequence from the group of corresponding extracted regions by using the identified transitions to generate an order in which the extracted regions associated with these transitions are to be played;

(h) mapping onto the extracted region of the 3D depth map, the extracted region of the new video sequence that was specified as first to be played to produce a 3D image of the object;

(i) rendering the 3D image of the object from a prescribed novel viewpoint;

(j) combining the 3D image of the object with a background image to create a frame of the 3D video animation; and

(k) repeating process actions (b) and (c) for each subsequent, temporally-corresponding, frames of said at least two of the video cameras, and repeating process actions (h) through (j) for each subsequent extracted region of the new video sequence in the order specified in the synthesizing process action, to create each subsequent frame of the 3D video animation.

3. A system for generating a 3D video animation of an object, comprising:

a general purpose computing device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

(a) input a video sprite of an object, said video sprite comprising a sequence of frames that when viewed in order shows the object moving in place,

(b) simultaneously capture images of a scene containing the object from two or more different cameras positioned at different locations, wherein at least one of the cameras is a video camera which captures sequential images of the object to generate an input video clip,

(c) estimate a 3D depth map of the scene containing the object using contemporaneous images captured by at least two of the cameras,

(d) extract the region of the 3D depth map representing the object,

(e) map onto the extracted region of the, 3D depth map, the first frame of the video sprite to produce a 3D image of the object,

(f) render the 3D image of the object from a prescribed novel viewpoint,

(g) combine the 3D image of the object with a background image to create a frame of the 3D video animation, and

(h) repeat the execution of the program modules responsible for performing (c) and (d) for each subsequent, temp orally-corresponding, frame of at least two of the video cameras, and repeat the execution of the program modules responsible for performing (e) through (g) for each subsequent frame of the video spite, to create each subsequent frame of the 3D video animation.

4. A computer-readable medium having computer-executable instructions for generating a 3D video animation of an object, said computer-executable instructions comprising:

(a) inputting a video sprite of an object, said video sprite comprising a sequence of frames that when viewed in order shows the object moving in place;

(b) simultaneously capturing images of a scene containing the object from two or more different cameras positioned at different locations, wherein at least one of the cameras is a video camera which captures sequential images of the object to generate an input video clip;

(c) estimating a 3D depth map of the scene containing the object using a first, contemporaneous, image captured by at least two of the cameras;

(d) extracting the region of the 3D depth map representing the object;

(e) mapping onto the extracted region of the 3D depth map, the first frame of the video sprite to produce a 3D image of the object;

(f) rendering the 3D image of the object from a prescribed novel viewpoint;

(g) combining the 3D image of the object with a background image to create a frame of the 3D video animation; and

(h) repeating instructions (e) through (g) for each subsequent frame of the video spite, to create each subsequent frame of the 3D video animation.