Multi-layer image-based rendering for video synthesis
First Claim
1. A computerized method for synthesizing video, comprising the steps of:
- providing image data associated with a first object from a first layer, the first image layer for the first object including a plurality of views of the first object, each view of the first object augmented with data representing pixel correspondences between the views;
providing image data associated with a second object from a second layer, the second layer independent of the first layer, the second image layer for the second object including a plurality of views of the second object, each view of the second object augmented with data representing pixel correspondences between the views;
combining the image data of the first and second objects dependent on relative pixel depth provided by the pixel correspondences to generate composite images of the first and second objects; and
rendering from the composite images a geometrically correct output image of the first and second objects as viewed from an arbitrary viewpoint wherein the first and second objects move independently with respect to the arbitrary viewpoint.
4 Assignments
0 Petitions
Accused Products
Abstract
A computerized method and related computer system synthesize video from a plurality of sources of image data. The sources include a variety of image data types such a collection of image stills, a sequence of video frames, and 3-D models of objects. Each source provides image data associated with an object. One source provides image data associated with a first object, and a second source provides image data associated with a second object. The image data of the first and second objects are combined to generate composite images of the first and second objects. From the composite images, an output image of the first and second objects as viewed from an arbitrary viewpoint is generated. Gaps of pixels with unspecified pixel values may appear in the output image. Accordingly, a pixel value for each of these “missing pixels” is obtained by using an epipolar search process to determine which one of the sources of image data should provide the pixel value for that missing pixel.
-
Citations
14 Claims
-
1. A computerized method for synthesizing video, comprising the steps of:
-
providing image data associated with a first object from a first layer, the first image layer for the first object including a plurality of views of the first object, each view of the first object augmented with data representing pixel correspondences between the views;
providing image data associated with a second object from a second layer, the second layer independent of the first layer, the second image layer for the second object including a plurality of views of the second object, each view of the second object augmented with data representing pixel correspondences between the views;
combining the image data of the first and second objects dependent on relative pixel depth provided by the pixel correspondences to generate composite images of the first and second objects; and
rendering from the composite images a geometrically correct output image of the first and second objects as viewed from an arbitrary viewpoint wherein the first and second objects move independently with respect to the arbitrary viewpoint. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8)
finding a pixel in the output image with an unspecified pixel value; and
determining which one of the layers should provide a pixel value for the unspecified pixel value.
-
-
3. The computerized method of claim 1, further comprising the step of:
modifying the image data associated with the one of the objects before combining the image data of the first and second objects.
-
4. The computerized method of claim 1, wherein the first and second objects are independently moving objects.
-
5. The computerized method of claim 4, wherein one of the moving objects is an articulated figure.
-
6. The computerized method of claim 1, wherein the layer associated with the first object includes a set of views of the first object, and further comprising the steps of:
-
selecting a subset of views from the set of views, the subset of views augmented with data representing pixel correspondences;
selecting a new viewpoint for viewing the first object; and
generating new image data associated with the first object using the selected subset of views, the image data representing a new view of the first object as viewed from the new viewpoint.
-
-
7. The computerized method of claim 1, wherein the layer is selected from the group consisting of a set of still images, a video sequence of frames, and viewpoints of a 3-D model.
-
8. The computerized method of claim 1, further comprising the steps of:
-
providing image data associated with a third object from a third layer; and
combining the image data of the third object when combining the image data of the first and second objects; and
wherein the image data of the first object includes a still image, the image data of the second object includes a frame of a video sequence, and the image data of the third object includes a 3-D model of the third object.
-
-
9. A computerized method for synthesizing video, comprising the steps of:
-
providing a first view of a first object from a first layer associated with the first object and a second view of a second object from a second layer associated with the second object, the first layer independent of the second layer;
generating pixel values corresponding to first view of the first object;
generating pixel values corresponding to the second view of the second object; and
combining the pixel values of the first view and the pixel values of the second view dependent on relative pixel depth provided by the pixel correspondences to generate a composite image of the first and second objects. - View Dependent Claims (10)
providing another view of the first object viewed from a different viewpoint than the first view of the first object; and
generating from the composite image of the first and second objects a geometrically correct output image of the first and second objects as viewed from the different viewpoint.
-
-
11. A computer system for synthesizing video comprising:
-
a plurality of independent layers, each layer providing image data associated with an object, the image data including a set of views of the object each view of the object augmented with data representing pixel correspondences between the views;
a composite image generator combining the image data associated with the objects dependent on relative pixel depth provided by the pixel correspondences to generate composite images of the objects; and
a view generator generating from the composite images a geometrically correct output image of the objects as viewed from an arbitrary viewpoint. - View Dependent Claims (12, 13, 14)
-
Specification