Multi-layer image-based rendering for video synthesis

• US 6,266,068 B1
• Filed: 03/13/1998
• Issued: 07/24/2001
• Est. Priority Date: 03/13/1998
• Status: Expired due to Term

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Compaq Computer Corporation (HP Inc.)
• Inventors: Kang, Sing Bing, Rehg, James M.
• Primary Examiner(s): Zimmerman, Mark
• Assistant Examiner(s): Nguyen, Kimbinh T.
• Application Number:
  US09/039,022
• Time in Patent Office: 1,229 Days
• Field of Search: 345/433, 345/429, 345/419, 345/435, 345/422, 382/284
• US Class Current: 345/629
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text

Abstract

A computerized method and related computer system synthesize video from a plurality of sources of image data. The sources include a variety of image data types such a collection of image stills, a sequence of video frames, and 3-D models of objects. Each source provides image data associated with an object. One source provides image data associated with a first object, and a second source provides image data associated with a second object. The image data of the first and second objects are combined to generate composite images of the first and second objects. From the composite images, an output image of the first and second objects as viewed from an arbitrary viewpoint is generated. Gaps of pixels with unspecified pixel values may appear in the output image. Accordingly, a pixel value for each of these “missing pixels” is obtained by using an epipolar search process to determine which one of the sources of image data should provide the pixel value for that missing pixel.

Claims

1. A computerized method for synthesizing video, comprising the steps of:

providing image data associated with a first object from a first layer, the first image layer for the first object including a plurality of views of the first object, each view of the first object augmented with data representing pixel correspondences between the views;

providing image data associated with a second object from a second layer, the second layer independent of the first layer, the second image layer for the second object including a plurality of views of the second object, each view of the second object augmented with data representing pixel correspondences between the views;

combining the image data of the first and second objects dependent on relative pixel depth provided by the pixel correspondences to generate composite images of the first and second objects; and

rendering from the composite images a geometrically correct output image of the first and second objects as viewed from an arbitrary viewpoint wherein the first and second objects move independently with respect to the arbitrary viewpoint.

2. The computerized method of claim 1, further comprising the steps of:

finding a pixel in the output image with an unspecified pixel value; and

determining which one of the layers should provide a pixel value for the unspecified pixel value.

3. The computerized method of claim 1, further comprising the step of:

modifying the image data associated with the one of the objects before combining the image data of the first and second objects.

4. The computerized method of claim 1, wherein the first and second objects are independently moving objects.

5. The computerized method of claim 4, wherein one of the moving objects is an articulated figure.

6. The computerized method of claim 1, wherein the layer associated with the first object includes a set of views of the first object, and further comprising the steps of:

selecting a subset of views from the set of views, the subset of views augmented with data representing pixel correspondences;

selecting a new viewpoint for viewing the first object; and

generating new image data associated with the first object using the selected subset of views, the image data representing a new view of the first object as viewed from the new viewpoint.

7. The computerized method of claim 1, wherein the layer is selected from the group consisting of a set of still images, a video sequence of frames, and viewpoints of a 3-D model.

8. The computerized method of claim 1, further comprising the steps of:

providing image data associated with a third object from a third layer; and

combining the image data of the third object when combining the image data of the first and second objects; and

wherein the image data of the first object includes a still image, the image data of the second object includes a frame of a video sequence, and the image data of the third object includes a 3-D model of the third object.

9. A computerized method for synthesizing video, comprising the steps of:

providing a first view of a first object from a first layer associated with the first object and a second view of a second object from a second layer associated with the second object, the first layer independent of the second layer;

generating pixel values corresponding to first view of the first object;

generating pixel values corresponding to the second view of the second object; and

combining the pixel values of the first view and the pixel values of the second view dependent on relative pixel depth provided by the pixel correspondences to generate a composite image of the first and second objects.

10. The method of claim 9, further comprising the steps of:

providing another view of the first object viewed from a different viewpoint than the first view of the first object; and

generating from the composite image of the first and second objects a geometrically correct output image of the first and second objects as viewed from the different viewpoint.

11. A computer system for synthesizing video comprising:

a plurality of independent layers, each layer providing image data associated with an object, the image data including a set of views of the object each view of the object augmented with data representing pixel correspondences between the views;

a composite image generator combining the image data associated with the objects dependent on relative pixel depth provided by the pixel correspondences to generate composite images of the objects; and

a view generator generating from the composite images a geometrically correct output image of the objects as viewed from an arbitrary viewpoint.

12. The system of claim 11, wherein the view generator finds a pixel in the output image with an unspecified pixel value and determines which one of the sources of image data should provide a pixel value for the unspecified pixel value.

13. The system of claim 11, further comprising a user interface for modifying the image data associated with the one of the objects before combining the image data of the first and second objects.

14. The system of claim 11, wherein the layer is selected from the group consisting of a set of still images, a video sequence of frames, and viewpoints of a 3-D model.