Method of displaying an output image of a scene from a freely selectable viewpoint

• US 7,292,238 B1
• Filed: 04/28/1998
• Issued: 11/06/2007
• Est. Priority Date: 05/02/1997
• Status: Expired due to Fees

• Current Assignee: Koninklijke Philips N.V.
• Original Assignee: Koninklijke Philips N.V.
• Inventors: Van Overveld, Cornelis W. A. M., Cuijpers, Maurice J. M.
• Primary Examiner(s): Nguyen; Phu K.
• Application Number:
  US09/067,910
• Time in Patent Office: 3,479 Days
• Field of Search: 345/433, 345/425, 345/419, 345/427, 345/606, 345/607, 382/286, 382/164
• US Class Current: 345/419
• CPC Class Codes: G06T 15/10 Geometric effects

Abstract

A number of input images of a scene is taken from different viewpoints. A respective model of the scene is made for each input image. The model contains descriptions of the location of surface patches in the space of the scene, substantially each surface patch corresponding to a set of pixels in the input image. A viewpoint is selected and views are taken from the viewpoint, each of a respective input images according to the corresponding model. For each pixel in an output image one of the views is selected and a pixel from the selected view is displayed at the pixel in the output image.

Claims

1. A method of displaying an output image of a scene from a freely selectable viewpoint, the method comprising the steps of

obtaining input images of the scene, each from a respective input viewpoint;

computing model information representing points located in a space of the scene from which input information in the input images originates;

selecting the selectable viewpoint;

determining for an output pixel in the output image which output point represented in the model information is visible in the output image at that output pixel;

determining a pixel value of that output pixel from the image information in the input image or input images which originated from the output point;

displaying the output pixel in the output image according to the pixel value,

computing a respective model for each input image, the respective model comprising information about surface patches located in the space of the scene, substantially each surface patch corresponding to a respective set of pixels in the respective input image;

determining for each respective model which, if any, respective point from the surface patches of that respective model is visible in the output image at the output pixel according to that respective model,

selecting the output point from the respective points on the basis of comparison of parameters of the surface patches containing the respective points in different models for which the respective points have been found;

determining the pixel value from the image information in the input image corresponding to the respective model from which the output point is selected.

2. A method according to claim 1, wherein for selecting the output point from the respective points, preference is given to respective points with smaller distance to the selected viewpoint.

3. A method according to claim 1, wherein for selecting the output point from the respective points, preference is given to respective points with smaller difference between a first and second angle between a normal of the surface patch to which the respective point belongs and lines of sight to the respective point, from the selected viewpoint and the respective viewpoint of the input image corresponding to the respective point respectively.

4. A method according to claim 1, for selecting the output point, preference is given to respective points with smaller difference between a first and second size of a projection of the output pixel in the output image and an input pixel in the input image corresponding to the respective point respectively, when the output pixel and the input pixel are back-projected onto the surface patch.

5. A method according to claim 1, wherein for each particular input image an ordered list of intervals in the output image is computed, the intervals corresponding to segments of respective surface patches successively visible along a scan-line in the output image, the scan-line containing the output pixel, the output point being determined by determining the interval in which the output pixel lies, and wherein the ordered list is computed by processing the line-segments successively in an order in which they are visible along a path in the particular input image, a representation being kept of a continuous range along the scan-line spanned by the intervals of processed line-segments, a foreground/background position of a line-segment being determined by comparing the line-segment to an auxiliary line segment extending between points in the model corresponding to the endpoints of the continuous range.

6. A method according to claim 1, wherein altered pixel values are computed for pixels of the input images, so as to represent the effect of lighting changes, the method comprising

selecting a source point in the scene and a lighting distribution image located relative to the source point, a respective lighting pixel value being associated with each light pixel in the lighting distribution image;

determining for each respective model which, if any, respective lighting point from the surface patches of that respective model would be visible in the lighting distribution image at a light pixel according to that respective model,

selecting an alterable point from the respective lighting points on the basis of comparison of parameters of the surface patches containing the respective lighting points in different models for which the respective lighting points have been found;

altering the image information in the input image corresponding to the respective model from which the alterable point is selected at an alterable pixel corresponding to the alterable point.

7. A method according to claim 1 comprising

obtaining further input images from a further scene, each from a respective further viewpoint in combination with a respective further models comprising information about surface patches located in a space of the further scene, substantially each surface patch corresponding to a respective set of pixels in the further input image;

designating a collection of further surface patches from the further models;

selecting position and attitude for the collection of further surface patches relative to the further scene;

determining respective auxiliary viewpoints relative to the further scene so that each of the respective auxiliary viewpoints has a same spatial relation to the further surface patches as a respective one of the viewpoints when the further surface patches are located according to the selected position and attitude;

computing a set of auxiliary images of the further surfaces in the collection from the auxiliary viewpoints and computing auxiliary models describing the further surface patches in the collection as far as they correspond to sets of pixels in the auxiliary images;

replacing a pixel value and model information in the input images by a pixel value and auxiliary model information from the auxiliary images prior to computing the output image when a depth from the respective viewpoint of the input image to a point in the scene represented by the pixel value is greater than a depth from the corresponding auxiliary viewpoint to a visible point on a further surface patch from the collection.

8. A device for displaying an output image of a scene from a freely selectable viewpoint, the device comprising

memory for storing input images of the scene, each from a respective input viewpoint;

memory for storing model information representing points in the scene from which input information in the input images originates;

means for selecting the selectable viewpoint;

means for determining for an output pixel in the output image which output point represented in the model information is visible in the output image at that output pixel;

means for determining a pixel value of that output pixel from the image information in the input image or images which originated from the output point;

a display unit receiving the pixel value for displaying the output pixel in the output image according to the pixel value,

wherein the memory for storing model information is arranged to store a respective model for each input image, the respective model comprising information about surface patches located in the space of the scene, substantially each surface patch corresponding to a respective set of pixels in the respective input image;

the device comprising

means for determining for each respective model which, if any, respective point from the surface patches of that respective model is visible in the output image at the output pixel according to that respective model,

means for selecting the output point from the respective points on the basis of comparison of parameters of the surface patches containing the respective points in different models for which the respective point has been found;

means for determining the pixel value from the image information in the input image corresponding to the respective model from which the output point is selected.

9. A device according to claim 8 wherein altered pixel values are computed for pixels of the input images, so as to represent the effect of lighting changes, the device comprising

means for selecting a source point in the scene and a lighting distribution image located relative to the source point, a respective lighting pixel value being associated with each light pixel in the lighting distribution image;

means for determining for each respective model which, if any, respective lighting point from the surface patches of that respective model would be visible in the lighting distribution image at a light pixel according to that respective model,

means for selecting an alterable point from the respective lighting points on the basis of comparison of parameters of the surface patches containing the respective lighting points in different models for which the respective lighting points have been found;

means for altering the image information according to a lighting model in the input image corresponding to the respective model from which the alterable point is selected.

10. A device according to claim 8, wherein the means for determining for each model which output point is visible determine for each particular input image an ordered list of intervals in the output image is computed, the intervals corresponding to segments of respective surface patches successively visible along a scan-line in the output image, the scan-line containing the output pixel, the output point being determined by determining the interval in which the output pixel lies, and wherein the ordered list is computed by processing the line-segments successively in an order in which they are visible along a path in the particular input image, a representation being kept of a continuous range along the scan-line spanned by the intervals of processed line-segments, the foreground/background position of a line-segment being determined by comparing the line-segment to an auxiliary line segment extending between points in the model corresponding to the endpoints of the continuous range.

11. A device according to claim 8, comprising means for

obtaining further input images from a further scene, each from a respective further viewpoint;

designating a collection of further surface patches from the further models;

selecting a position and attitude for the collection of further surface patches relative to the further scene;

determining respective auxiliary viewpoints relative to the further scene so that each of the respective auxiliary viewpoints has a same spatial relation to the further surface patches as a respective one of the viewpoints when the further surface patches are located according to the selected position and attitude;

computing a set of auxiliary images of the further surfaces in the collection from the auxiliary viewpoints and computing auxiliary model information representing depth of the further surface patches in the collection from the auxiliary viewpoints;

replacing a pixel values and model information in the input images by a pixel value and auxiliary model information from the auxiliary images prior to computing the output image when a depth from the respective viewpoint of the input image to a point in the scene represented by the pixel value is greater than a depth from the corresponding auxiliary viewpoint to a visible point on a further surface patch from the collection.