Method and system for digital plenoptic imaging
First Claim
1. A method in a computer system for generating coefficients of a plenoptic function describing light rays emanating from an object, the method comprising:
- receiving a plurality of pictures of the object, each picture having been taken at a vantage point and having a plurality of pixels, each pixel having an actual initial intensity value, a location, and a direction, the location and direction of each pixel being derived from the vantage point at which the picture is taken;
initializing each of the coefficients based on the location and direction of the pixels of the plurality of pictures such that the initialized coefficients represent the actual initial intensity values of the pixels, each coefficient having an initial confidence level indicating confidence that the initialization sets the coefficient to its final value;
for each of a plurality of levels of resolution of the coefficients,for each of a plurality of coefficients at the level of resolution,combining coefficients at the next higher level of resolution based on the confidence level of the combined coefficients to generate an averaged coefficient and confidence level for the level of resolution, the averaged coefficient representing an average of the actual initial pixel intensity values represented by the combined coefficients at the next higher level of resolution; and
for each of the plurality of levels of resolution of the coefficients,for each of a plurality of coefficients at the level of resolution,distributing the averaged coefficients and confidence levels to the coefficients and confidence levels at the next lower level of resolution based on the confidence levels at the level of resolution and the next lower level of resolution so that the representation of the actual initial pixel intensity values of the coefficients at the next lower level of resolution is adjusted.
5 Assignments
0 Petitions
Accused Products
Abstract
A computer-based method and system for digital 3-dimensional imaging of an object which allows for viewing images of the object from arbitrary vantage points. The system, referred to as the Lumigraph system, collects a complete appearance of either a synthetic or real object (or a scene), stores a representation of the appearance, and uses the representation to render images of the object from any vantage point. The appearance of an object is a collection of light rays that emanate from the object in all directions. The system stores the representation of the appearance as a set of coefficients of a 4-dimensional function, referred to as the Lumigraph function. From the Lumigraph function with these coefficients, the Lumigraph system can generate 2-dimensional images of the object from any vantage point. The Lumigraph system generates an image by evaluating the Lumigraph function to identify the intensity values of light rays that would emanate from the object to form the image. The Lumigraph system then combines these intensity values to form the image.
-
Citations
26 Claims
-
1. A method in a computer system for generating coefficients of a plenoptic function describing light rays emanating from an object, the method comprising:
-
receiving a plurality of pictures of the object, each picture having been taken at a vantage point and having a plurality of pixels, each pixel having an actual initial intensity value, a location, and a direction, the location and direction of each pixel being derived from the vantage point at which the picture is taken; initializing each of the coefficients based on the location and direction of the pixels of the plurality of pictures such that the initialized coefficients represent the actual initial intensity values of the pixels, each coefficient having an initial confidence level indicating confidence that the initialization sets the coefficient to its final value; for each of a plurality of levels of resolution of the coefficients, for each of a plurality of coefficients at the level of resolution, combining coefficients at the next higher level of resolution based on the confidence level of the combined coefficients to generate an averaged coefficient and confidence level for the level of resolution, the averaged coefficient representing an average of the actual initial pixel intensity values represented by the combined coefficients at the next higher level of resolution; and for each of the plurality of levels of resolution of the coefficients, for each of a plurality of coefficients at the level of resolution, distributing the averaged coefficients and confidence levels to the coefficients and confidence levels at the next lower level of resolution based on the confidence levels at the level of resolution and the next lower level of resolution so that the representation of the actual initial pixel intensity values of the coefficients at the next lower level of resolution is adjusted. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)
-
-
13. A method in a computer system for generating coefficients of a plenoptic function describing light rays emanating from a physical object external to the computer system, the method comprising:
-
collecting a plurality of pictures of the physical object, each picture having a vantage point; initializing each of the coefficients based on the collected plurality of pictures, each coefficient having an initial confidence level indicating the level of confidence that the initialization of the coefficient is accurate; for each of a plurality of levels of resolution of the coefficients, for each of a plurality of coefficients at the level, combining coefficients at the next higher resolution level based on the confidence levels of each coefficient to generate an averaged coefficient and confidence level for the level; and for each of the plurality of levels of resolution of the coefficients, for catch of a plurality of coefficients at the level, distributing the averaged coefficients and confidence levels to the coefficients and confidence levels at the next lower resolution level based on the confidence levels at the level and the next lower resolution level.
-
-
14. A method in a computer system for establishing intensity values for calculated light rays that represent an appearance of a physical object external to the computer system, each calculated light ray having a location and direction, the method comprising:
-
collecting a plurality of pictures of the object, each picture having been taken at a vantage point and having pixels, each pixel representing the intensity value of a collected light ray with a location and direction, the location and direction based on the vantage point at which the picture is taken; and for each calculated light ray, determining which collected light rays of which pictures contribute to the intensity value of the calculated light ray based on the location and direction of the collected light rays and on the location and direction of the calculated light ray; and setting the intensity value or the calculated light ray to a combination of the intensity values of the determined collected light rays, wherein intensity value xij,p,q for the calculated ray (si, tj, up, vq)is;
##EQU24## where is a collection of all collected light rays, where xs,t,u,v is the intensity value of collected light ray (s, t, u, v), where Bi,j,p,q (s,t,u,v) represents a basis function, and where wi,j,p,q is a confidence level represented by;
##EQU25## - View Dependent Claims (15, 16, 17, 18)
-
-
19. A method in a computer system for establishing an intensity value for calculated light rays that represent an appearance of a physical object external to the computer system, comprising:
-
providing a plurality of actual light rays representing the appearance, each actual light ray having an intensity value, location, and direction; and for each calculated light ray, determining which actual light rays contribute to the intensity value of the calculated list ray based on the location and direction of the actual light rays and on the location and direction of the calculated light ray; and setting the intensity value of the calculated light ray to a combination of the intensity values of the determined actual light rays, wherein the intensity value xi,j,p,q for the calculated ray (si tj, up, vq) is;
##EQU26## where is a collection of all collected light rays, where xs,t,u,v the intensity value of an actual light ray (s, t, u, v), where Bi,j,p,q (s,t,u,v) represents a basis function, and where wi,j,p,q is a confidence level represented by;
##EQU27##20. - View Dependent Claims (20, 21, 22)
-
-
23. A method in a computer system for determining intensity levels for the pixels of an image of an object from a collection of actual intensity levels of the image of the object at various vantage points comprising:
-
setting an initial intensity level for each pixel of the image, each initial intensity level being derived from the actual intensity levels of vantage points near the pixel; setting a confidence level for each pixel of the image, the confidence level indicating a likelihood that the initial intensity level accurately represents the actual intensity level from the vantage point of the pixel; for each of a plurality of groups of pixels of the image, calculating an average intensity level for the group wherein the initial intensity levels of the pixels in the group are weighted based on the confidence levels of the pixels in the group; and for each of the plurality of groups of pixels of the image, adjusting the initial intensity levels of the pixels in the group based on the calculated average intensity level of the pixels in the group to give a final intensity level for each of the pixel in the group; wherein the adjusting of the initial intensity levels of the pixels in the group includes adjusting the calculated intensity levels for successively smaller groups of pixel based on previously adjusted average intensity levels for larger groups of pixels.
-
-
24. A method in a computer system for determining intensity levels for the pixels of an image of an object from a collection of actual intensity levels of the image of the object at various vantage points, comprising:
-
setting an initial intensity level for each pixel of the image, each initial intensity level being derived from the actual intensity levels of vantage points near the pixel; setting a confidence level for each pixel of the image, the confidence level indicating a likelihood that the initial intensity level accurately represents the actual intensity level from the vantage point of the pixel; for each of a plurality of groups of pixels of the image, calculating an average intensity level for the group wherein the initial intensity levels of the pixels in the group are weighted based on the confidence levels of the pixels in the group; and for each of the plurality of groups of pixels of the image, adjusting the initial intensity levels of the pixels in the group based on the calculated average intensity level of the pixels in the group to give a final intensity level for each of the pixel in the group; wherein the calculating of an average intensity level for each group includes calculating average intensity levels for successively larger groups of pixels based on previously calculated average intensity levels for smaller groups of pixels and wherein the adjusting of the initial intensity levels of the pixels in the group includes adjusting the calculated intensity levels for successively smaller groups of pixel based on previously adjusted average intensity levels for larger groups of pixels.
-
-
25. A computer-readable medium containing instructions for causing a computer system to determine intensity levels for the pixels of an image of an object from a collection of actual intensity levels of the image of the object at various vantage points, comprising:
-
setting an initial intensity level for each pixel of the image, each initial intensity level being derived from the actual intensity levels of vantage points near the pixel; setting a confidence level for each pixel of the image, the confidence level indicating a likelihood that the initial intensity level accurately represents the actual intensity level from the vantage point of the pixel; for each of a plurality of groups of pixels of the image, calculating a combined intensity level for the group wherein the initial intensity levels of the pixels in the group are weighted based on the confidence levels of the pixels in the group; for each of the plurality of groups of pixels of the image, adjusting the initial intensity levels of the pixels in the group based on the calculated combined intensity level of the pixels in the group to give a final intensity level for each of the pixel in the group wherein the adjusting of the initial intensity levels of the pixels in the group includes adjusting the calculated intensity levels for successively smaller groups of pixel based on previously adjusted combined intensity levels for larger groups of pixels.
-
-
26. A computer-readable medium containing instructions for causing a computer system to determine intensity levels for the pixels of an image of an object from a collection of actual intensity levels of the image of the object at various vantage points, comprising:
-
setting an initial intensity level for each pixel of the image, each initial intensity level being derived from the actual intensity levels of vantage points near the pixel; setting a confidence level for each pixel of the image, the confidence level indicating a likelihood that the initial intensity level accurately represents the actual intensity level from the vantage point of the pixel; for each of a plurality of groups of pixels of the image, calculating a combined intensity level for the group wherein the initial intensity levels of the pixels in the group are weighted based on the confidence levels of the pixels in the group; and for each of the plurality of groups of pixels of the image, adjusting the initial intensity levels of the pixels in the group based on the calculated combined intensity level of the pixels in the group to give a final intensity level for each of the pixel in the group wherein the calculating of the combined intensity level for each group includes calculating combined intensity levels for successively larger groups of pixels based on previously calculated combined intensity levels for smaller groups of pixels and wherein the adjusting of the initial intensity levels of the pixels in the group includes adjusting the calculated intensity levels to successively smaller groups of pixel based on previously adjusted combined intensity levels for larger groups of pixels.
-
Specification