Method and system for digital plenoptic imaging

• US 6,009,188 A
• Filed: 03/20/1996
• Issued: 12/28/1999
• Est. Priority Date: 02/16/1996
• Status: Expired due to Term

• Current Assignee: Uber Technologies, Inc.
• Original Assignee: Microsoft Corporation
• Inventors: Grzeszczuk, Radek, Gortler, Steven J., Szeliski, Richard S., Cohen, Michael F.
• Primary Examiner(s): Boudreau, Leo H.
• Assistant Examiner(s): Werner, Brian P.
• Application Number:
  US08/620,830
• Time in Patent Office: 1,378 Days
• Field of Search: 382/154, 382/173, 382/285, 382/294, 382/299, 382/300, 382/148, 382/240, 382/274, 382/275, 382/254, 345/419, 345/430, 345/425, 345/432, 345/952, 345/953, 356/376, 348/39, 348/43, 348/50, 348/121, 348/135, 348/142, 359/23, 359/463, 250/559.22, 250/559.23
• US Class Current: 382/154
• CPC Class Codes: G06T 15/06 Ray-tracing
• CPC Class Codes: G06T 15/50 Lighting effects

Abstract

A computer-based method and system for digital 3-dimensional imaging of an object which allows for viewing images of the object from arbitrary vantage points. The system, referred to as the Lumigraph system, collects a complete appearance of either a synthetic or real object (or a scene), stores a representation of the appearance, and uses the representation to render images of the object from any vantage point. The appearance of an object is a collection of light rays that emanate from the object in all directions. The system stores the representation of the appearance as a set of coefficients of a 4-dimensional function, referred to as the Lumigraph function. From the Lumigraph function with these coefficients, the Lumigraph system can generate 2-dimensional images of the object from any vantage point. The Lumigraph system generates an image by evaluating the Lumigraph function to identify the intensity values of light rays that would emanate from the object to form the image. The Lumigraph system then combines these intensity values to form the image.

Claims

1. A method in a computer system for generating coefficients of a plenoptic function describing light rays emanating from an object, the method comprising:

receiving a plurality of pictures of the object, each picture having been taken at a vantage point and having a plurality of pixels, each pixel having an actual initial intensity value, a location, and a direction, the location and direction of each pixel being derived from the vantage point at which the picture is taken;

initializing each of the coefficients based on the location and direction of the pixels of the plurality of pictures such that the initialized coefficients represent the actual initial intensity values of the pixels, each coefficient having an initial confidence level indicating confidence that the initialization sets the coefficient to its final value;

for each of a plurality of levels of resolution of the coefficients,

for each of a plurality of coefficients at the level of resolution,

combining coefficients at the next higher level of resolution based on the confidence level of the combined coefficients to generate an averaged coefficient and confidence level for the level of resolution, the averaged coefficient representing an average of the actual initial pixel intensity values represented by the combined coefficients at the next higher level of resolution; and

for each of the plurality of levels of resolution of the coefficients,

for each of a plurality of coefficients at the level of resolution,

distributing the averaged coefficients and confidence levels to the coefficients and confidence levels at the next lower level of resolution based on the confidence levels at the level of resolution and the next lower level of resolution so that the representation of the actual initial pixel intensity values of the coefficients at the next lower level of resolution is adjusted.

2. The method of claim 1 wherein the plenoptic function is a 4-dimensional plenoptic function such that the four dimensions identify a light ray with a certain direction that emanates from the object and passes through a point on a surface of a convex volume surrounding the object.

3. The method of claim 1 wherein the plenoptic function is a 5-dimensional plenoptic function such that the five dimensions identify a light ray with a certain direction that emanates from the object and passes through a point outside a convex volume surrounding the object.

4. The method of claim 1 wherein the pictures are taken with a camera at a plurality of arbitrary vantage points.

5. The method of claim 4 including determining the vantage point of the picture from the picture itself.

6. The method of claim 4 including the step of determining the vantage point from information external to the picture.

7. The method of claim 1 wherein the pictures are taken with a camera at a plurality of pre-defined vantage points.

8. The method of claim 1 wherein before taking the pictures, the object is placed on a stage with background markers for use in identifying the vantage points at which a picture is taken.

9. The method of claim 1 wherein each coefficient corresponds to an intensity value associated with an area on a surface of a convex volume surrounding the object and with a direction such that the intensity value represents an average intensity of light rays that emanate from the object and pass through the area in the direction.

10. The method of claim 1 including the step of rendering an image of the object by evaluating the plenoptic function for each pixel of the image.

11. The method of claim 1 wherein the step of combining combines coefficients according to a constant basis function.

12. The method of claim 1 wherein the step of combining combines coefficient according to a quadralinear basis function.

13. A method in a computer system for generating coefficients of a plenoptic function describing light rays emanating from a physical object external to the computer system, the method comprising:

collecting a plurality of pictures of the physical object, each picture having a vantage point;

initializing each of the coefficients based on the collected plurality of pictures, each coefficient having an initial confidence level indicating the level of confidence that the initialization of the coefficient is accurate;

for each of a plurality of levels of resolution of the coefficients,

for each of a plurality of coefficients at the level,

combining coefficients at the next higher resolution level based on the confidence levels of each coefficient to generate an averaged coefficient and confidence level for the level; and

for each of the plurality of levels of resolution of the coefficients,

for catch of a plurality of coefficients at the level,

distributing the averaged coefficients and confidence levels to the coefficients and confidence levels at the next lower resolution level based on the confidence levels at the level and the next lower resolution level.

14. A method in a computer system for establishing intensity values for calculated light rays that represent an appearance of a physical object external to the computer system, each calculated light ray having a location and direction, the method comprising:

collecting a plurality of pictures of the object, each picture having been taken at a vantage point and having pixels, each pixel representing the intensity value of a collected light ray with a location and direction, the location and direction based on the vantage point at which the picture is taken; and

for each calculated light ray,

determining which collected light rays of which pictures contribute to the intensity value of the calculated light ray based on the location and direction of the collected light rays and on the location and direction of the calculated light ray; and

setting the intensity value or the calculated light ray to a combination of the intensity values of the determined collected light rays,

wherein intensity value x_ij,p,q for the calculated ray (s_i, t_j, u_p, v_q)is;

##EQU24## where is a collection of all collected light rays, where x_s,t,u,v is the intensity value of collected light ray (s, t, u, v), where B_i,j,p,q (s,t,u,v) represents a basis function, and where w_i,j,p,q is a confidence level represented by;

##EQU25##

15. The method of claim 14 wherein the basis function is a constant basis function.

16. The method of claim 14 wherein the basis function is:

17. The method of claim 14 wherein the basis function is a quadralinear basis function.

18. The method of claim 14 wherein the basis function is:

where h_st is the distance between neighboring locations of calculated light rays and h_uv is the distance between neighboring directions of calculated light rays.

19. A method in a computer system for establishing an intensity value for calculated light rays that represent an appearance of a physical object external to the computer system, comprising:

providing a plurality of actual light rays representing the appearance, each actual light ray having an intensity value, location, and direction; and

for each calculated light ray,

determining which actual light rays contribute to the intensity value of the calculated list ray based on the location and direction of the actual light rays and on the location and direction of the calculated light ray; and

setting the intensity value of the calculated light ray to a combination of the intensity values of the determined actual light rays,

wherein the intensity value x_i,j,p,q for the calculated ray (s_i t_j, u_p, v_q) is;

##EQU26## where is a collection of all collected light rays, where x_s,t,u,v the intensity value of an actual light ray (s, t, u, v), where B_i,j,p,q (s,t,u,v) represents a basis function, and where w_i,j,p,q is a confidence level represented by;

##EQU27##20.

20. The method of claim 19 wherein the basis function is:

21. The method of claim 19 wherein the basis function is a quadralinear basis function.

22. The method of claim 19 wherein the basis function is:

where h_st is the distance between neighboring locations of calculated light rays and where h_uv is the distance between neighboring directions of calculated light rays.

23. A method in a computer system for determining intensity levels for the pixels of an image of an object from a collection of actual intensity levels of the image of the object at various vantage points comprising:

setting an initial intensity level for each pixel of the image, each initial intensity level being derived from the actual intensity levels of vantage points near the pixel;

setting a confidence level for each pixel of the image, the confidence level indicating a likelihood that the initial intensity level accurately represents the actual intensity level from the vantage point of the pixel;

for each of a plurality of groups of pixels of the image, calculating an average intensity level for the group wherein the initial intensity levels of the pixels in the group are weighted based on the confidence levels of the pixels in the group; and

for each of the plurality of groups of pixels of the image, adjusting the initial intensity levels of the pixels in the group based on the calculated average intensity level of the pixels in the group to give a final intensity level for each of the pixel in the group;

wherein the adjusting of the initial intensity levels of the pixels in the group includes adjusting the calculated intensity levels for successively smaller groups of pixel based on previously adjusted average intensity levels for larger groups of pixels.

24. A method in a computer system for determining intensity levels for the pixels of an image of an object from a collection of actual intensity levels of the image of the object at various vantage points, comprising:

setting an initial intensity level for each pixel of the image, each initial intensity level being derived from the actual intensity levels of vantage points near the pixel;

setting a confidence level for each pixel of the image, the confidence level indicating a likelihood that the initial intensity level accurately represents the actual intensity level from the vantage point of the pixel;

for each of a plurality of groups of pixels of the image, calculating an average intensity level for the group wherein the initial intensity levels of the pixels in the group are weighted based on the confidence levels of the pixels in the group; and

for each of the plurality of groups of pixels of the image, adjusting the initial intensity levels of the pixels in the group based on the calculated average intensity level of the pixels in the group to give a final intensity level for each of the pixel in the group;

wherein the calculating of an average intensity level for each group includes calculating average intensity levels for successively larger groups of pixels based on previously calculated average intensity levels for smaller groups of pixels and wherein the adjusting of the initial intensity levels of the pixels in the group includes adjusting the calculated intensity levels for successively smaller groups of pixel based on previously adjusted average intensity levels for larger groups of pixels.

25. A computer-readable medium containing instructions for causing a computer system to determine intensity levels for the pixels of an image of an object from a collection of actual intensity levels of the image of the object at various vantage points, comprising:

setting an initial intensity level for each pixel of the image, each initial intensity level being derived from the actual intensity levels of vantage points near the pixel;

setting a confidence level for each pixel of the image, the confidence level indicating a likelihood that the initial intensity level accurately represents the actual intensity level from the vantage point of the pixel;

for each of a plurality of groups of pixels of the image, calculating a combined intensity level for the group wherein the initial intensity levels of the pixels in the group are weighted based on the confidence levels of the pixels in the group;

for each of the plurality of groups of pixels of the image, adjusting the initial intensity levels of the pixels in the group based on the calculated combined intensity level of the pixels in the group to give a final intensity level for each of the pixel in the group

wherein the adjusting of the initial intensity levels of the pixels in the group includes adjusting the calculated intensity levels for successively smaller groups of pixel based on previously adjusted combined intensity levels for larger groups of pixels.

26. A computer-readable medium containing instructions for causing a computer system to determine intensity levels for the pixels of an image of an object from a collection of actual intensity levels of the image of the object at various vantage points, comprising:

setting an initial intensity level for each pixel of the image, each initial intensity level being derived from the actual intensity levels of vantage points near the pixel;

setting a confidence level for each pixel of the image, the confidence level indicating a likelihood that the initial intensity level accurately represents the actual intensity level from the vantage point of the pixel;

for each of a plurality of groups of pixels of the image, calculating a combined intensity level for the group wherein the initial intensity levels of the pixels in the group are weighted based on the confidence levels of the pixels in the group; and

for each of the plurality of groups of pixels of the image, adjusting the initial intensity levels of the pixels in the group based on the calculated combined intensity level of the pixels in the group to give a final intensity level for each of the pixel in the group

wherein the calculating of the combined intensity level for each group includes calculating combined intensity levels for successively larger groups of pixels based on previously calculated combined intensity levels for smaller groups of pixels and wherein the adjusting of the initial intensity levels of the pixels in the group includes adjusting the calculated intensity levels to successively smaller groups of pixel based on previously adjusted combined intensity levels for larger groups of pixels.