Method and system for digital plenoptic imaging

• US 6,023,523 A
• Filed: 06/30/1997
• Issued: 02/08/2000
• Est. Priority Date: 02/16/1996
• Status: Expired due to Term

• Current Assignee: Uber Technologies, Inc.
• Original Assignee: Microsoft Corporation
• Inventors: Gortler, Steven, Szeliski, Richard S., Cohen, Michael F.
• Primary Examiner(s): Boudreau, Leo H.
• Assistant Examiner(s): Werner, Brian P.
• Application Number:
  US08/885,259
• Time in Patent Office: 953 Days
• Field of Search: 382/154, 382/173, 382/285, 382/294, 382/299, 382/300, 345/419, 345/425, 345/952, 345/953, 345/426, 345/475, 345/420, 345/430, 345/427, 356/376, 348/39, 348/50, 348/121, 348/135, 348/142, 359/23, 359/463, 359/462
• US Class Current: 382/154
• CPC Class Codes: G06T 15/06 Ray-tracing
• CPC Class Codes: G06T 15/50 Lighting effects

Abstract

A computer-based method and system for digital 3-dimensional imaging of an object which allows for viewing images of the object from arbitrary vantage points. The system, referred to as the Lumigraph system, collects a complete appearance of either a synthetic or real object (or a scene), stores a representation of the appearance, and uses the representation to render images of the object from any vantage point. The appearance of an object is a collection of light rays that emanate from the object in all directions. The system stores the representation of the appearance as a set of coefficients of a 4-dimensional function, referred to as the Lumigraph function. From the Lumigraph function with these coefficients, the Lumigraph system can generate 2-dimensional images of the object from any vantage point. The Lumigraph system generates an image by evaluating the Lumigraph function to identify the intensity values of light rays that would emanate from the object to form the image. The Lumigraph system then combines these intensity values to form the image.

Claims

1. A method for generating an image of an object from a vantage point, the object being represented by a plenoptic function having coefficients stored in a memory, the coefficients representing intensity values of light rays emanating from the object that pass through predefined points on a surface surrounding the object and in predefined directions through each point, the method comprising:

receiving the coefficients of the plenoptic function, the plenoptic function for generating intensity values of light rays emanating from the object and passing through the surface surrounding the object; and

for each of a plurality of pixels of the image,

determining a point on the surface surrounding the object and a direction of a light ray that passes through the determined point and that contributes to the pixel of the image;

generating an intensity for the pixel by,

when a stored coefficient represents the intensity value of the light ray emanating from the object that passes through the determined point in the determined direction, evaluating the plenoptic function at that determined point and direction to generate an intensity for the pixel; and

when a stored coefficient does not represent the intensity value of the light ray emanating from the object that passes through the determined point in the determined direction, generating the intensity for the pixel by,

identifying multiple light rays emanating from the object that pass through predefined points on the surface surrounding the object and in predefined directions through each point such that the light rays pass near the pixel;

evaluating the plenoptic function at the predefined point and direction for each of the identified light rays to determine an intensity; and

combining the determined intensities for the multiple identified light rays into the generated intensity for the pixel; and

setting the pixel to the generated intensity.

2. The method of claim 1 wherein the determining of a point on the surface includes identifying a center of projection of the image and wherein the direction is the direction from the center of projection to that pixel.

3. The method of claim 1 wherein the coefficients of the plenoptic function correspond to intensity levels at predefined points on the surface in predefined directions and the evaluating of the plenoptic function includes selecting the intensity level of the predefined point and predefined direction of a light ray emanating from the object that passes nearest to that pixel.

4. The method of claim 1 wherein the coefficients of the plenoptic function correspond to intensity levels of Lumigraph light rays at predefined points on the surface in predefined directions and the evaluating of the plenoptic function includes combining intensity levels of the predefined points and predefined directions of the Lumigraph light rays emanating from the object that pass nearest to that pixel in the direction of a center of projection of the image.

5. The method of claim 4 wherein the intensity levels are combined using a weighting based on distance between that pixel and the light rays that pass nearest that pixel.

6. The method of claim 4 wherein the intensity levels are combined using a quadralinear basis function.

7. The method of claim 6 wherein the quadralinear basis function is ##EQU20##

8. The method of claim 6 wherein intensity levels are combined using the following equation:

where s and t identify that pixel on the image and u and v represent the direction of the light ray that passes through that pixel, where i and j represent a predefined point and p and q represent a predefined direction, and where B represents the basis function.

9. A method for generating an image of an object from a vantage point, the object being represented by a plenoptic function having coefficients stored in a memory, the coefficients representing intensity values of light rays emanating from the object that pass through predefined points on a surface surrounding the object and in predefined directions through each point, the method comprising:

receiving the coefficients of the plenoptic function, the plenoptic function for generating intensity values of light rays emanating from the object and passing through the surface surrounding the object;

for each of a plurality of pre-defined points on the surface,

identifying a plurality of directions of light rays that pass through an area on the surface around the pre-defined point, the plurality of directions corresponding to a plurality of coefficients of the plenoptic function that define a texture map;

identifying an area on a graphics object representing the area on the surface; and

filling the identified area with the texture defined by the texture map; and

rendering the graphics object with the filled texture to generate an image of the object.

10. The method of claim 9 wherein the coefficients of the plenoptic function correspond to intensity levels at predefined points on the surface in predefined directions, wherein the coefficients for each predefined point are stored in a 2-dimensional array, and wherein the defined texture map is a portion of the 2-dimensional array.

11. The method of claim 10 wherein the filling of the identified area uses bilinear interpolation supported by a computer system.

12. The method of claim 10 wherein the filling of the identified are uses alpha blending supported by a computer system.

13. The method of claim 10 wherein the filling of the identified are uses a combination of bilinear interpolation and alpha blending supported by a computer system.

14. A method in a computer system for generating an image of an object, the object being represented by a data structure representing intensity values of light rays emanating from the object, the data structure storing intensity values of light rays that pass through predefined points on a surface surrounding the object and in predefined directions through each point, the image of the object having a center of projection, comprising:

for each predefined point,

selecting an area around the predefined point;

identifying the light rays in the data structure that emanate from the object and pass through the selected area and intersect at the center of projection;

defining as a texture map the intensity values of the identified light rays;

identifying an area of the image through which the identified light rays pass; and

filling the identified area in accordance with the defined texture map.

15. The method of claim 14 wherein the filling includes a bilinear interpolation of the texture map.

16. The method of claim 14 wherein the filling includes alpha blending of the texture map with intensity values previously filled in the identified area.

17. The method of claim 14 wherein the selected areas around the predefined points are non-overlapping squares.

18. The method of claim 14 including rendering the filled identified areas.

19. The method of claim 14 wherein the identified areas of the image are hexagons defined by 6 predefined points near each predefined point.

20. The method of claim 19 wherein the filling includes alpha blending of triangular regions of the hexagons.

21. The method of claim 14 wherein the filling approximates a quadralinear interpolation with a bilinear interpolation and alpha blending.

22. The method of claim 14 wherein a 3-dimensional shape of the object is represented by a depth data structure indicating a position of the surface of the object at various points and wherein the identifying of the light rays factors in the depths of the object at positions from which the light rays emanate.

23. The method of claim 14 wherein the identifying of the light rays factors depths of the object from which the light rays emanate.

24. The method of claim 14 including generating the data structure before the identifying of the light rays in the data structure.

25. The method of claim 14 including receiving the stored intensity values for the identified light rays before the filling of the identified area.

26. The method of claim 14 wherein the data structure stores the intensity values in a compressed form.

27. A method in a computer system for generating an image of an object whose appearance is represented by a Lumigraph data structure, the Lumigraph data structure storing intensity values corresponding to light rays that emanate from the object at predefined directions and that pass through predefined points on a surface surrounding the object, the image having a center of projection, the method comprising:

for each pixel of the image,

identifying from the data structure the predefined points and predefined directions of light rays that pass through the image near the pixel and pass near the center of projection;

generating a single intensity value for the pixel by,

when the Lumigraph data structure has an intensity value corresponding to a light ray that emanates from the object in such a direction as to pass through the pixel, determining that the single intensity value for the pixel is the intensity value for the light ray; and

when the Lumigraph data structure does not have an intensity value corresponding to a light ray that emanates from the object in such a direction as to pass through the pixel, combining the intensity values indicated by the identified predefined points and predefined directions into the single intensity value; and

setting the intensity value of the pixel to the generated intensity value.

28. The method of claim 27 wherein the identifying of predefined points and predefined direction of light rays factors in surface depth of the object.

29. The method of claim 28 wherein the surface depth of the object is represented by a depth data structure indicating depth of the object at various points on its surface.

30. The method of claim 27 including generating the data structure before the identifying of the predefined points and directions from the data structure.

31. The method of claim 27 including receiving the stored intensity values for the light rays corresponding to the identified predefined points and predefined directions before the generating of the single intensity value for the pixel.

32. The method of claim 27 wherein the data structure stores the intensity values in a compressed form.

33. A method in a computer system for generating an image of an object whose appearance is represented by a Lumigraph data structure, the Lumigraph data structure storing intensity values corresponding to light rays that emanate from the object at predefined directions and that pass through predefined points on a surface surrounding the object, the image having a center of projection, the method comprising:

for each pixel of the image,

identifying from the data structure the predefined points and predefined directions of light rays that pass through the image near the pixel and pass near the center of projection, the identifying factoring in surface depth of the object, the surface depth represented by a depth data structure indicating depth of the object at various points on its surface, the depth data structure being an octree data structure;

generating an intensity value for the pixel by combining the intensity values indicated by the identified predefined points and predefined directions; and

setting the intensity value of the pixel to the generated single intensity value.

34. The method of claim 33 wherein the computer system supports the representing of intensity levels of an object with an octree data structure, representing the depth of the object as an intensity level stored the octree data structure, and generating a 2-dimensional image of the object represented by the octree data structure so that the intensity level of the pixels of the image indicate the depth of the object.

35. A computer-readable medium containing instructions for causing a computer system to generate an image of an object, the object being represented by a data structure representing intensity values of light rays emanating from the object, the data structure storing intensity values of light rays emanating from the object that pass through predefined points on a surface surrounding the object and in predefined directions through each point, the image of the object having a center of projection, by:

for each predefined point,

selecting an area around the predefined point;

identifying the light rays in the data structure that emanate from the object and pass through the selected area and intersect near the center of projection;

defining as a texture map the intensity values of the identified light rays;

identifying an area of the image through which the identified light rays pass; and

filling the identified area in accordance with the defined texture map.

36. The computer-readable medium of claim 35 wherein the filling includes a bilinear interpolation of the texture map.

37. The computer-readable medium of claim 35 wherein the filling includes alpha blending of the texture map with intensity values previously filled in the identified area.

38. The computer-readable medium of claim 35 wherein the selected areas around the predefined points are non-overlapping squares.

39. The computer-readable medium of claim 35 including rendering the filled identified areas.

40. The computer-readable medium of claim 35 wherein the identified areas of the image are hexagons defined by 6 predefined points near each predefined point.

41. The computer-readable medium of claim 40 wherein the filling includes alpha blending of triangular regions of the hexagons.

42. The computer-readable medium of claim 35 wherein the filling approximates a quadralinear interpolation with a bilinear interpolation and alpha blending.

43. The computer-readable medium of claim 35 wherein a 3-dimensional shape of the object is represented by a depth data structure indicating a position of the surface of the object at various points and wherein the identifying of the light rays factors in the depths of the object at positions from which the light rays emanate.

44. The computer-readable medium of claim 35 wherein the identifying of the light rays factors depths of the object from which the light rays emanate.

45. The computer-readable medium of claim 35 wherein the instructions further cause the computer system to generate the data structure before the identifying of the light rays in the data structure.

46. The computer-readable medium of claim 35 wherein the instructions further cause the computer system to receive the stored intensity values for the identified light rays before the filling of the identified area.

47. The computer-readable medium of claim 35 wherein the data structure stores the intensity values in a compressed form.

48. A computer-readable medium containing instructions for causing a computer system to generate an image of an object whose appearance is represented by a data structure representing intensity levels corresponding to light rays that emanate from the object, the data structure storing intensity levels of light rays emanating from the object that pass through predefined points on a surface surrounding the object and in predefined directions through each point, the image having a center of projection, by:

for each pixel of the image,

identifying from the data structure the predefined points and predefined directions of light rays that pass through the image near the pixel and pass near the center of projection;

generating a single intensity level for the pixel by,

when the data structure has an intensity level corresponding to a light ray that emanates from the object in such a direction as to pass through the pixel, determining that the single intensity level for the pixel is the intensity level for the light ray; and

when the data structure does not have an intensity level corresponding to a light ray that emanates from the object in such a direction as to pass through the pixel, combining the intensity levels indicated by the identified predefined points and predefined directions into the single intensity level; and

setting the intensity level of the pixel to the generated intensity level.

49. The computer-readable medium of claim 48 wherein the identifying of predefined points and predefined direction of light rays factors in surface depth of the object.

50. The computer-readable medium of claim 49 wherein the surface depth of the object is represented by a depth data structure indicating depth of the object at various points on its surface.

51. The computer-readable medium of claim 48 wherein the instructions further cause the computer system to generate the data structure before the identifying of the predefined points and directions from the data structure.

52. The computer-readable medium of claim 48 wherein the instructions further cause the computer system to receive the stored intensity levels for the light rays corresponding to the identified predefined points and predefined directions before the generating of the single intensity level.

53. The computer-readable medium of claim 48 wherein the data structure stores the intensity levels in a compressed form.

54. A computer-readable medium containing instructions for causing a computer system to generate an image of an object whose appearance is represented by a data structure representing intensity levels corresponding to light rays that emanate from the object, the data structure storing intensity levels of light rays emanating from the object that pass through predefined points on a surface surrounding the object and in predefined directions through each point, the image having a center of projection, by:

for each pixel of the image,

identifying from the data structure the predefined points and predefined directions of light rays that pass through the image near the pixel and pass near the center of projection, the identifying factoring in surface depth of the object, the surface depth of the object represented by a depth data structure indicating depth of the object at various points on its surface, the depth data structure being an octree data structure;

generating an intensity level for the pixel by combining the intensity levels indicated by the identified predefined points and predefined directions; and

setting the intensity level of the pixel to the generated single intensity level.

55. The computer-readable medium of claim 54 wherein the computer system supports the representing of intensity levels of an object with an octree data structure, representing the depth of the object as an intensity level stored the octree data structure, and generating a 2-dimensional image of the object represented by the octree data structure so that the intensity level of the pixels of the image indicate the depth of the object.