Method and system for light field rendering

• US 6,097,394 A
• Filed: 04/28/1997
• Issued: 08/01/2000
• Est. Priority Date: 04/28/1997
• Status: Expired due to Term

• Current Assignee: Interflo Technologies, Inc.
• Original Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Inventors: Levoy, Marc, Hanrahan, Pat
• Primary Examiner(s): Zimmerman, Mark K.
• Assistant Examiner(s): Lee, Albert K.
• Application Number:
  US08/848,569
• Time in Patent Office: 1,191 Days
• Field of Search: 345/419, 345/426, 345/475, 345/427
• US Class Current: 345/419
• CPC Class Codes: G02B 30/50 the image being built up from image elements distributed over a 3D volume, e.g. voxels
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses

Abstract

Described is a simple and robust method and system for generating new views from arbitrary camera positions without depth information or feature matching, simply by combining and resampling the available images. This technique interprets input images as two-dimensional slices of a four dimensional function--the light field. This function completely characterizes the flow of light through unobstructed space in a static scene with fixed illumination. A sampled representation for light fields allows for both efficient creation and display of inward and outward looking views. Light fields may be created from large arrays of both rendered and digitized image. The latter are acquired with a video camera mounted on a computer-controlled gantry. Once a light field has been created, new views may be constructed in real time by extracting slices in appropriate directions. Also described is a compression system that is able to compress generated light fields by more than a factor of 100:1 with very little loss of fidelity. Issues of antialiasing during creation and resampling during slice extraction are also addressed.

Claims

1. A method for creating a new image of a scene, comprising the steps of:

acquiring light field radiance data by collecting an array of planar images of the scene as observed from a first planar (u,v) surface, said planar array of planar images having inter-element coherence;

storing and compressing said acquired light field radiance data;

computing line parameters for image rays originating from the scene which pass through said first planar (u,v) surface and which intercept a second planar (x,y) pixel surface using projective mapping; and

calculating the radiance of said image rays intercepting said (x,y) pixel surface by using a subset of said acquired light-field radiance data.

2. The method of claim 1, further comprising the step of:

selectively decompressing a subset of said acquired light-field radiance data to calculate the radiance of said image rays intercepting said second planar (x,y) pixel surface.

3. The method of claim 2, wherein the step of selectively decompressing a subset of said acquired light-field radiance data utilizes a decoding stage wherein a file containing compressed data is loaded into memory and a second dequantization stage is executed for calculating the radiance of image rays.

4. The method of claim 1, wherein the step of storing and compressing said light-field radiance data includes a two-stage pipeline compression scheme.

5. The method of claim 4, wherein the two-stage compression scheme comprises a fixed rate vector quantization step and an entropy coding step.

6. The method of claim 5, wherein the entropy coding step comprises Lempel-Ziv coding.

7. The method of claim 1, wherein light-field radiance data is decompressed as required with the steps of:

gzip decoding as a file is loaded into memory; and

dequantization.

8. The method of claim 1, wherein the step of calculating the radiance of said image rays includes reconstructing a four-dimensional plenoptic function for said image rays from the acquired light field data.

9. The method of claim 8, wherein the four-dimensional plenoptic function is approximated by interpolating the four-dimensional function from the subset of images proximate to the intersection of said image rays with said (u,v) plane.

10. A method for creating a new image of a scene at a new viewpoint comprising the steps of:

acquiring light-field radiance data along at least one planar (u,v) surface by collecting an array of planar images directed towards the center of the scene, said array of planar images having inter-element coherence;

storing and compressing said acquired light field radiance data;

selecting a planar pixel element along a new viewpoint plane;

computing line parameters for image rays originating from the scene which pass through said at least one planar (u,v) surface and which intercept said selected pixel element; and

calculating the radiance for said image rays intersecting said pixel element by using a subset of said acquired light-field radiance data.

11. The method of claim 10, wherein each said image ray is represented by an intersection between two planes.

12. The method of claim 11, wherein projective mapping is used to map image rays onto the planar surface of said selected pixel element.

13. The method of claim 12, further comprising the step of:

selectively decompressing a subset of said acquired lightfield radiance data which is required to calculate the radiance of said image rays intercepting the planar surface of said selected pixel element.

14. The method of claim 12, wherein the step of storing and compressing said light field radiance data includes a two-stage pipeline compression scheme.

15. The method of claim 14, wherein the two-stage compression scheme comprises a fixed rate vector quantization step and an entropy coding step.

16. The method of claim 10, further comprising the step of:

selectively decompressing a subset of said acquired light field radiance data to calculate the radiance of said image rays intercepting said selected pixel element, wherein the step of decompressing includes a decoding stage wherein a file containing compressed data is loaded into memory and a second dequantization stage executed for computing the radiance of said image rays.

17. The method of claim 10, wherein said step of computing the radiance for said image rays comprises a resampling process in which a four-dimensional light field radiance function is approximated by interpolating nearest neighbor images.

18. The method of claim 10, wherein said step of computing the radiance for said image rays comprises reconstructing a four dimensional plenoptic function.

19. A system for creating and displaying new images of a scene comprising:

means for acquiring light field data composed of an array of planar images of the scene along at least one plan a surface, said array of planar images having inter-element coherence;

means for storing and compressing said light-field data;

means for computing image rays intercepting pixels at a new location; and

resampling means for calculating the radiance of said image rays using said compressed light field data.

20. The system of claim 19 wherein said means for acquiring light field data comprises a camera and further comprising:

a computer controlled gantry to which said camera is mounted.

21. The system of claim 20, further comprising pan and tilt motors for said camera.

22. The system of claim 20, wherein said camera has a wide angle lens.

23. The system of claim 19, further comprising:

means for prefiltering the light field data to alleviate aliasing.

24. The system of claim 23, wherein said means for prefiltering the light field comprises at least one of the group consisting of a pixel filter and an aperture filter.

25. The system of claim 19, wherein said means for acquiring light field data comprises a virtual camera.

26. The system of claim 25, further comprising:

rendering means to generate the image of said virtual camera.

27. The system of claim 19, further comprising:

means for selectively decompressing a subset of said compressed light field data to generate a new image.