Method and System for Decoding and Displaying 3D Light Fields

• US 20080043096A1
• Filed: 04/04/2007
• Published: 02/21/2008
• Est. Priority Date: 04/04/2006
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Vetro, Anthony, Zwicker, Matthias, Yea, Sehoon, Matusik, Wojciech, Pfister, Hanspeter
• Granted Patent: US 8,044,994 B2
• Time in Patent Office: Days
• US Class Current: 348/51
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

A method and system acquire and display light fields. A continuous light field is reconstructed from input samples of an input light field of a 3D scene acquired by cameras according to an acquisition parameterization. The continuous light is reparameterized according to a display parameterization and then prefiltering and sampled to produce output samples having the display parametrization. The output samples are displayed as an output light field using a 3D display device. The reconstruction can be performed by interpolating the input samples having the different views;

Claims

1. A method for processing a light field of a three dimensional scene to he displayed on a three dimensional display device, in which the light field is acquired from a plurality of different views by a plurality of cameras, there being one view for each camera, and in which each view has an associated spatial resolution, comprising, the steps of:

determining, in a decoder, a set of target views and an associated target spatial resolution for each target view according to a display parameterization of the three dimensional display device; and

decoding, in the decoder, a scalable video bitstream to produce a decoded light field for the set of target views and the associated target spatial resolution for each target view, in which the scalable video bitstream is an encoded representation of the light field and has temporal dependencies and spatial dependencies.

2. The method of claim 1, further comprising;

discarding portions of the scalable video bitstream corresponding to selected non-target views during the decoding.

4. The method of claim 1, in which the set of target views have a minimum, baseline distance between the plurality of different views acquired by the plurality of cameras.

5. The method of claim 1, in which target views have equal baseline distances between spatially adjacent target views.

6. The method of claim 1, in which the target spatial resolution of each target view is equal to a spatial resolution of the three dimensional display device.

7. The method of claim 1, in which the target spatial resolution is higher than a spatial resolution of the three dimensional display device.

8. The method of claim 1, in which the target spatial resolution of each target view is lower than a spatial resolution of the three dimensional display device.

9. The method of claim 1, in which the decoding also produces a decoded depth map.

10. The method of claim 1, further comprising;

encoding the light field for the plurality of different views with the spatial resolution of each, view in a scalable manner to produce the scalable video bitstream.

11. The method of claim 1, in which the set of target views is larger than the plurality of different views.

12. The method of claim 11, in which additional views for the set of target views are generated using the decoded depth maps and view interpolation during the decoding.

13. The method of claim 10, in which the encoding determines a maximum disparity between the plurality of different views, and further comprising:

signaling the maximum disparity to the decoder.

14. The method of claim 13, in which the maximum disparity is signaled in supplemental enhancement information messages according to a H.264/AVC video coding standard.

15. The method of claim 13, in which the decoder decodes the maximum disparity.

16. The method of claim 13, in which the maximum disparity is expressed at a pixel resolution.

17. The method of claim 13, in which the maximum disparity is expressed at a sub-pixel resolution.

18. The method of claim 1, further comprising:

encoding a plurality of depth maps to produce a depth map bitstream,

19. The method of claim 18, in which the depth map bitstream is multiplexed into the scalable video bitstream.

20. The method of claim 10, further comprising:

interpolating input samples of the acquired light field to produce a reparameterized light field, in which the interpolating is according to an acquisition parameterization and the display parameterization;

filtering the reparameterized light field to produce a filtered light field; and

sampling the filtered light field to produce output samples of the sampled light field having the display parameterization,

21. The method of claim 20, in which the interpolating uses depth maps acquired of the scene,

22. The method of claim 20, in which the interpolating further comprises:

reprojecting the input samples using an unstructured lumigraph rendering process.

23. The method of claim 20, in which the sampling is an oversampling in a plane of the three dimensional display device to reduce inter-perspective aliasing in an oversampled light field, and in which the oversampled light field is subsampled according to the display parametrization to produce the output samples.

24. The method of claim 20, in which the interpolating generates additional views of the three dimensional scene.

25. The method of claim 23, in which the oversampling is by a factor of (d+1)/2, where d is the maximum disparity of the different views.

26. The method of claim 1, further comprising:

interpolating samples of the decoded light field to produce a reparameterized light field, in which the interpolating is according to an acquisition parameterization and the display parameterization;

filtering the reparameterized light field to produce a filtered light field; and

sampling the filtered light field to produce samples of the sampled output light field having the display parameterization.

27. The method of claim 26, in which the interpolating uses the decoded depth map.

28. The method of claim 27, in which the interpolating further comprises:

reprojecting the samples of the decoded light field using an unstructured lumigraph rendering process.

29. The method of claim 26, in which the sampling is an oversampling in a plane of the three dimensional display device to reduce inter-perspective aliasing in an oversampled light field, and in which the oversampled light field is subsampled according to the display parametrization to produce the samples of the sampled output light field.

30. The method of claim 26, in which the interpolating generates additional views of the three dimensional scene.

31. The method of claim 29, in which the oversampling is by a factor of (d+1)/2, where d is a maximum disparity of the different views.

32. The method of claim 15, in which the maximum disparity is modified according to the target views.

3. The method of claim L further comprising;

discarding portions of the scalable video bitstream corresponding to a higher spatial resolution than the target spatial resolutions during the decoding.

33. A system for processing a light field of a three dimensional scene to be displayed on a three dimensional display device, in which the light field is acquired from a plurality of different views by a plurality of cameras, there being one view for each camera, and in which each view has an associated spatial resolution, comprising:

a video decoder configured to determining a set of target views and an associated target spatial resolution for each target view according to a display parameterization of the three dimensional display device, and the decoder farther comprising;

means for decoding, in the decoder, a scalable video bitstream to produce a decoded light field for the set of target views and the associated target spatial resolution for each target view, in which the scalable video bitstream encodes the acquired light field and has temporal dependencies and spatial dependencies.

34. The system of claim 33, in which the decoding generates additional views for the set of target views using decoded depth maps and view interpolation.

35. The system of claim 33, further comprising:

means for interpolating input samples of the acquired light field to produce a reparameterized light field, in which the interpolating is according to an acquisition parameterization and the display parameterization;

means for filtering the reparameterized light field to produce a filtered light field; and

means for sampling the filtered light field to produce output samples of the sampled light field having the display parameterization.