Method and system for decoding and displaying 3D light fields

• US 8,044,994 B2
• Filed: 04/04/2007
• Issued: 10/25/2011
• Est. Priority Date: 04/04/2006
• Status: Expired due to Fees

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Vetro, Anthony, Zwicker, Matthias, Yea, Sehoon, Matusik, Wojciech, Pfister, Hanspeter
• Primary Examiner(s): Nguyen; Quang N.
• Application Number:
  US11/696,596
• Publication Number:
  US 20080043096A1
• Time in Patent Office: 1,665 Days
• Field of Search: None
• US Class Current: 348/42
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

A method and system acquire and display light fields. A continuous light field is reconstructed from input samples of an input light field of a 3D scene acquired by cameras according to an acquisition parameterization. The continuous light is reparameterized according to a display parameterization and then prefiltering and sampled to produce output samples having the display parametrization. The output samples are displayed as an output light field using a 3D display device. The reconstruction can be performed by interpolating the input samples having the different views.

Claims

1. A method for processing a light field of a three dimensional scene to be displayed on a three dimensional display device, in which the light field is acquired from a plurality of different views by a plurality of cameras, there being one view for each camera, and in which each view has an associated spatial resolution, comprising, the steps of:

determining, in a video decoder, a set of target views and an associated target spatial resolution for each target view according to a display parameterization of the three dimensional display device;

decoding, in the video decoder, a scalable video bitstream to produce a decoded light field for the set of target views and the associated target spatial resolution for each target view, in which the sealable video bitstream is an encoded representation of the light field and has temporal dependencies and spatial dependencies;

encoding the light field for the plurality of different views with the spatial resolution of each view in a scalable manner to produce the scalable video bitstream;

interpolating input samples of the acquired light field to produce a reparameterized light field, in which the interpolating is according to an acquisition parameterization and the display parameterization;

filtering the reparameterized light field to produce a filtered light field; and

sampling the filtered light field to produce output samples of the sampled light field having the display parameterization, in which the sampling is an oversampling in a plane of the three dimensional display device to reduce inter-perspective aliasing in an oversampled light field, and in which the oversampled light field is subsampled according to the display parameterization to produce the output samples.

2. The method of claim 1, further comprising;

discarding portions of the scalable video bitstream corresponding to selected non-target views during the decoding.

4. The method of claim 1, in which the set of target views have a minimum, baseline distance between the plurality of different views acquired by the plurality of cameras.

5. The method of claim 1, in which target views have equal baseline distances between spatially adjacent target views.

6. The method of claim 1, in which the target spatial resolution of each target view is equal to a spatial resolution of the three dimensional display device.

7. The method of claim 1, in which the target spatial resolution is higher than a spatial resolution of the three dimensional display device.

8. The method of claim 1, in which the target spatial resolution of each target view is lower than a spatial resolution of the three dimensional display device.

9. The method of claim 1, in which the decoding also produces a decoded depth map.

10. The method of claim 1, in which the set of target views is larger than the plurality of different views.

11. The method of claim 10, in which additional views for the set of target views are generated using the decoded depth maps and view interpolation during the decoding.

12. The method of claim 1, in which the encoding determines a maximum disparity between the plurality of different views, and further comprising:

signaling the maximum disparity to the video decoder.

13. The method of claim 12, in which the maximum disparity is signaled in supplemental enhancement information messages according to a H.264/AVC video coding standard.

14. The method of claim 12, in which the decoder decodes the maximum disparity.

15. The method of claim 14, in which the maximum disparity is modified according to the target views.

16. The method of claim 12, in which the maximum disparity is expressed at a pixel resolution.

17. The method of claim 12, in which the maximum disparity is expressed at a sub-pixel resolution.

18. The method of claim 1, further comprising:

encoding a plurality of depth maps to produce a depth map bitstream.

19. The method of claim 18, in which the depth map bitstream is multiplexed into the scalable video bitstream.

20. The method of claim 1, in which the interpolating uses dept maps acquired of the scene.

21. The method of claim 1, in which the interpolating further comprises:

reprojecting the input samples using an unstructured lumigraph rendering process.

22. The method of claim 1, in which the interpolating generates additional views of the three dimensional scene.

23. The method of claim 1, in which the oversampling is by a factor of (d+1)/2, where d is the maximum disparity of the different views.

24. The method of claim 1, further comprising:

interpolating samples of the decoded light field to produce a reparameterized light field, in which the interpolating is according to an acquisition parameterization and the display parameterization;

filtering the reparameterized light field to produce a filtered light field; and

sampling the filtered light field to produce samples of the sampled output light field having the display parameterization.

25. The method of claim 24, in which the interpolating uses the decoded depth map.

26. The method of claim 25, in which the interpolating further comprises:

reprojecting the samples of the decoded light field using an unstructured lumigraph rendering process.

27. The method of claim 24, in which the sampling is an oversampling in a plane of the three dimensional display device to reduce inter-perspective aliasing in an oversampled light field, and in which the oversampled light field is subsampled according to the display parametrization to produce the samples of the sampled output light field.

28. The method of claim 27, in which the oversampling is by a factor of (d+1)/2, where d is a maximum disparity of the different views.

29. The method of claim 24, in which the interpolating generates additional views of the three dimensional scene.

3. The method of claim L further comprising;

discarding portions of the scalable video bitstream corresponding to a higher spatial resolution than the target spatial resolutions during the decoding.

30. A system for processing a light field of a three dimensional scene to be displayed on a three dimensional display device, in which the light field is acquired from a plurality of different views by a plurality of cameras, there being one view for each camera, and in which each view has an associated spatial resolution, comprising:

a video decoder configured to determine a set of target views and an associated target spatial resolution for each target view according to a display parameterization of the three dimensional display device, and the video decoder further comprising;

means for decoding, in the video decoder, a scalable video bitstream to produce a decoded light field for the set of target views and the associated target spatial resolution for each target view, in which the sealable video bitstream is an encoded representation of the light field and has temporal dependencies and spatial dependencies;

means for encoding the light field for the plurality of different views with the spatial resolution of each view in a scalable manner to produce the scalable video bitstream;

means for interpolating input samples of the acquired light field to produce a reparameterized light field, in which the interpolating is according to an acquisition parameterization and the display parameterization;

means for filtering the reparameterized light field to produce a filtered light field; and

means for sampling the filtered light field to produce output samples of the sampled light field having the display parameterization, in which the sampling is an oversampling in a plane of the three dimensional display device to reduce inter-perspective aliasing in an oversampled light field, and in which the oversampled light field is subsampled according to the display parameterization to produce the output samples.

31. The system of claim 30, in which the decoding generates additional views for the set of target views using decoded depth maps and view interpolation.