Joint view expansion and filtering for automultiscopic 3D displays

• US 9,756,316 B2
• Filed: 11/03/2014
• Issued: 09/05/2017
• Est. Priority Date: 11/04/2013
• Status: Active Grant

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Didyk, Piotr Krzysztof, Sitthi-Amorn, Pitchaya, Matusik, Wojciech, Durand, Frederic, Freeman, William T.
• Primary Examiner(s): Ortiz Criado, Jorge L
• Assistant Examiner(s): GADOMSKI, STEFAN J
• Application Number:
  US14/531,548
• Publication Number:
  US 20150124062A1
• Time in Patent Office: 1,037 Days
• Field of Search: None
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/302 for viewing without the aid of special glasses, i.e. using autostereoscopic displays
• CPC Class Codes: H04N 13/351 for displaying simultaneously

Abstract

Multi-view autostereoscopic displays provide an immersive, glasses-free 3D viewing experience, but they preferably use correctly filtered content from multiple viewpoints. The filtered content, however, may not be easily obtained with current stereoscopic production pipelines. The proposed method and system takes a stereoscopic video as an input and converts it to multi-view and filtered video streams that may be used to drive multi-view autostereoscopic displays. The method combines a phase-based video magnification and an interperspective antialiasing into a single filtering process. The whole algorithm is simple and may be efficiently implemented on current GPUs to yield real-time performance. Furthermore, the ability to retarget disparity is naturally supported. The method is robust and works transparent materials, and specularities. The method provides superior results when compared to the state-of-the-art depth-based rendering methods. The method is showcased in the context of a real-time 3D videoconferencing system.

Claims

1. A method of rendering a three-dimensional (3D) video display, comprising the computer-implemented steps, by a processor and associated memory:

receiving a video stream formed of a sequence of frames, each frame having image content corresponding to a plurality of views, the views being initial views;

applying one or more spatial band pass filters to the received image content resulting in filtered images, each spatial band pass filter having a respective spatial frequency band;

from the filtered images, computing one or more output images that synthesize additional views with respect to the initial views, the output images computed from the filtered images of a given spatial band pass filter corresponding to different visual disparities for the respective spatial frequency band of that given band pass filter;

the computing of output images including removing inter-view aliasing by performing filtering of the output images;

(i) according to local depth using phase shift including correcting the phase shift in each sub-band of the respective spatial frequency band separately based on the phase shift in a given corresponding sub-band having a lower frequency as compared with the respective spatial frequency band, and (ii) based upon performing a steerable pyramid decomposition and a corresponding attenuation of coefficients of the steerable pyramid decomposition; and

driving a display with the computed and anti-aliased filtered output images, rendering a multi-view autostereoscopic 3D video display.

2. A method as claimed in claim 1 wherein the received video stream is a 3D stereo video stream of images having two views per frame.

3. A method as claimed in claim 1 wherein the step of applying one or more spatial band pass filters includes applying a one-dimensional (1D) filter.

4. A method as claimed in claim 1 wherein the step of applying spatial band pass filters includes applying a two-dimensional (2D) filter.

5. A method as claimed in claim 1, wherein the step of computing the output images results in a stereo disparity expansion of views, the output images include a disparity range, and the disparity range in the output images is user adjustable by any of:

(i) adjusting a magnification factor in the given spatial band pass filter, and (ii) at least one of defining and translating a disparity mapping function to map a certain phase shift at the spatial frequency of the given spatial band pass filter to a new phase shift.

6. A method as claimed in claim 1 wherein the step of computing includes interpolating in-between views.

7. A method as claimed in claim 1 wherein the step of applying spatial band pass filters captures correspondence between views using phase differences for multiple spatial frequencies and orientations separately, and in the step of computing, local depth is represented as a plurality of values instead of as a single value.

8. A method as claimed in claim 1 wherein the step of driving the display is in real-time to the step of receiving the video stream.

9. A method as claimed in claim 1 further comprising prealigning the initial views with each other before applying the spatial band pass filters.

10. A method as claimed in claim 1 wherein the optional antialiasing is used for adding depth-of-field effect.

11. A method as claimed in claim 1 wherein the plurality of views includes two or more views.

12. A computer-implemented system for rendering a three-dimensional (3D) video display, comprising:

at least one processor configured to implement;

a receiving module configured to receive a video stream formed of a sequence of frames, each frame having image content corresponding to a plurality of views, the views being initial views;

a computing module responsive to the receiving module and configured to apply one or more spatial band pass filters to the received image content resulting in filtered images, each spatial band pass filter having a respective spatial frequency band;

the computing module further configured to compute, from the filtered images, one or more output images that synthesize additional views with respect to the initial views, the output images computed from the filtered images of a given spatial band pass filter corresponding to different visual disparities for the respective spatial frequency band of that given band pass filter;

the computing module further configured to remove inter-view aliasing by performing filtering of the output images;

(i) according to local depth using phase shift including correcting the phase shift in each sub-band of the respective spatial frequency band separately based on the phase shift in a given corresponding sub-band having a lower frequency as compared with the respective spatial frequency band, and (ii) based upon performing a steerable pyramid decomposition and a corresponding attenuation of coefficients of the steerable pyramid decomposition; and

a display module coupled to receive from the computing module and configured to drive a display with the computed and anti-aliased filtered output images, rendering a multi-view autostereoscopic 3D video display.

13. A system as claimed in claim 12 wherein the system is a real-time 3D video conferencing system.

14. A system as claimed in claim 12 wherein the received video stream is a 3D stereo video stream of images having two views per frame.

15. A system as claimed in claim 12 wherein the computing module is further configured to apply at least one one-dimensional (1D) filter corresponding to at least one of the one or more spatial band pass filters.

16. A system as claimed in claim 12 wherein the computing module is further configured to apply at least one two-dimensional (2D) filter corresponding to at least one of the one or more spatial band pass filters.

17. A system as claimed in claim 12 wherein the computing module is further configured to compute the output images, resulting in a stereo disparity expansion of views, the output images further include a disparity range, and the display module is further configured to enable a user to adjust the disparity range in the output images by any of:

(i) adjust a magnification factor in the given spatial band pass filter, and (ii) at least one of define and translate a disparity mapping function to map a certain phase shift at the spatial frequency of the given spatial band pass filter to a new phase shift.

18. A system as claimed in claim 12 wherein the computing module is further configured to interpolate in-between views.

19. A system as claimed in claim 12 wherein the computing module is further configured to apply spatial band pass filters including capturing correspondence between views using phase differences for multiple spatial frequencies and orientations separately, and the computing module is further configured to compute local depth including representing local depth as a plurality of values instead of as a single value.

20. A system as claimed in claim 12 wherein the display module is further configured to drive the display and the computing module is further configured to receive the video stream in real-time.

21. A system as claimed in claim 12 further comprising the computing module being configured to prealign the initial views with each other before the computing module is configured to apply the one or more spatial band pass filters.

22. A system as claimed in claim 12 wherein the optional antialiasing is used for adding depth-of-field effect.

23. A system as claimed in claim 12 wherein the plurality of views includes two or more views.

24. A non-transitory computer readable medium having stored thereon a sequence of instructions which, when loaded and executed by a processor coupled to an apparatus, causes the apparatus to:

receive a video stream formed of a sequence of frames, each frame having image content corresponding to a plurality of views, the views being initial views;

apply one or more spatial band pass filters to the received image content resulting in filtered images, each spatial band pass filter having a respective spatial frequency band;

compute, from the filtered images, one or more output images that synthesize additional views with respect to the initial views, the output images computed from the filtered images of a given spatial band pass filter corresponding to different visual disparities for the respective spatial frequency band of that given band pass filter;

including removing inter-view aliasing by performing filtering of the output images;

(i) according to local depth using phase shift including correcting the phase shift in each sub-band of the respective spatial frequency band separately based on the phase shift in a given corresponding sub-band having a lower frequency as compared with the respective spatial frequency band, and (ii) based upon performing a steerable pyramid decomposition and a corresponding attenuation of coefficients of the steerable pyramid decomposition; and

drive a display with the computed and anti-aliased filtered output images, rendering a multi-view autostereoscopic 3D video display.