Methods for Full Parallax Compressed Light Field Synthesis Utilizing Depth Information

• US 20160360177A1
• Filed: 08/22/2016
• Published: 12/08/2016
• Est. Priority Date: 04/23/2015
• Status: Active Grant

• Current Assignee: Ostendo Technologies Inc.
• Original Assignee: Ostendo Technologies Inc.
• Inventors: Graziosi, Danillo Bracco, Alpaslan, Zahir Y., El-Ghoroury, Hussein S.
• Granted Patent: US 10,070,115 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 6/00 Light guides; Structural details of arrangements comprising light guides and other optical elements, e.g. couplings
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/351 for displaying simultaneously

Abstract

An innovative method for synthesis of compressed light fields is described. Compressed light fields are commonly generated by sub-sampling light field views. The suppressed views must then be synthesized at the display, utilizing information from the compressed light field. The present invention describes a method for view synthesis that utilizes depth information of the scene to reconstruct the absent views. An innovative view merging method coupled with an efficient hole filling procedure compensates for depth misregistrations and inaccuracies to produce realistic synthesized views for full parallax light field displays.

Claims

1. A method for synthesis of a light field from a data set representing a plurality of predetermined views and respective disparity values comprising:

a) projecting the plurality of predetermined views using the respective disparity values to a single view position to provide a number of predetermined views at the single view position that is equal to the number of the predetermined views;

b) merging the plurality of predetermined views to define a merged view, wherein the merging includes;

i) evaluating each of the plurality of predetermined views to determine an apparent reliability of each predetermined view;

ii) for each pixel, selecting the most apparently reliable view not previously selected and using the respective disparity value or noting a position of a hole;

iii) again selecting the most apparently reliable view not previously selected and filling holes found in ii) if hole filling pixels are present;

iv) repeating ii) and iii) for still further predetermined views until all predetermined views have been processed; and

c) performing a hole-filling operation on the merged view to define a synthesized light field for display.

2. The method of claim 1 wherein the projecting the plurality of predetermined views that is different from the position of any of the plurality of predetermined views using the respective disparity values to a single view position.

3. The method of claim 1 wherein for each pixel for which a disparity value has been used in b) ii), replacing that disparity value as part of b) iii) as if filling a hole if that replacement disparity is larger than the disparity value used in b) ii).

4. The method of claim 1, further comprising:

determining a threshold based on noise in the respective disparity values; and

for each pixel in the most reliable view selected in b), ii), the respective disparity value for any respective pixel is used only if the absolute value of difference in the respective disparity and the disparity for the respective pixel of the next most reliable view is less than the threshold, otherwise the larger of the two disparities is used as the disparity of the respective pixel.

5. The method of claim 1 wherein the plurality of predetermined views are selected from a number of views greater than the predetermined plurality of views.

6. The method of claim 1 wherein a data set representing a plurality of predetermined views includes an identification of the position of each of the predetermined views, and wherein the position of at least one of the predetermined views is determined by analysis of the view.

7. The method of claim 1 wherein in b), i), the apparent reliability of each predetermined view is determined by finding a frequency of occurrence of each disparity in the respective view, and establishing the apparent reliability responsive to a weighted sum of N most frequent disparities, wherein the weights are provided as values of the disparity'"'"'s repetition in the respective view, wherein the same value of N is used for all predetermined views, and N is selected responsive to a number of different disparity values in a warped view.

8. The method of claim 1 wherein the hole filling operation of c) comprises comparing pixels at opposite boundaries of the hole and filling the hole by repeating the pixel of a boundary representing a background relative to the opposite boundary.

9. The method of claim 8 wherein the opposite boundaries are opposite boundaries along a horizontal line in the merged view.

10. The method of claim 1 wherein between a) and b), the method further comprises dividing the projected plurality of predetermined views into M by N blocks of pixels, and then completing b) and c), block by block.

11. The method of claim 10 wherein a size of the M by N blocks is adaptive based on the number of holes in the blocks after completing b) and c), block by block.

12. The method of claim 10 wherein after c), the synthesized light field is subjected to further processing.

13. The method of claim 12 wherein the further processing includes comparing block luminance of neighboring blocks and adjusting color levels according to neighboring color levels.

14. The method of claim 12 wherein the further processing includes filtering.

15. The method of claim 14 wherein the further processing includes deblocking filtering.