Content Adaptive Light Field Compression

• US 20170142427A1
• Filed: 11/15/2016
• Published: 05/18/2017
• Est. Priority Date: 11/16/2015
• Status: Active Grant

• Current Assignee: Ostendo Technologies Inc.
• Original Assignee: Ostendo Technologies Inc.
• Inventors: El-Ghoroury, Hussein S., Graziosi, Danillo Bracco, Alpaslan, Zahir Y., McNeill, Dale A.
• Granted Patent: US 10,448,030 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: G06T 9/20 Contour coding, e.g. using detection of edges
• CPC Class Codes: H04N 19/103 Selection of coding mode or of prediction mode
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

A compression method for high-resolution light field display is disclosed for applications in which computer memory constraints and latency are critical. The disclosed compression algorithm takes advantage of the 3D structure of a light field to compress the raw light field information with a fixed compression ratio and simple decoding instructions. The compressed high-resolution light field achieves a reduced bandwidth with acceptable quality, and is packed in a way that can be transmitted using common interfaces. In a preferred embodiment, the compression algorithm is used as a post-processing stage after light field information acquisition or after rendering. In a further preferred embodiment, the compression algorithm is incorporated into the acquisition or rendering procedure to reduce memory and rendering processing. These and further embodiments generate a compressed light field with all information required for direct display.

Claims

1. A method for compressing high-definition light field hogel data for transmission of the compressed high-resolution light field hogel data in a limited-bandwidth by wire or wireless transmission systems, comprising:

establishing a plurality of compression methods, each compression method having a light field hogel data dependent metric indicative of a light field reconstruction quality the respective compression method will provide;

determining for each compression method, a value of the light field hogel data dependent metric for the high-definition light field hogel data to be compressed;

selecting the compression method yielding the light field hogel data dependent metric indicative of a most favorable light field reconstruction quality;

compressing the high-definition light field hogel data according to the compression method selected; and

packing a compression mode and disparity values with the compressed high-definition light field hogel data, the compression mode being indicative of the compression method selected.

2. The method of claim 1 wherein the compression methods compress a pair of adjacent hogels into a single hogel.

3. The method of claim 1 wherein the compression methods compress more than a pair of adjacent hogels into a single hogel.

4. The method of claim 3 wherein the high-definition light field hogel data comprises hogel anglet data, and wherein bicubic anglet interpolation is used in compressing the high-definition light field hogel data according to the compression method selected.

5. The method of claim 1 wherein the high-definition light field hogel data comprises multicolor high-definition light field hogel data, and wherein the high-definition light field hogel data is high-definition light field hogel data for each color or primary color in the multicolor high-definition light field hogel data, and wherein the method for compressing the high-definition light field hogel data is carried out on a color hogel by color hogel basis.

6. The method of claim 5 wherein one of the compression modes Mode_ANGLET is a method of compressing corresponding anglets of adjacent color hogels of the same color.

7. The method of claim 6 wherein a pair of adjacent anglets of one color hogel and a corresponding pair of anglets of an adjacent color hogel of the same color are compressed by packing one color anglet of a first of the adjacent pair of color hogels with the corresponding color anglet of a second of the adjacent pair of color hogels as the compressed high-definition light field hogel data.

8. The method of claim 7 wherein the high-definition light field hogel data dependent metric for the compression mode Mode_ANGLET is calculated by;

for each color hogel

a) separating odd and even hogels, the odd and even hogels being adjacent hogels;

b) for odd and even anglets in each of the odd and even hogels, the odd and even anglets being adjacent anglets in the respective hogel and each anglet having an intensity;

1) determining the absolute difference in intensity of the even anglet of the odd hogel and the intensity of the odd anglet of the odd hogel;

2) determining the absolute difference in intensity of the even anglet of the even hogel and the intensity of the odd anglet of the even hogel; and

3) summing the absolute differences of

     1) and

     2).

9. The method of claim 8:

wherein a second and third of the compression modes are methods of compressing adjacent color hogels of the same color;

wherein the high-definition light field hogel data dependent metrics for the methods of compressing adjacent color hogels of the same color is calculated by;

for each color hogel

a) separating odd and even hogels, the odd and even hogels being adjacent hogels;

b) for odd and even anglets in each of the odd and even hogels, the odd and even anglets being corresponding adjacent anglets in the respective hogel and each anglet having an intensity;

1) determining the absolute difference in intensity of the even hogel anglets and the intensity of the odd hogel anglets as displaced as the high-definition light field hogel data dependent metric SAD_{ODD HOGEL} for the compression mode Mode_{ODD HOGEL} for compressing adjacent color hogels; and

2) determining the absolute difference in intensity of the odd hogel anglets and the intensity of the even hogel anglets as displaced as the compressed high-definition light field hogel data dependent metric SAD_{EVEN HOGEL} for the compression mode Mode_{EVEN HOGEL} for compressing adjacent color hogels;

3) the displacement or displacements of the even hogel anglets as displaced and the odd hogel anglets as displaced each being estimated by varying the displacement amount or amounts until the respective high-definition light field hogel data dependent metric for the respective compression mode reaches a minimum value;

wherein for the compression mode having the high-definition light field hogel data dependent metric referred to as SAD_{ODD HOGEL}, the packing is a packing of the mode Mode_{ODD HOGEL}, the disparity or disparities equal to the displacement or displacements of the anglets of the odd hogel, and the intensity of the anglets of the odd hogel, and for the high-definition light field hogel data dependent metric referred to as SAD_{EVEN HOGEL}, the packing is a packing of the mode Mode_{EVEN HOGEL}, the disparity or disparities equal to the displacement or displacements of the anglets of the odd hogel, and intensity of the anglets of the even hogel;

the method further comprising decoding the compressed high-definition light field hogel data by;

i) unpacking the packed compression mode, disparity value or values and compressed high-definition light field hogel data;

ii) when the unpacked compression mode is Mode_ANGLET, replicate odd anglets;

iii) when the unpacked compression mode is Mode_{EVEN HOGEL}, generate displaced even hogel anglets;

iv) when the unpacked compression mode is Mode_{ODD HOGEL}, generate displaced odd hogel anglets; and

repeat ii) through iv) until a respective frame is done.

10. The method of claim 5 wherein two of the compression modes are methods of compressing adjacent color hogels of the same color.

11. The method of claim 10 wherein the high-definition light field hogel data dependent metrics for the methods of compressing adjacent color hogels of the same color is calculated by;

for each color hogel

a) separating odd and even hogels, the odd and even hogels being adjacent hogels;

b) for odd and even anglets in each of the odd and even hogels, the odd and even anglets being adjacent anglets in the respective hogel and each anglet having an intensity;

1) determining the absolute difference in intensity of the even hogel anglets and the intensity of the odd hogel anglets as displaced as the high-definition light field hogel data dependent metric SAD_{ODD HOGEL} for the compression mode Mode_{ODD HOGEL} for compressing adjacent color hogels; and

2) determining the absolute difference in intensity of the odd hogel anglets and the intensity of the even hogel anglets as displaced as the compressed high-definition light field hogel data dependent metric SAD_{EVEN HOGEL} for the compression mode Mode_{EVEN HOGEL} for compressing adjacent color hogels;

3) the displacement or displacements of the even hogel anglets as displaced and the odd hogel anglets as displaced each being estimated by varying the displacement amount or amounts until the respective high-definition light field hogel data dependent metric for the respective compression mode reaches a minimum value

12. The method of claim 11 wherein for the compression mode having the high-definition light field hogel data dependent metric referred to as SAD_{ODD HOGEL}, the packing is a packing of the mode Mode_{ODD HOGEL}, the disparity or disparities equal to the displacement or displacements of the anglet of the odd hogel, and the intensity of the anglet of the odd hogel, and for the high-definition light field hogel data dependent metric referred to as SAD_{EVEN HOGEL}, the packing is a packing of the mode Mode_{EVEN HOGEL}, the disparity or disparities equal to the displacement or displacements of the anglets of the odd hogel, and intensity of the anglets of the even hogel.

13. The method of claim 1 wherein the method is applied to segments of a hogel.

14. The method of claim 1 wherein a fixed disparity value is applied to a block of anglets.

15. The method of claim 1 further comprising decoding and reconstruction of a frame of high-definition light field hogel data compressed by:

a) unpacking the packed compression mode, disparity values and compressed high-definition light field hogel data to extract the packed compression mode, the disparity values and the compressed high-definition light field hogel data;

b) reconstructing the high-definition light field hogel data from the compressed high-definition light field hogel data using the compression mode to select a reconstruction method from a plurality of reconstruction methods, each reconstruction method corresponding to a respective one of the plurality of compression methods.

16. The method of claim 15 wherein the high-definition light field hogel data is anglet data, and wherein the reconstructing of the high-definition light field hogel data from the compressed high-definition light field hogel data uses per anglet disparity values.

17. The method of claim 1 further comprising:

selecting hogels to be compressed and disparity values to be used for reconstruction using a visibility test;

compressing the selected hogels using the compression method of claim 1;

packing the compression mode and disparity values with the compressed high-definition light field hogel data for further transmission.

18. The method of claim 17, further comprising extracting a bounding box of objects in the scene.

19. The method of claim 18, wherein a hogel that is selected is a hogel of a pair of hogels with the most anglets hitting an aligned surface of the bounding box.

20. The method of claim 18, further comprising calculating the disparity values from the depth of the bounding box or from the most representative depth value of the compressed hogel.