Method and apparatus for block-based compression of light-field images

• US 8,155,456 B2
• Filed: 04/29/2008
• Issued: 04/10/2012
• Est. Priority Date: 04/29/2008
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Babacan, Sevket Derin, Georgiev, Todor G.
• Primary Examiner(s): Liu, Li
• Application Number:
  US12/111,735
• Publication Number:
  US 20090268970A1
• Time in Patent Office: 1,442 Days
• Field of Search: None
• US Class Current: 382/232
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/60 using transform coding
• CPC Class Codes: H04N 19/90 using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 5/91 Television signal processing therefor
• CPC Class Codes: H04N 9/8042 involving data reduction

Abstract

A method and apparatus for the block-based compression of light-field images. Light-field images may be preprocessed by a preprocessing module into a format that is compatible with the blocking scheme of a block-based compression technique, for example JPEG. The compression technique is then used to compress the preprocessed light-field images. The light-field preprocessing module reshapes the angular data in a captured light-field image into shapes compatible with the blocking scheme of the compression technique so that blocking artifacts of block-based compression are not introduced in the final compressed image. Embodiments may produce compressed 2D images for which no specific light-field image viewer is needed to preview the full light-field image. Full light-field information is contained in one compressed 2D image.

Claims

1. A computer-implemented method, comprising:

obtaining a light-field image, wherein the light-field image comprises a plurality of microimages, wherein the dimensions of the microimages are different than blocking dimensions used by a block-based compression technique;

generating a plurality of blocks in a preprocessed light-field image from the plurality of microimages in the light-field image, wherein each block in the preprocessed light-field image contains light-field information from at least one microimage in the light-field image, and wherein the dimensions of each block in the preprocessed light-field image are the same as the blocking dimensions used by the block-based compression technique;

wherein, to generate a block in the preprocessed light-field image, the method further comprises applying a cropping technique to light-field information from at least one microimage, applying an interpolation technique to light-field information from at least one microimage, or applying a cropping technique followed by an interpolation technique to light-field information from at least one microimage;

compressing the preprocessed light-field image in accordance with the block-based compression technique to generate a compressed light-field image; and

storing the compressed light-field image to a storage medium.

2. The method as recited in claim 1, wherein said generating a plurality of blocks in a preprocessed light-field image from the microimages in the light-field image comprises, for each microimage:

cropping the microimage to extract a region of pixels from the microimage, wherein the dimensions of the cropped region are smaller than the dimensions of the block in the preprocessed light-field image; and

interpolating the cropped region of pixels to generate pixel values for the pixels in a corresponding block in the preprocessed light-field image from the pixel values of the pixels in the cropped region.

3. The method as recited in claim 2, wherein said interpolating the cropped region of pixels comprises solving the Laplace equation to generate the pixel values for the pixels in the corresponding block from the pixel values of the pixels in the cropped region.

4. The method as recited in claim 1, wherein the dimensions of the blocks in the preprocessed light-field image are eight pixels by eight pixels.

5. The method as recited in claim 1, further comprising, prior to said generating a plurality of blocks in a preprocessed light-field image from the plurality of microimages in the light-field image, normalizing the light-field image in accordance with a calibration image.

6. The method as recited in claim 1, wherein the light-field image is captured with a plenoptic camera, wherein each microimage is approximately circular, and wherein each block in the preprocessed light-field image contains light-field information from a single corresponding microimage in the light-field image.

7. The method as recited in claim 6, further comprising, prior to said generating a plurality of blocks in a preprocessed light-field image from each microimage the microimages in the light-field image:

locating each microimage in the light-field image; and

extracting the located microimages from the light-field image.

8. The method as recited in claim 1, wherein said generating a plurality of blocks in a preprocessed light-field image from the microimages in the light-field image comprises:

generating an intermediate image comprising blocks from the microimages in the light-field image, wherein said generating the intermediate image comprises, for each pixel location in each microimage, extracting a pixel from the pixel location and placing the pixel in a block of the intermediate image at a pixel location in the block corresponding to the current microimage'"'"'s location in the light-field image, wherein the dimensions of the blocks in the intermediate image are different than the blocking dimensions used by the block-based compression technique; and

generating the plurality of blocks in the preprocessed light-field image from the blocks in the intermediate image, wherein each block in the preprocessed light-field image contains light-field information from a corresponding block in the intermediate image, and wherein the dimensions of each block in the preprocessed light-field image are the same as the blocking dimensions used by the block-based compression technique.

9. The method as recited in claim 1, further comprising displaying the compressed light-field image using an application that is configured to open and display files compressed using the block-based compression technique.

10. A system, comprising:

at least one processor; and

a memory comprising program instructions, wherein the program instructions are executable by the at least one processor to;

obtain a light-field image, wherein the light-field image comprises multiple a plurality of microimages, wherein the dimensions of the microimages are different than blocking dimensions used by a block-based compression technique;

generate a plurality of blocks in a preprocessed light-field image from the plurality of microimages in the light-field image, wherein each block contains light-field information from at least one microimage in the light-field image, and wherein the dimensions of each block n the preprocessed light-field image are the same as the blocking dimensions used by the block-based compression technique;

wherein, to generate a block in the preprocessed light-field image, the program instructions are executable by the at least one processor to apply a cropping technique to light-field information from at least one microimage, apply an interpolation technique to light-field information from at least one microimage, or apply a cropping technique followed by an interpolation technique to light-field information from at least one microimage;

compress the preprocessed light-field image in accordance with the block-based compression technique to generate a compressed light-field image; and

store the compressed light-field image to a storage device.

11. The system as recited in claim 10, wherein, to generate a plurality of blocks in a preprocessed light-field image from each the microimages in the light-field image, the program instructions are executable by the at least one processor to, for each microimage:

crop the microimage to extract a region of pixels from the microimage, wherein the dimensions of the cropped region are smaller than the dimensions of the block in the preprocessed light-field image; and

interpolate the cropped region of pixels to generate pixel values for the pixels in a corresponding block in the preprocessed light-field image from the pixel values of the pixels in the cropped region.

12. The system as recited in claim 11, wherein, to interpolate the cropped region of pixels, the program instructions are executable by the at least one processor to solve the Laplace equation to generate the pixel values for the pixels in the corresponding block from the pixel values of the pixels in the cropped region.

13. The system as recited in claim 10, wherein the dimensions of the blocks in the preprocessed light-field image are eight pixels by eight pixels.

14. The system as recited in claim 10, wherein the program instructions are executable by the at least one processor to, prior to said generate a plurality of blocks in a preprocessed light-field image from the plurality of microimages in the light-field image, normalize the light-field image in accordance with a calibration image for the light-field camera.

15. The system as recited in claim 10, wherein the light-field image is captured with a plenoptic camera, wherein each microimage is approximately circular, and wherein each block in the preprocessed light-field image contains light-field information from a single corresponding microimage in the light-field image.

16. The system as recited in claim 15, wherein the program instructions are executable by the at least one processor to, prior to said generate a plurality of blocks in a preprocessed light-field image from the microimages in the light-field image:

locate each microimage in the light-field image; and

extract the located microimages from the light-field image.

17. The system as recited in claim 10, wherein, to generate a plurality of blocks in a preprocessed light-field image from the microimages in the light-field image, the program instructions are executable by the at least one processor to:

generate an intermediate image comprising blocks from the microimages in the light-field image, wherein, to generate the intermediate image, the program instructions are executable by the at least one processor to, for each pixel location in each microimage, extract a pixel from the pixel location and place the pixel in a block of the intermediate image at a pixel location in the block corresponding to the current microimage'"'"'s location in the light-field image, wherein the dimensions of the blocks in the intermediate image are different than the blocking dimensions used by the block-based compression technique; and

generate the plurality of blocks in the preprocessed light-field image from the blocks in the intermediate image, wherein each block in the preprocessed light-field image contains light-field information from a corresponding block in the intermediate image, and wherein the dimensions of each block in the preprocessed light-field image are the same as the blocking dimensions used by the block-based compression technique.

18. A non-transitory computer-readable storage medium storing program instructions, wherein the program instructions are computer-executable to implement:

obtaining a light-field image, wherein the light-field image comprises a plurality of microimages, wherein the dimensions of the microimages are different than blocking dimensions used by a block-based compression technique;

generating a plurality of blocks in a preprocessed light-field image from the plurality of microimages in the light-field image, wherein each block contains light-field information from at least one microimage in the light-field image, and wherein the dimensions of each block in the preprocessed light-field image are the same as the blocking dimensions used by the block-based compression technique;

wherein, to generate a block in the preprocessed light-field image, the program instructions are computer-executable to implement applying a cropping technique to light-field information from at least one microimage, applying an interpolation technique to light-field information from at least one microimage, or applying a cropping technique followed by an interpolation technique to light-field information from at least one microimage;

compressing the preprocessed light-field image in accordance with the block-based compression technique to generate a compressed light-field image; and

storing the compressed light-field image to a storage medium.

19. The non-transitory computer-readable storage medium as recited in claim 18, wherein, in said generating a plurality of blocks in a preprocessed light-field image from the microimages in the light-field image, the program instructions are computer-executable to implement, for each microimage:

cropping the microimage to extract a region of pixels from the microimage, wherein the dimensions of the cropped region are smaller than the dimensions of the block in the preprocessed light-field image; and

interpolating the cropped region of pixels to generate pixel values for the pixels in a corresponding block in the preprocessed light-field image from the pixel values of the pixels in the cropped region.

20. The non-transitory computer-readable storage medium as recited in claim 19, wherein, in said interpolating the cropped region of pixels, the program instructions are computer-executable to implement solving the Laplace equation to generate the pixel values for the pixels in the corresponding block from the pixel values of the pixels in the cropped region.

21. The non-transitory computer-readable storage medium as recited in claim 18, wherein the dimensions of the blocks in the preprocessed light-field image are eight pixels by eight pixels.

22. The non-transitory computer-readable storage medium as recited in claim 18, wherein the program instructions are computer-executable to implement, prior to said generating a plurality of blocks in a preprocessed light-field image from the plurality of microimages in the light-field image, normalizing the light-field image in accordance with a calibration image for the light-field camera.

23. The non-transitory computer-readable storage medium as recited in claim 18, wherein the light-field image is captured with a plenoptic camera, wherein each microimage is approximately circular, and wherein each block in the preprocessed light-field image contains light-field information from a single corresponding microimage in the light-field image.

24. The non-transitory computer-readable storage medium as recited in claim 23, wherein the program instructions are computer-executable to implement, prior to said generating a plurality of blocks in a preprocessed light-field image from the microimages in the light-field image:

locating each microimage in the light-field image; and

extracting the located microimages from the light-field image.

25. The non-transitory computer-readable storage medium as recited in claim 18, wherein, in said generating a plurality of blocks in a preprocessed light-field image from the microimages in the light-field image, the program instructions are computer-executable to implement:

generating an intermediate image comprising blocks from the microimages in the light-field image, wherein, in said generating the intermediate image, the program instructions are computer-executable to implement, for each pixel location in each microimage, extracting a pixel from the pixel location and placing the pixel in a block of the intermediate image at a pixel location in the block corresponding to the current microimage'"'"'s location in the light-field image, wherein the dimensions of the blocks in the intermediate image are different than the blocking dimensions used by the block-based compression technique; and

generating the plurality of blocks in the preprocessed light-field image from the blocks in the intermediate image, wherein each block in the preprocessed light-field image contains light-field information from a corresponding block in the intermediate image, and wherein the dimensions of each block in the preprocessed light-field image are the same as the blocking dimensions used by the block-based compression technique.