Systems and methods for decoding light field image files

• US 8,542,933 B2
• Filed: 09/28/2012
• Issued: 09/24/2013
• Est. Priority Date: 09/28/2011
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: Pelican Imaging Corporation
• Inventors: Venkataraman, Kartik, Nisenzon, Semyon
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US13/631,736
• Publication Number:
  US 20130077882A1
• Time in Patent Office: 361 Days
• Field of Search: 382/154, 382/232, 382/233, 382244-246, 382/275, 382/276, 382/300, 348 42- 60, 348/349, 348/350, 348/E13.075, 359376-378, 359/458, 359462-477, 345/419, 345/501, 353/609, 356 12- 14
• US Class Current: 382/233
• CPC Class Codes: G06T 15/08 Volume rendering
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/596 from three or more stereo images
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00
• CPC Class Codes: G06T 9/20 Contour coding, e.g. using detection of edges
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/178 Metadata, e.g. disparity information
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 19/593 involving spatial prediction techniques
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/625 using discrete cosine transform [DCT]
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 2013/0081 : Depth or disparity estimation from stereoscopic image signals

Abstract

Systems and methods in accordance with embodiments of the invention are configured to render images using light field image files containing an image synthesized from light field image data and metadata describing the image that includes a depth map. One embodiment of the invention includes a processor and memory containing a rendering application and a light field image file including an encoded image and metadata describing the encoded image, where the metadata comprises a depth map that specifies depths from the reference viewpoint for pixels in the encoded image. In addition, the rendering application configures the processor to: locate the encoded image within the light field image file; decode the encoded image; locate the metadata within the light field image file; and post process the decoded image by modifying the pixels based on the depths indicated within the depth map to create a rendered image.

Claims

1. A system for rendering an image using a light field image file including an encoded image and metadata describing the encoded image, where the metadata comprises a depth map that specifies depths from the reference viewpoint for pixels in the encoded image, the system comprising:

a processor; and

memory containing a rendering application and a light field image file including an encoded image and metadata describing the encoded image, where the encoded image is an image of a scene synthesized from a reference viewpoint using a plurality of lower resolution images that capture the scene from different viewpoints and the metadata comprises a depth map that specifies depths from the reference viewpoint for pixels in the encoded image, and pixels from the lower resolution images that are occluded in the reference viewpoint;

wherein the rendering application configures the processor to;

locate the encoded image within the light field image file;

decode the encoded image;

locate the metadata within the light field image file; and

post process the decoded image by modifying the pixels based on the depths indicated within the depth map to create a rendered image by rendering an image from a different viewpoint using the depth map and the pixels from the lower resolution images that are occluded in the reference viewpoint.

2. The system of claim 1, wherein the rendering application configuring the processor to post process the decoded image by modifying the pixels based on the depths indicated within the depth map to create the rendered image comprises applying a depth based effect to the pixels of the decoded image.

3. The system of claim 2, wherein the depth based effect comprises at least one effect selected from the group consisting of:

modifying the focal plane of the decoded image;

modifying the depth of field of the decoded image;

modifying the blur in out-of-focus regions of the decoded image;

locally varying the depth of field of the decoded image;

creating multiple focus areas at different depths within the decoded image; and

applying a depth related blur.

4. The system of claim 2, wherein the metadata is located within an Application marker segment within the light field image file.

5. The system of claim 4, wherein the Application marker segment is identified using the APP9 marker.

6. The system of claim 4, wherein:

the depth map is encoded in accordance with the JPEG standard using lossless compression; and

the rendering application configures the processor to;

locate at least one Application marker segment containing the metadata comprising the depth map; and

decode the depth map using the JPEG decoder.

7. The system of claim 4, wherein:

the metadata in the light field image file includes descriptions of the pixels from the lower resolution images that are occluded in the reference viewpoint including the color, location, and depth of the occluded pixels; and

rendering an image from a different viewpoint using the depth map and the pixels from the lower resolution images that are occluded in the reference viewpoint further comprises;

shifting pixels from the decoded image and the occluded pixels in the metadata to the different viewpoint based upon the depths of the pixels;

determining pixel occlusions; and

generating an image from the different viewpoint using the shifted pixels that are not occluded and by interpolating to fill in missing pixels using adjacent pixels that are not occluded.

8. The system of claim 4, wherein the image rendered from the different viewpoint is part of a stereo pair of images.

9. The system of claim 4, wherein:

the metadata in the light field image file further comprises a confidence map for the depth map, where the confidence map indicates the reliability of the depth values provided for pixels by the depth map; and

rendering an image from a different viewpoint using the depth map and the pixels from the lower resolution images that are occluded in the reference viewpoint further comprises applying at least one filter based upon the confidence map.

10. The system of claim 4, wherein:

the metadata in the light field image file further comprises an edge map that indicates pixels in the decoded image that lie on a discontinuity; and

rendering an image from a different viewpoint using the depth map and the pixels from the lower resolution images that are occluded in the reference viewpoint further comprises applying at least one filter based upon the edge map.

11. The system of claim 10, wherein the edge map identifies whether a pixel lies on an intensity discontinuity.

12. The system of claim 10, wherein the edge map identifies whether a pixel lies on an intensity and depth discontinuity.

13. The system of claim 10, wherein:

the edge map is encoded in accordance with the JPEG standard using lossless compression; and

the rendering application configures the processor to;

locate at least one Application marker segment containing the metadata comprising the edge map; and

decode the edge map using the JPEG decoder.

14. The system of claim 4, wherein:

the metadata in the light field image file further comprises a missing pixel map that indicates pixels in the decoded image that do not correspond to a pixel from the plurality of low resolution images of the scene and that are generated by interpolating pixel values from adjacent pixels in the synthesized image; and

rendering an image from a different viewpoint using the depth map and the pixels from the lower resolution images that are occluded in the reference viewpoint further comprises ignoring pixels based upon the missing pixel map.

15. The system of claim 14, wherein:

the missing pixel map is encoded in accordance with the JPEG standard using lossless compression; and

the rendering application configures the processor to;

locate at least one Application marker segment containing the metadata comprising the missing pixel; and

decode the missing pixel map using the JPEG decoder.

20. The system of claim 1, wherein:

the metadata in the light field image file includes descriptions of the pixels from the lower resolution images that are occluded in the reference viewpoint including the color, location, and depth of the occluded pixels; and

rendering an image from a different viewpoint using the depth map and the pixels from the lower resolution images that are occluded in the reference viewpoint further comprises;

shifting pixels from the decoded image and the occluded pixels in the metadata to the different viewpoint based upon the depths of the pixels;

determining pixel occlusions; and

generating an image from the different viewpoint using the shifted pixels that are not occluded and by interpolating to fill in missing pixels using adjacent pixels that are not occluded.

21. The system of claim 1, wherein the image rendered from the different viewpoint is part of a stereo pair of images.

22. The system of claim 1, wherein:

the metadata in the light field image file further comprises a confidence map for the depth map, where the confidence map indicates the reliability of the depth values provided for pixels by the depth map; and

rendering an image from a different viewpoint using the depth map and the pixels from the lower resolution images that are occluded in the reference viewpoint further comprises applying at least one filter based upon the confidence map.

23. The system of claim 1, wherein:

the metadata in the light field image file further comprises an edge map that indicates pixels in the decoded image that lie on a discontinuity; and

rendering an image from a different viewpoint using the depth map and the pixels from the lower resolution images that are occluded in the reference viewpoint further comprises applying at least one filter based upon the edge map.

24. The system of claim 23, wherein the edge map identifies whether a pixel lies on an intensity discontinuity.

25. The system of claim 23, wherein the edge map identifies whether a pixel lies on an intensity and depth discontinuity.

26. The system of claim 1, wherein:

the metadata in the light field image file further comprises a missing pixel map that indicates pixels in the decoded image that do not correspond to a pixel from the plurality of low resolution images of the scene and that are generated by interpolating pixel values from adjacent pixels in the synthesized image; and

rendering an image from a different viewpoint using the depth map and the pixels from the lower resolution images that are occluded in the reference viewpoint further comprises ignoring pixels based upon the missing pixel map.

27. The system of claim 1, wherein:

the light field image file conforms to the JFIF standard and the encoded image is encoded in accordance with the JPEG standard;

the memory comprises a JPEG decoder application; and

the rendering application configures the processor to;

locate the encoded image by locating a Start of Image marker within the light field image file; and

decode the encoded image using the JPEG decoder.

16. A method of for rendering an image using a light field image file including an encoded image and metadata describing the encoded image, where the encoded image is an image of a scene synthesized from a reference viewpoint using a plurality of lower resolution images that capture the scene from different viewpoints and the metadata comprises a depth map that specifies depths from the reference viewpoint for pixels in the encoded image, and pixels from the lower resolution images that are occluded in the reference viewpoint, the method comprising:

locating said encoded image within said light field image file using a rendering device;

decoding the encoded image using the rendering device;

locating the metadata within the light field image file using the rendering device; and

post processing the decoded image by modifying the pixels based on the depths indicated within the depth map and the pixels from the lower resolution images that are occluded in the reference viewpoint to create a rendered image using the rendering.

17. The method of claim 16, wherein post processing the decoded image by modifying the pixels based on the depths indicated within the depth map to create the rendered image comprises applying a depth based effect to the pixels of the decoded image using the rending device.

18. The method of claim 17, wherein the depth based effect comprises at least one effect selected from the group consisting of:

modifying the focal plane of the decoded image using the rendering device;

modifying the depth of field of the decoded image using the rendering device;

modifying the blur in out-of-focus regions of the decoded image using the rendering device;

locally varying the depth of field of the decoded image using the rendering device;

creating multiple focus areas at different depths within the decoded image using the rendering device; and

applying a depth related blur using the rendering device.

19. A non-transitory machine readable medium containing processor instructions, where execution of the instructions by a processor causes the processor to perform a process comprising:

locating an encoded image within a light field image file, where the light field image file includes an encoded image that is an image of a scene synthesized from a reference viewpoint using a plurality of lower resolution images that capture the scene from different viewpoints and metadata describing the encoded image comprising a depth map that specifies depths from the reference viewpoint for pixels in the encoded image, and pixels from the lower resolution images that are occluded in the reference viewpoint;

decoding the encoded image;

locating the metadata within the light field image file; and

post processing the decoded image by modifying the pixels based on the depths indicated within the depth map and the pixels from the lower resolution images that are occluded in the reference viewpoint to create a rendered image.