Systems and methods for decoding image files containing depth maps stored as metadata

• US 10,430,682 B2
• Filed: 07/09/2018
• Issued: 10/01/2019
• Est. Priority Date: 09/28/2011
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: Venkataraman, Kartik, Nisenzon, Semyon, Lelescu, Dan
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US16/030,597
• Publication Number:
  US 20180330182A1
• Time in Patent Office: 449 Days
• CPC Class Codes: G06T 15/08 Volume rendering
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/596 from three or more stereo images
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00
• CPC Class Codes: G06T 9/20 Contour coding, e.g. using detection of edges
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/178 Metadata, e.g. disparity information
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 19/593 involving spatial prediction techniques
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/625 using discrete cosine transform [DCT]
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 2013/0081 : Depth or disparity estimation from stereoscopic image signals

Abstract

Systems and methods in accordance with embodiments of the invention are configured to render images using light field image files containing an image synthesized from light field image data and metadata describing the image that includes a depth map. One embodiment of the invention includes a processor and memory containing a rendering application and a light field image file including an encoded image, a set of low resolution images, and metadata describing the encoded image, where the metadata comprises a depth map that specifies depths from the reference viewpoint for pixels in the encoded image. In addition, the rendering application configures the processor to: locate the encoded image within the light field image file; decode the encoded image; locate the metadata within the light field image file; and post process the decoded image by modifying the pixels based on the depths indicated within the depth map and the set of low resolution images to create a rendered image.

Claims

1. A system for rendering an image using an image container file, the system comprising:

a processor; and

memory containing a rendering application and an image container file, wherein the image container file comprises;

an encoded image synthesized from a plurality of images captured by a plurality of heterogeneous cameras, wherein each camera of the plurality of heterogeneous cameras captures the scene from a different viewpoint;

a depth map that specifies depths from a reference viewpoint for pixels in the encoded image based on disparity between pixels of the plurality of images;

an auxiliary map that provides information corresponding to pixel locations within a synthesized image; and

metadata describing the image container file, wherein the metadata comprises offset information to locate the encoded image, the depth map, and the auxiliary map;

wherein the rendering application configures the processor to;

locate the encoded image within the image container file;

decode the encoded image;

locate the depth map and the auxiliary map within the image container file; and

post process the decoded image to apply a depth based effect to the pixels of the decoded image based on the depth map, the auxiliary map, and the metadata to create a rendered image.

2. The system of claim 1, wherein the image container file further comprises at least one of the plurality of images captured by the plurality of heterogeneous cameras.

3. The system of claim 2, wherein post processing the decoded image comprises interpolating to fill in occluded pixels based on adjacent pixels that are not occluded.

4. The system of claim 1, wherein the metadata further comprises occluded pixel data for pixels that are occluded.

5. The system of claim 1, wherein the encoded image is focused at a first focal plane, wherein post processing comprises modifying the pixels so that the rendered image is focused at a different, second focal plane.

6. The system of claim 1, wherein the depth based effect simulates motion parallax in the decoded image.

7. The system of claim 1, wherein the depth based effect simulates a dolly zoom in the decoded image.

8. The system of claim 1, wherein at least one of the plurality of images have a lower resolution than the encoded image.

9. The system of claim 1, wherein the encoded image is synthesized from a first image from a first viewpoint and a second image from a second viewpoint, wherein the encoded image is synthesized from a different, third viewpoint.

10. The system of claim 1, wherein at least one of the depth map and the auxiliary map are encoded as an image.

11. A method for rendering an image using an image container file, the method comprising:

locating the encoded image within an image container file, wherein the image container file comprises;

an encoded image synthesized from a plurality of images captured by a plurality of heterogeneous cameras, wherein each camera of the plurality of heterogeneous cameras captures the scene from a different viewpoint;

a depth map that specifies depths from a reference viewpoint for pixels in the encoded image based on disparity between pixels of the plurality of images;

an auxiliary map that provides information corresponding to pixel locations within a synthesized image; and

metadata describing the image container file, wherein the metadata comprises offset information to locate the encoded image, the depth map, and the auxiliary map;

decoding the encoded image;

locating the depth map and the auxiliary map within the image container file; and

post processing the decoded image to apply a depth based effect to the pixels of the decoded image based on the depth map, the auxiliary map, and the metadata to create a rendered image.

12. The method of claim 11, wherein the image container file further comprises at least one of the plurality of images captured by the plurality of heterogeneous cameras.

13. The method of claim 12, wherein post processing the decoded image comprises interpolating to fill in occluded pixels based on adjacent pixels that are not occluded.

14. The method of claim 11, wherein the metadata further comprises occluded pixel data for pixels that are occluded.

15. The method of claim 11, wherein the encoded image is focused at a first focal plane, wherein post processing comprises modifying the pixels so that the rendered image is focused at a different, second focal plane.

16. The method of claim 11, wherein the depth based effect simulates motion parallax in the decoded image.

17. The method of claim 11, wherein the depth based effect simulates a dolly zoom in the decoded image.

18. The method of claim 11, wherein at least one of the plurality of images have a lower resolution than the encoded image.

19. The method of claim 11, wherein the encoded image is synthesized from a first image from a first viewpoint and a second image from a second viewpoint, wherein the encoded image is synthesized from a different, third viewpoint.

20. The method of claim 11, wherein at least one of the depth map and the auxiliary map are encoded as an image.