Systems and methods for decoding image files containing depth maps stored as metadata

• US 10,019,816 B2
• Filed: 12/30/2016
• Issued: 07/10/2018
• Est. Priority Date: 09/28/2011
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: Fotonation Cayman Limited (Adeia Inc.)
• Inventors: Venkataraman, Kartik, Nisenzon, Semyon, Lelescu, Dan
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US15/396,024
• Publication Number:
  US 20170178363A1
• Time in Patent Office: 557 Days
• CPC Class Codes: G06T 15/08 Volume rendering
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/596 from three or more stereo images
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00
• CPC Class Codes: G06T 9/20 Contour coding, e.g. using detection of edges
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/178 Metadata, e.g. disparity information
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 19/593 involving spatial prediction techniques
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/625 using discrete cosine transform [DCT]
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 2013/0081 : Depth or disparity estimation from stereoscopic image signals

Abstract

Systems and methods in accordance with embodiments of the invention are configured to render images using light field image files containing an image synthesized from light field image data and metadata describing the image that includes a depth map. One embodiment of the invention includes a processor and memory containing a rendering application and a light field image file including an encoded image, a set of low resolution images, and metadata describing the encoded image, where the metadata comprises a depth map that specifies depths from the reference viewpoint for pixels in the encoded image. In addition, the rendering application configures the processor to: locate the encoded image within the light field image file; decode the encoded image; locate the metadata within the light field image file; and post process the decoded image by modifying the pixels based on the depths indicated within the depth map and the set of low resolution images to create a rendered image.

Claims

1. A system for rendering image files containing depth maps, the system comprising:

a display;

a processor; and

a memory containing a rendering application and an image file comprising;

a reference encoded image from a reference viewpoint; and

metadata describing at least one supplementary encoded image taken from a supplementary viewpoint using at least one supplementary camera stored as metadata within the image file; and

metadata describing a depth map based on the parallax variation between the encoded reference image and the at least one encoded supplementary image that specifies depths from the reference viewpoint for pixels in the encoded reference image;

wherein the rendering application directs the processor to;

locate the reference image within the image file;

synthesize an image based on the reference image and the depth map;

render the synthesized image; and

display the synthesized image using the display.

2. The system of claim 1, wherein:

the rendering application further configures the processor to synthesize a virtual viewpoint using the reference viewpoint and the at least one supplementary viewpoint; and

the synthesized image is further based on the virtual viewpoint such that the displayed synthesized image appears to have been taken from the virtual viewpoint.

3. The system of claim 1, wherein the metadata is located within an Application marker segment within the image file.

4. The system of claim 3, wherein the Application marker segment is identified using the APP9 marker.

5. The system of claim 3, wherein the depth map is encoded in accordance with the JPEG standard using lossless compression.

6. The system of claim 1, wherein the metadata further comprises pixels that are occluded by the reference viewpoint.

7. The system of claim 1, wherein the metadata further describes an Auxiliary Map Data.

8. The system of claim 1, wherein the metadata further describes a confidence map.

9. The system of claim 1, wherein the metadata further describes an edge map.

10. The system of claim 1, wherein the metadata further describes a missing pixel map.

11. The system of claim 1, wherein the rendering application further directs the processor to determine the boundary between foreground objects and a background in the reference image using the depth map.

12. The system of claim 11, wherein the rendering application further directs the processor to apply a blur effect to the determined background in the reference image.

13. The system of claim 1, wherein the reference encoded image is taken from the reference viewpoint using a reference camera.

14. The system of claim 13, wherein the reference image is stored in metadata within the image file.

15. The system of claim 1, wherein the rendering application further directs the processor to store the synthesized image in the image file.

16. The system of claim 1, wherein:

the reference encoded image is taken from a reference viewpoint using a reference camera;

the reference image is stored in metadata within the image file; and

the rendering application further directs the processor to store the synthesized image in the image file.

17. A system for storing image files containing depth maps, the system comprising:

a display;

a processor;

an array camera comprising a plurality of cameras; and

a memory containing a rendering application and an image file comprising;

a stereo pair of encoded images comprising;

a reference encoded image from a reference viewpoint; and

a supplementary encoded image from a supplementary viewpoint;

wherein the supplementary encoded image is taken by a camera in the plurality of cameras; and

wherein the supplementary encoded image is stored as metadata within the image file; and

metadata describing a depth map based on the parallax variation between the reference encoded image and the at least one supplementary encoded image that specifies depths from the reference viewpoint for pixels in the encoded reference image;

wherein the rendering application directs the processor to;

locate the reference encoded image within the image file;

synthesize an image based on the reference encoded image and the depth map;

render the synthesized image; and

display the synthesized image using the display.

18. The system of claim 17, wherein the reference encoded image is taken by a camera in the plurality of cameras from the reference viewpoint.

19. The system of claim 17, wherein the reference encoded image is in the RGB domain, and the rendering application further directs the processor to convert the supplementary encoded image to the YCbCr domain.

20. The system of claim 17, wherein a first camera in the plurality of cameras has a different field of view from a second camera in the plurality of cameras.

21. The system of claim 17, wherein the rendering application further directs the processor to store the synthesized image in the image file.

22. The system of claim 17, wherein the reference encoded image is taken from a reference viewpoint using a reference camera.

23. The system of claim 17, wherein the reference image is stored in metadata within the image file.

24. The system of claim 17, wherein the rendering application further directs the processor to store the synthesized image in the image file.

25. The system of claim 17, wherein:

the reference encoded image is taken from a reference viewpoint using a reference camera;

the reference image is stored in metadata within the image file; and

the rendering application further directs the processor to store the synthesized image in the image file.