Systems and methods for encoding image files containing depth maps stored as metadata

• US 10,275,676 B2
• Filed: 01/08/2018
• Issued: 04/30/2019
• Est. Priority Date: 09/28/2011
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: Venkataraman, Kartik, Nisenzon, Semyon, Lelescu, Dan
• Primary Examiner(s): Dang, Duy M
• Application Number:
  US15/865,208
• Publication Number:
  US 20180197035A1
• Time in Patent Office: 477 Days
• CPC Class Codes: G06T 15/08 Volume rendering
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/596 from three or more stereo images
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00
• CPC Class Codes: G06T 9/20 Contour coding, e.g. using detection of edges
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/178 Metadata, e.g. disparity information
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 19/593 involving spatial prediction techniques
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/625 using discrete cosine transform [DCT]
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 2013/0081 : Depth or disparity estimation from stereoscopic image signals

Abstract

Systems and methods for storing images synthesized from light field image data and metadata describing the images in electronic files in accordance with embodiments of the invention are disclosed. One embodiment includes a processor and memory containing an encoding application and light field image data, where the light field image data comprises a plurality of low resolution images of a scene captured from different viewpoints. In addition, the encoding application configures the processor to synthesize a higher resolution image of the scene from a reference viewpoint using the low resolution images, where synthesizing the higher resolution image involves creating a depth map that specifies depths from the reference viewpoint for pixels in the higher resolution image; encode the higher resolution image; and create a light field image file including the encoded image, the low resolution images, and metadata including the depth map.

Claims

1. A system for generating modifiable blur effects, comprising:

a processor;

an array of cameras comprising a plurality of cameras;

a display; and

a memory containing machine readable instructions;

wherein the machine readable instructions direct the processor to;

obtain image data using the array of cameras, where the image data comprises a plurality of images of a scene captured from different viewpoints that includes a reference image captured from a reference viewpoint;

create a depth map that specifies depths for pixels in the reference image using at least a portion of the image data;

apply a depth based blur effect during synthesis of an image using the reference image and the depth map;

encode the synthesized image and the reference image;

write the encoded images and the depth map to an image file;

store the image file in the memory;

retrieve the image from the memory;

decode the synthesized image;

display the synthesized image using the display;

create a modified synthesized image by applying a second depth based blur effect using the depth map;

display the modified synthesized image using the display;

encode the modified synthesized image; and

store the modified synthesized image in the image file.

2. The system of claim 1, wherein the depth map is written to the image file as metadata.

3. The system of claim 2, wherein the metadata is located within an Application marker segment within the image file.

4. The system of claim 2, wherein the machine readable instructions direct the processor to encode the depth map.

5. The system of claim 1, wherein the machine readable instructions further direct the processor to:

generate a confidence map for the depth map, where the confidence map indicates the reliability of at least one depth value for at least one pixel in the depth map; and

write the confidence map to the image file as metadata.

6. The system of claim 1, wherein the machine readable instructions further direct the processor to utilize the reference image to create the modified synthesized image.

7. The system of claim 1, wherein the encoded reference image is written to the image file as metadata.

8. The system of claim 1, wherein the machine readable instructions further direct the processor to:

encode the images in the plurality of images; and

write the encoded images in the plurality of images to the image file as metadata.

9. The system of claim 1, wherein:

at least one camera in the plurality of cameras is configured to capture image data using RGB color channels; and

the machine readable instructions further direct the processor to convert the image data captured in the RGB color channels into a YCbCr format.

10. The system of claim 1, wherein the plurality of images of a scene captured from different viewpoints comprises a stereo pair of images.

11. A method for generating modifiable blur effects, comprising:

obtaining image data using an array of cameras comprising a plurality of cameras, where the image data comprises a plurality of images of a scene captured from different viewpoints that includes a reference image captured from a reference viewpoint;

creating a depth map that specifies depths for pixels in the reference image using a processor configured by machine readable instructions based upon at least a portion of the image data;

applying a depth based blur effect during synthesis of an image using the processor configured by machine readable instructions based upon the reference image and the depth map;

encoding the synthesized image and the reference image using the processor configured by machine readable instructions;

writing the encoded images and the depth map to an image file using the processor configured by machine readable instructions;

storing the image file to a memory using the processor configured by machine readable instructions;

retrieving the image from the memory using the processor configured by machine readable instructions;

decoding the synthesized image using the processor configured by machine readable instructions;

displaying the synthesized image using a display using the processor configured by machine readable instructions;

creating a modified synthesized image by applying a second depth based blur effect using using the processor configured by machine readable instructions based upon the depth map;

displaying the modified synthesized image using the display;

encoding the modified synthesized image using the processor configured by machine readable instructions; and

storing the modified synthesized image in the image file using the processor configured by machine readable instructions.

12. The method of claim 11, wherein the metadata is located within an Application marker segment within the image file.

13. The method of claim 11, wherein the depth map is written to the image file as metadata.

14. The method of claim 13 further comprising encoding the depth map using the processor configured by machine readable instructions.

15. The method of claim 11, wherein generating the depth map further comprises:

generating a confidence map for the depth map using the processor configured by machine readable instructions, where the confidence map indicates the reliability of at least one depth value for at least one pixel in the depth map; and

writing the confidence map to the image file as metadata using the processor configured by machine readable instructions.

16. The method of claim 11, wherein creating the modified synthesized image further comprises utilizing the reference image.

17. The method of claim 11, wherein the encoded reference image is written to the image file as metadata.

18. The method of claim 11, further comprising:

encoding the images in the plurality of images using the processor configured by machine readable instructions; and

writing the encoded images in the plurality of images to the image file as metadata using the processor configured by machine readable instructions.

19. The method of claim 11, wherein:

capture image data using RGB color channels using at least one camera in the plurality of cameras; and

converting the image data captured in the RGB color channels into a YCbCr format using the processor configured by machine readable instructions.

20. The method of claim 11, wherein the plurality of images of a scene captured from different viewpoints comprises a stereo pair of images.