Systems and methods for generating compressed light field representation data using captured light fields, array geometry, and parallax information

• US 9,462,164 B2
• Filed: 02/21/2014
• Issued: 10/04/2016
• Est. Priority Date: 02/21/2013
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: Pelican Imaging Corporation
• Inventors: Venkataraman, Kartik, Lelescu, Dan, Molina, Gabriel
• Primary Examiner(s): ADAMS, EILEEN M
• Application Number:
  US14/186,871
• Publication Number:
  US 20140232822A1
• Time in Patent Office: 956 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/95 Computational photography systems, e.g. light-field imaging systems

Abstract

Systems and methods for the generating compressed light field representation data using captured light fields in accordance embodiments of the invention are disclosed. In one embodiment, an array camera includes a processor and a memory connected configured to store an image processing application, wherein the image processing application configures the processor to obtain image data, wherein the image data includes a set of images including a reference image and at least one alternate view image, generate a depth map based on the image data, determine at least one prediction image based on the reference image and the depth map, compute prediction error data based on the at least one prediction image and the at least one alternate view image, and generate compressed light field representation data based on the reference image, the prediction error data, and the depth map.

Claims

1. An array camera, comprising:

a processor; and

a memory connected to the processor and configured to store an image processing application;

wherein the image processing application configures the processor to;

obtain image data, wherein;

the image data comprises a set of images comprising a reference image and at least one alternate view image; and

each image in the set of images comprises a set of pixels;

generate a depth map based on the image data, where the depth map describes the distance from the viewpoint of the reference image with respect to objects imaged by pixels within the reference image by performing a parallax detection process to generate the depth map, where the parallax detection process identifies variations in the position of objects within the image data along epipolar lines between the reference image and the at least one alternate view image;

determine at least one prediction image based on the reference image and the depth map, where the prediction images correspond to at least one alternate view image;

compute prediction error data based on the at least one prediction image and the at least one alternate view image, where a portion of prediction error data describes the difference in photometric information between a pixel in a prediction image and a pixel in at least one alternate view image corresponding to the prediction image; and

generate compressed light field representation data based on the reference image, the prediction error data, and the depth map.

2. The array camera of claim 1, further comprising an array camera module comprising an imager array having multiple focal planes and an optics array configured to form images through separate apertures on each of the focal planes;

wherein the array camera module is configured to communicate with the processor; and

wherein the obtained image data comprises images captured by the imager array.

3. The array camera of claim 2, wherein the reference image corresponds to an image captured using one of the focal planes within the image array.

4. The array camera of claim 3, wherein the at least one alternate view image corresponds to the image data captured using the focal planes within the image array separate from the focal planes associated with the reference image.

5. The array camera of claim 2, wherein the reference image corresponds to a virtual image formed based on the images in the array.

6. The array camera of claim 1, wherein the image processing application further configures the processor to compress the generated compressed light field representation data.

7. The array camera of claim 6, wherein the generated compressed light field representation data is compressed using JPEG-DX.

8. A method for generating compressed light field representation data, comprising:

obtaining image data using an array camera, where the image data comprises a set of images comprising a reference image and at least one alternate view image and the images in the set of images comprise a set of pixels;

identifying at least one pixel in the at least one alternative view image corresponding to a reference pixel in the reference image using the array camera;

determining fractional pixel locations within the identified at least one pixel using the array camera, where a fractional pixel location maps to a plurality of pixels in at least one alternative view image;

mapping fractional pixel locations to a specific pixel location within the alternate view image having a determined fractional pixel location using the array camera;

generating a depth map based on the image data using the array camera, where the depth map describes the distance from the viewpoint of the reference image with respect to objects imaged by pixels within the reference image based on the alternate view images;

determining a set of prediction images based on the reference image and the depth map using the array camera, where a prediction image in the set of prediction images is a representation of a corresponding alternate view image in the at least one alternate view image;

computing prediction error data by calculating the difference between a prediction image in the set of prediction images and the corresponding alternate view image that describes the difference in photometric information between a pixel in the reference image and a pixel in an alternate view image using the array camera; and

generating compressed light field representation data based on the reference image, the prediction error data, and the depth map using the array camera.

9. The method of claim 8, wherein the reference image is a virtual image interpolated from a virtual viewpoint within the image data.

10. The method of claim 8, further comprising identifying areas of low confidence within the computed prediction images based on the at least one alternate view image, the reference image, and the depth map using the array camera, where an area of low confidence indicate areas where the information stored in a determined prediction image indicate areas in the reference viewpoint where the pixels in the determined prediction image may not photometrically correspond to the corresponding pixels in the alternate view image.

11. The method of claim 10, further comprising:

identifying at least one additional reference image within the image data using the array camera, where the at least one additional reference image is separate from the reference image;

determining at least one supplemental prediction image based on the reference image, the at least one additional reference image, and the depth map using the array camera; and

computing the supplemental prediction error data based on the at least one alternate additional reference image and the at least one supplemental prediction image using the array camera, where the generated compressed light field representation data further comprises the supplemental prediction error data.

12. The method of claim 11, wherein identifying the at least one additional reference image comprises:

generating an initial additional reference image based on the reference image and the depth map using the array camera, where the initial additional reference image comprises pixels projected from the viewpoint of the reference image based on the depth map; and

forming the additional reference image based on the initial additional reference image and the prediction error data using the array camera, where the additional reference image comprises pixels based on interpolations of pixels propagated from the reference image and the prediction error data.

13. An array camera, comprising:

a processor; and

a memory connected to the processor and configured to store an image processing application;

wherein the image processing application configures the processor to;

obtain image data, wherein;

the image data comprises a set of images comprising a reference image and at least one alternate view image; and

each image in the set of images comprises a set of pixels;

generate a depth map based on the image data, where the depth map describes the distance from the viewpoint of the reference image with respect to objects imaged by pixels within the reference image;

determine at least one prediction image based on the reference image and the depth map, where the prediction images correspond to at least one alternate view image;

compute prediction error data based on the at least one prediction image and the at least one alternate view image, where a portion of prediction error data describes the difference in photometric information between a pixel in a prediction image and a pixel in at least one alternate view image corresponding to the prediction image, by;

identifying at least one pixel in the at least one alternative view image corresponding to a reference pixel in the reference image;

determining fractional pixel locations within the identified at least one pixel, where a fractional pixel location maps to a plurality of pixels in at least one alternative view image; and

mapping fractional pixel locations to a specific pixel location within the alternate view image having a determined fractional pixel location;

generate compressed light field representation data based on the reference image, the prediction error data, and the depth map.

14. The array camera of claim 13, wherein the mapping fractional pixel locations is determined as the pixel being nearest neighbor within the alternative view image.

15. The array camera of claim 13, wherein the image processing application configures the processor to map the fractional pixel locations based on the depth map, where the pixel in the alternate view image is likely to be similar based on its proximity to the corresponding pixel location determined using the depth map of the reference image.

16. An array camera, comprising:

a processor; and

a memory connected to the processor and configured to store an image processing application;

wherein the image processing application configures the processor to;

obtain image data, wherein;

the image data comprises a set of images comprising a reference image and at least one alternate view image; and

each image in the set of images comprises a set of pixels;

generate a depth map based on the image data, where the depth map describes the distance from the viewpoint of the reference image with respect to objects imaged by pixels within the reference image;

determine at least one prediction image based on the reference image and the depth map, where the prediction images correspond to at least one alternate view image;

compute prediction error data based on the at least one prediction image and the at least one alternate view image, where a portion of prediction error data describes the difference in photometric information between a pixel in a prediction image and a pixel in at least one alternate view image corresponding to the prediction image;

identify areas of low confidence within the computed prediction images based on the at least one alternate view image, the reference image, and the depth map, where an area of low confidence indicate areas where the information stored in a determined prediction image indicate areas in the reference viewpoint where the pixels in the determined prediction image may not photometrically correspond to the corresponding pixels in the alternate view image; and

generate compressed light field representation data based on the reference image, the prediction error data, and the depth map.

17. The array camera of claim 16, wherein the image processing application further configures the processor to disregard identified areas of low confidence.

18. The array camera of claim 16, wherein:

the image processing application further configures the processor to;

identify at least one additional reference image within the image data, where the at least one additional reference image is separate from the reference image;

determine at least one supplemental prediction image based on the reference image, the at least one additional reference image, and the depth map; and

compute the supplemental prediction error data based on the at least one alternate additional reference image and the at least one supplemental prediction image; and

the generated compressed light field representation data further comprises the supplemental prediction error data.

19. The array camera of claim 18, wherein the generated compressed light field representation data further comprises the at least one additional reference image.

20. The array camera of claim 18, wherein the image processing application configures the processor to identify the at least one additional reference image by:

generating an initial additional reference image based on the reference image and the depth map, where the initial additional reference image comprises pixels projected from the viewpoint of the reference image based on the depth map; and

forming the additional reference image based on the initial additional reference image and the prediction error data, where the additional reference image comprises pixels based on interpolations of pixels propagated from the reference image and the prediction error data.