Systems and methods for estimating depth and visibility from a reference viewpoint for pixels in a set of images captured from different viewpoints

• US 10,380,752 B2
• Filed: 12/29/2017
• Issued: 08/13/2019
• Est. Priority Date: 08/21/2012
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: Ciurea, Florian, Venkataraman, Kartik, Molina, Gabriel, Lelescu, Dan
• Primary Examiner(s): Du, Haixia
• Application Number:
  US15/858,974
• Publication Number:
  US 20180211402A1
• Time in Patent Office: 592 Days
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0088 Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image
• CPC Class Codes: H04N 23/16 Optical arrangements associated therewith, e.g. for beam-splitting or for colour correction

Abstract

Systems in accordance with embodiments of the invention can perform parallax detection and correction in images captured using array cameras. Due to the different viewpoints of the cameras, parallax results in variations in the position of objects within the captured images of the scene. Methods in accordance with embodiments of the invention provide an accurate account of the pixel disparity due to parallax between the different cameras in the array, so that appropriate scene-dependent geometric shifts can be applied to the pixels of the captured images when performing super-resolution processing. In a number of embodiments, generating depth estimates considers the similarity of pixels in multiple spectral channels. In certain embodiments, generating depth estimates involves generating a confidence map indicating the reliability of depth estimates.

Claims

1. An array camera, comprising:

a plurality of cameras;

a processor;

memory containing an image processing application and calibration data;

wherein the processor is configured by the image processing application to produce an image by;

directing the plurality of cameras to capture a plurality of images including a reference image and at least one alternate view image;

retrieving the calibration data from memory;

rectifying the plurality of captured images using the calibration data;

performing a calibration process;

determining a range of disparities to search based upon characteristics of at least one of the plurality of images;

measuring similarity of corresponding pixels in the plurality of images at a plurality of disparities;

generating an initial depth map; and

refocusing the reference image using the initial depth map;

wherein the processor is configured by the image processing application to measure similarity of corresponding pixels in the plurality of images at the plurality of disparities using a cost function.

2. The array camera of claim 1, wherein the calibration process comprises:

obtaining additional scene-independent geometric corrections that compensate for scene-independent geometric distortions that remain after rectification using offline the calibration data.

3. The array camera of claim 2, wherein scene-independent geometric corrections determined using rectification are determined at a sub-pixel resolution.

4. The array camera of claim 1, wherein the processor is configured by the image processing application to:

determine characteristics of at least one of the plurality of images;

determine the range of disparities to search based upon the determined characteristics.

5. The array camera of claim 1, wherein the processor is configured by the image processing application to select a set of disparities to search across the range of disparities.

6. The array camera of claim 5, wherein the set of disparities do not provide uniform sampling of disparity.

7. The array camera of claim 1, wherein the processor is configured by the image processing application to select a set of disparities to search across the range of disparities based upon prior information about where objects are in the at least one of the plurality of images.

8. The array camera of claim 1, wherein the processor is configured by the image processing application to measure similarity of corresponding pixels using a block-based similarity measure.

9. The array camera of claim 1, wherein the processor is configured by the image processing application to filter calculated costs using an edge preserving filter.

10. The array camera of claim 9, wherein the processor is configured by the image processing application to filter the calculated costs using a bilateral filter.

11. The array camera of claim 10, wherein the processor is configured by the image processing application to determine filter weights of the bilateral filter based upon the reference image and the resulting filter weights are applied to the calculated costs.

12. The array camera of claim 1, wherein the processor is configured by the image processing application to identify pixels that may be occluded using the initial depth map by identifying pixel depth values that are inconsistent with local pixel depth values.

13. The array camera of claim 1, wherein the processor is configured by the image processing application to identify occluded pixels based upon scene-dependent geometric shifts of the pixels.

14. The array camera of claim 1, wherein the processor is configured by the image processing application to generate a depth of an occluded pixel by averaging depth of surrounding pixels.

15. The array camera of claim 1, wherein the processor is configured by the image processing application to generate a depth of an occluded pixel by interpolating depths of adjacent pixels for which depths have been calculated.

16. The array camera of claim 1, wherein the processor is configured by the image processing application to generate a depth estimate for a given pixel location in the initial depth map by:

identifying at least one pixel in the plurality of images that corresponds to the given pixel location in the reference image based upon expected disparity at the plurality of depths;

comparing the similarity of the pixels that are identified as corresponding at each of the plurality of depths; and

selecting a depth from the plurality of depths at which the identified corresponding pixels have a highest degree of similarity as a depth estimate for the given pixel location.

17. The array camera of claim 16, wherein the processor is configured by the image processing application to select the depth from the plurality of depths at which the identified corresponding pixels have the highest degree of similarity as the depth estimate for the given pixel location by selecting the depth from the plurality of depths at which a spatially filtered cost function for the identified corresponding pixels indicates the highest level of similarity.

18. The array camera of claim 16, wherein the processor is configured by the image processing application to compare the similarity of the pixels that are identified as corresponding at each of the plurality of depths by comparing intensity values of corresponding pixels across color channels.