Systems and methods for synthesizing images from image data captured by an array camera using restricted depth of field depth maps in which depth estimation precision varies

• US 9,800,856 B2
• Filed: 12/12/2016
• Issued: 10/24/2017
• Est. Priority Date: 03/13/2013
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: Fotonation Cayman Limited
• Inventors: Venkataraman, Kartik, Nisenzon, Semyon, Chatterjee, Priyam, Molina, Gabriel
• Primary Examiner(s): Ye, Lin
• Assistant Examiner(s): Cowan, Euel
• Application Number:
  US15/376,529
• Publication Number:
  US 20170094243A1
• Time in Patent Office: 316 Days
• Field of Search: 348 44, 348 47, 348 48, 3482181
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Systems and methods are described for generating restricted depth of field depth maps. In one embodiment, an image processing pipeline application configures a processor to: determine a desired focal plane distance and a range of distances corresponding to a restricted depth of field for an image rendered from a reference viewpoint; generate a restricted depth of field depth map from the reference viewpoint using the set of images captured from different viewpoints, where depth estimation precision is higher for pixels with depth estimates within the range of distances corresponding to the restricted depth of field and lower for pixels with depth estimates outside of the range of distances corresponding to the restricted depth of field; and render a restricted depth of field image from the reference viewpoint using the set of images captured from different viewpoints and the restricted depth of field depth map.

Claims

1. An image processing system, comprising:

a processor;

memory containing a set of images captured from different viewpoints and an image processing pipeline application;

wherein the image processing pipeline application configures the processor to;

determine a range of distances corresponding to a restricted depth of field;

generate a restricted depth of field depth map from the reference viewpoint using the set of images captured from different viewpoints, where the restricted depth of field depth map is generated by performing a disparity search that is performed using a greater density of depth samples within the range of distances corresponding to the restricted depth of field and a lower density of depth samples for distances outside the range of distances corresponding to the restricted depth of field; and

render a restricted depth of field image from the reference viewpoint using the set of images captured from different viewpoints and the restricted depth of field depth map.

2. The image processing system of claim 1, wherein the image processing pipeline application further configures the processor to render objects in the field image that are located away from the restricted depth of field to be blurred in an amount proportional to their distance from the plane of best focus.

3. The image processing system of claim 1, wherein the image processing pipeline application further configures the processor to automatically determine the range of distances corresponding to a restricted depth of field.

4. The image processing system of claim 3, wherein the image processing pipeline application further configures the processor to automatically determine the range of distances corresponding to the restricted depth of field by determining a distance to a surface of a scene object using the set of images captured from different viewpoints.

5. The image processing system of claim 4, wherein the image processing pipeline application further configures the processor to determine a distance to a surface of a scene object using the set of images captured from different viewpoints by:

generating an initial depth map and a confidence map from at least a portion of the set of images captured from different viewpoints, where the confidence map indicates the reliability of pixel depth estimates in the initial depth map; and

determining the depth of the surface of the scene object based upon at least one pixel depth estimate within the initial depth map marked as confident within the confidence map.

6. The image processing system of claim 4, wherein the image processing pipeline application further configures the processor to receive a user instruction identifying a surface of a scene object by:

generating a preview image from the set of images captured from different viewpoints, where the preview image includes a user interface cue; and

identifying a surface of a scene object visible within the set of images captured from different viewpoints based upon the location of the user interface cue.

7. The image processing system of claim 4, wherein the image processing pipeline application further configures the processor to determine the range of distances corresponding to the restricted depth of field based upon user instructions.

8. The image processing system of claim 1, wherein the image processing pipeline application further configures the processor to generate a restricted depth of field depth map by:

generating an initial depth map using the set of images captured from different viewpoints;

determining pixel locations with depth estimates from the initial depth map indicating that the pixel locations are likely to have depths within the range of distances corresponding to the restricted depth of field;

generating higher depth estimation precision depth estimates for at least some of the pixel locations that are likely to have depths within the range of distances corresponding to the restricted depth of field using the set of images captured from different viewpoints; and

generating a restricted depth of field depth map using at least some of the depth estimates from the initial depth map and at least some of the higher depth estimation precision depth estimates.

9. The image processing system of claim 8, wherein the image processing pipeline application further configures the processor to generate an initial depth map by:

downsampling at least some of the images in the set of images captured from different viewpoints to obtain a set of lower spatial resolution images; and

determining a low spatial resolution depth map using the set of lower spatial resolution images.

10. The image processing system of claim 9, wherein the image processing pipeline application further configures the processor to determine a low spatial resolution depth map using the set of lower spatial resolution images by:

performing a disparity search with respect to a given pixel location using the set of lower spatial resolution images;

wherein the disparity search is performed by searching a first set of disparities.

11. The image processing system of claim 10, wherein the image processing pipeline application further configures the processor to generate the higher precision depth estimates by:

performing a disparity search with respect to a given pixel location using the set of images captured from different viewpoints;

wherein the disparity search is performed by searching a second set of disparities that provides a greater density of depth samples than the first set of disparities; and

wherein a search performed using the second set of disparities provides greater depth estimation precision within the range of distances corresponding to the restricted depth of field than the precision of a depth estimate obtained within the same range of distances by a search performed using the first set of disparities.

12. The image processing system of claim 11, wherein the first set of disparities is not uniformly distributed with respect to disparity.

13. The image processing system of claim 11, wherein the first set of disparities is uniformly distributed with respect to disparity.

14. The image processing system of claim 11, wherein the second set of disparities is not uniformly distributed with respect to disparity.

15. The image processing system of claim 11, wherein the second set of disparities is uniformly distributed with respect to disparity.

16. The image processing system of claim 8, wherein the image processing pipeline application further configures the processor to:

generate an initial confidence map for the initial depth map; and

determine pixel locations with depth estimates from the initial depth map indicating that the pixel locations are likely to have depths within the range of distances corresponding to the restricted depth of field based upon the depth estimate for the pixel location in the initial depth map and the confidence of the depth estimate for the pixel location indicated by the initial confidence map.

17. The image processing system of claim 8, wherein the image processing pipeline application further configures the processor to determine pixel locations with depth estimates from the initial depth map indicating that the pixel locations are likely to have depths within the range of distances corresponding to the restricted depth of field based upon the depth estimate for the pixel location and a determination that the pixel is not contained within a textureless region.

18. An image processing system, comprising:

a processor;

memory containing a set of images captured from different viewpoints and an image processing pipeline application;

wherein the image processing pipeline application configures the processor to;

automatically determine a range of distances corresponding to a restricted depth of field by determining a distance to a surface of a scene object using the set of images captured from different viewpoints;

generate a restricted depth of field depth map from the reference viewpoint using the set of images captured from different viewpoints, where the restricted depth of field depth map is generated by performing a disparity search that is performed using a greater density of depth samples within the range of distances corresponding to the restricted depth of field and a lower density of depth samples for distances outside the range of distances corresponding to the restricted depth of field; and

render a restricted depth of field image from the reference viewpoint using the set of images captured from different viewpoints and the restricted depth of field depth map.

19. The image processing system of claim 18, wherein the image processing pipeline application further configures the processor to render objects in the field image that are located away from the restricted depth of field to be blurred in an amount proportional to their distance from the plane of best focus.

20. The image processing system of claim 18, wherein the image processing pipeline application further configures the processor to determine a distance to a surface of a scene object using the set of images captured from different viewpoints by:

generating an initial depth map and a confidence map from at least a portion of the set of images captured from different viewpoints, where the confidence map indicates the reliability of pixel depth estimates in the initial depth map; and

determining the depth of the surface of the scene object based upon at least one pixel depth estimate within the initial depth map marked as confident within the confidence map.