Multi-aperture depth map using partial blurring

• US 9,721,344 B2
• Filed: 05/23/2016
• Issued: 08/01/2017
• Est. Priority Date: 02/26/2015
• Status: Expired due to Fees

• Current Assignee: Dual Aperture International Co. Ltd.
• Original Assignee: Dual Aperture International Co. Ltd.
• Inventors: Wajs, Andrew
• Primary Examiner(s): HOSSAIN, FARZANA E
• Application Number:
  US15/162,154
• Publication Number:
  US 20160269710A1
• Time in Patent Office: 435 Days
• Field of Search: 348 49, 348149, 348152, 38 49, 38135, 38142
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/10148 Varying focus
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20024 Filtering details
• CPC Class Codes: G06T 2207/20192 Edge enhancement; Edge preservation
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/571 from focus
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/15 for colour aspects of image signals
• CPC Class Codes: H04N 13/211 using temporal multiplexing
• CPC Class Codes: H04N 13/218 using spatial multiplexing
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/12 with one sensor only
• CPC Class Codes: H04N 5/2226 : Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10

Abstract

The present disclosure overcomes the limitations of the prior art by using partial blur kernels, rather than using full blur kernels. For example, single-sided blur kernels may be used in order to accommodate different edge orientations. In one aspect, the single-sided blur kernels may be used in order to accommodate edges caused by occlusions, where the two sides of the edge are at different depths.

Claims

1. A computer-implemented method for processing blurred image data, comprising:

accessing first image data associated with a first image of an object, the first image captured using a first imaging system;

accessing second image data associated with a second image of the object, the second image captured using a second imaging system wherein a comparison of blurring by the first imaging system and blurring by the second imaging system varies as a function of object depth;

selecting a plurality of first windows from the first image data;

selecting a corresponding plurality of second windows from the second image data wherein corresponding first and second windows contain a same edge in the object;

estimating the comparison of blurring by the first and second imaging systems based on comparison of single-sided blurring of the first and second image data, wherein estimating the comparison of blurring by the first and second imaging systems comprises, for pairs of corresponding first and second windows, estimating the comparison of blurring by the first and second imaging systems based on single-sided blurring of the same edge in corresponding first and second windows, wherein for at least one of the edges, opposite sides of the edge are at different object depths; and

generating depth information based on said estimated comparisons.

2. The computer-implemented method of claim 1, wherein the comparison of blurring by the first imaging system and blurring by the second imaging system is a comparison of blur spot size of the first imaging system and blur spot size of the second imaging system.

3. The computer-implemented method of claim 1, wherein the comparison of blurring by the first imaging system and blurring by the second imaging system is a comparison of size of one side of a blur spot of the first imaging system and size of the same side of a blur spot of the second imaging system.

4. The computer-implemented method of claim 1, wherein:

estimating the comparison of blurring by the first and second imaging systems comprises, for pairs of corresponding first and second windows, estimating the comparison of blurring by the first and second imaging systems based on single-sided blurring on both sides of the same edge in corresponding first and second windows; and

generating depth information based on said estimated comparisons comprises generating separate depth information for each side of the same edge based on said estimated comparisons.

5. The computer-implemented method of claim 1, wherein the different object depths are caused by occlusion.

6. The computer-implemented method of claim 1, wherein:

estimating the comparison of blurring by the first and second imaging systems comprises determining a single-sided blur kernel that approximates single-sided blurring of the same edge in corresponding first and second windows, wherein different single-sided blur kernels correspond to different object depths; and

generating depth information for the object comprises selected the object depth that corresponds to the determined single-sided blur kernel.

7. The computer-implemented method of claim 6, wherein different single-sided blur kernels correspond to different object depths and to different orientations of the edge.

8. The computer-implemented method of claim 6, wherein the single-sided blur kernel is half a full blur kernel.

9. The computer-implemented method of claim 6, wherein determining a single-sided blur kernel that approximates single-sided blurring of the same edge in corresponding first and second windows comprises convolving the edges using different single-sided blur kernels.

10. The computer-implemented method of claim 6, wherein estimating the comparison of single-sided blurring by the first and second imaging systems comprises:

binarizing the same edge in corresponding first and second windows; and

determining a single-sided blur kernel that approximates single-sided blurring of the same binarized edge in corresponding first and second windows, wherein different single-sided blur kernels correspond to different object depths.

11. The computer-implemented method of claim 6, wherein the windows include edges but do not include parallel edges that are spaced close enough to interfere with each other when blurred by the single-sided blur kernels.

12. The computer-implemented method of claim 1, wherein:

estimating the comparison of blurring by the first and second imaging systems comprises;

for each single-sided blur kernel from a bank of single-sided blur kernels, wherein each single-sided blur kernel corresponds to a different object depth and to a different edge orientation, and the bank of single-sided blur kernels spans a range of object depths;

blurring the edge of the second window with the single-sided blur kernel; and

comparing the blurred edge and the same edge of the corresponding first window; and

generating depth information for the object comprises generating depth information for the object based on said comparisons.

13. The computer-implemented method of claim 12, wherein a size of the windows is larger than a size of the blur kernels.

14. The computer-implemented method of claim 1, wherein estimating the comparison of blurring by the first and second imaging systems comprises phase matching and equating energy in the same edges in corresponding first and second windows.

15. The computer-implemented method of claim 1, wherein the first imaging system is characterized by a first f-number and the second imaging system is characterized by a second f-number that is different than the first f-number.

16. The computer-implemented method of claim 1, wherein the first and second imaging systems are different parts of a dual-aperture imaging system, the first imaging system using a visible spectral band and characterized by a first f-number, and the second imaging system using an infrared spectral band and characterized by a second f-number that is slower than the first f-number.

17. A non-transitory computer-readable storage medium storing executable computer program instructions for processing blurred image data, the instructions executable by a processor and causing the processor to perform a method comprising:

accessing first image data associated with a first image of an object, the first image captured using a first imaging system;

accessing second image data associated with a second image of the object, the second image captured using a second imaging system wherein a comparison of blurring by the first imaging system and blurring by the second imaging system varies as a function of object depth;

selecting a plurality of first windows from the first image data;

selecting a corresponding plurality of second windows from the second image data wherein corresponding first and second windows contain a same edge in the object;

estimating the comparison of blurring by the first and second imaging systems based on comparison of single-sided blurring of the first and second image data, wherein estimating the comparison of blurring by the first and second imaging systems comprises, for pairs of corresponding first and second windows, estimating the comparison of blurring by the first and second imaging systems based on single-sided blurring of the same edge in corresponding first and second windows, wherein for at least one of the edges, opposite sides of the edge are at different object depths; and

generating depth information based on said estimated comparisons.