Multi-aperture depth map using blur kernels and edges

• US 9,721,357 B2
• Filed: 05/23/2016
• Issued: 08/01/2017
• Est. Priority Date: 02/26/2015
• Status: Expired due to Fees

• Current Assignee: Dual Aperture International Co. Ltd.
• Original Assignee: Dual Aperture International Co. Ltd.
• Inventors: Wajs, Andrew
• Primary Examiner(s): HOSSAIN, FARZANA E
• Application Number:
  US15/162,147
• Publication Number:
  US 20160267667A1
• Time in Patent Office: 435 Days
• Field of Search: 348 49, 348149, 348152, 348135, 348142
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/10148 Varying focus
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20024 Filtering details
• CPC Class Codes: G06T 2207/20192 Edge enhancement; Edge preservation
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/571 from focus
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/15 for colour aspects of image signals
• CPC Class Codes: H04N 13/211 using temporal multiplexing
• CPC Class Codes: H04N 13/218 using spatial multiplexing
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/12 with one sensor only
• CPC Class Codes: H04N 5/2226 : Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10

Abstract

The present disclosure overcomes the limitations of the prior art by using blurring of edges. For example, a first image may contain an edge and a second image may contain the same edge as the first image. The two images may be captured by imaging systems with blur characteristics that vary differently as a function of object depth. For example, a dual-aperture system may simultaneously capture a faster f-number visible image and a slower f-number infrared image. Depth information may be generated by comparing blurring of the same edge in the two images.

Claims

1. A computer-implemented method for processing blurred image data, comprising:

selecting a plurality of first windows from first image data associated with a first image of an object, the first image captured using a first imaging system;

selecting a corresponding plurality of second windows from second image data associated with a second image of the object wherein corresponding first and second windows contain a same edge in the object, the second image captured using a second imaging system, wherein a comparison of blurring by the first imaging system and blurring by the second imaging system varies as a function of object depth;

for pairs of corresponding first and second windows, estimating the comparison of blurring by the first and second imaging systems based on blurring of the same edge in corresponding first and second windows, wherein estimating the comparison of blurring by the first and second imaging systems comprises;

binarizing the same edge in corresponding first and second windows, and

determining a blur kernel that approximates blurring of the same binarized edge in corresponding first and second windows, wherein different blur kernels correspond to different object depths; and

generating depth information for the object based on said estimated comparisons comprises selecting the object depth that corresponds to the determined blur kernel.

2. The computer-implemented method of claim 1, wherein the comparison of blurring by the first imaging system and blurring by the second imaging system is a comparison of blur spot size of the first imaging system and blur spot size of the second imaging system.

3. The computer-implemented method of claim 1, wherein determining the blur kernel that approximates blurring of the same edge in corresponding first and second windows comprises blurring the edges using different blur kernels.

4. The computer-implemented method of claim 1, wherein determining the blur kernel that approximates blurring of the same edge in corresponding first and second windows comprises convolving the edges using different blur kernels.

5. The computer-implemented method of claim 1, wherein determining the blur kernel that approximates blurring of the same binarized edge in corresponding first and second windows comprises summing different blur kernels along the binarized edge.

6. The computer-implemented method of claim 5, wherein determining the blur kernel that approximates blurring of the same binarized edge in corresponding first and second windows comprises summing different blur kernels along the binarized edge, the blur kernel depending on an orientation of the edge.

7. The computer-implemented method of claim 1, wherein determining the blur kernel that approximates blurring of the same binarized edge in corresponding first and second windows comprises only summing mathematical operations.

8. The computer-implemented method of claim 1, wherein the windows include edges but do not include parallel edges that are spaced close enough to interfere with each other when blurred by the blur kernels.

9. The computer-implemented method of claim 1, wherein:

estimating the comparison of blurring by the first and second imaging systems comprises;

for each blur kernel from a bank of blur kernels, wherein each blur kernel corresponds to a different object depth, and the bank of blur kernels spans a range of object depths;

blurring the edge of the second window with the blur kernel; and

comparing the blurred edge and the same edge of the corresponding first window; and

generating depth information for the object comprises generating depth information for the object based on said comparisons.

10. The computer-implemented method of claim 9, wherein a size of the windows is different than a size of the blur kernels.

11. The computer-implemented method of claim 10, wherein a size of the windows is larger than a size of the blur kernels.

12. The computer-implemented method of claim 1, wherein estimating the comparison of blurring by the first and second imaging systems comprises phase matching the same edges in corresponding first and second windows.

13. The computer-implemented method of claim 12, wherein phase matching the same edges in corresponding first and second windows comprises:

applying derivative operators to the same edges in corresponding first and second windows; and

taking absolute values of the derivatives of the same edges in corresponding first and second windows.

14. The computer-implemented method of claim 1, wherein estimating the comparison of blurring by the first and second imaging systems comprises equating energy in the same edges in corresponding first and second windows.

15. The computer-implemented method of claim 14, wherein equating energy in the same edges in corresponding first and second windows comprises scaling amplitude of the same edges in corresponding first and second windows so that the amplitude-scaled edges have equal energy.

16. The computer-implemented method of claim 1, wherein the first imaging system is characterized by a first f-number and the second imaging system is characterized by a second f-number that is different than the first f-number.

17. The computer-implemented method of claim 1, wherein the first and second imaging systems are different parts of a dual-aperture imaging system, the first imaging system using a visible spectral band and characterized by a first f-number, and the second imaging system using an infrared spectral band and characterized by a second f-number that is slower than the first f-number.

18. A non-transitory computer-readable storage medium storing executable computer program instructions for processing blurred image data, the instructions executable by a processor and causing the processor to perform a method comprising:

selecting a plurality of first windows from first image data associated with a first image of an object, the first image captured using a first imaging system;

selecting a corresponding plurality of second windows from second image data associated with a second image of the object wherein corresponding first and second windows contain a same edge in the object, the second image captured using a second imaging system wherein a comparison of blurring by the first and second imaging systems varies as a function of object depth;

for pairs of corresponding first and second windows, estimating the comparison of blurring by the first and second imaging systems based on blurring of the same edge in corresponding first and second windows, wherein estimating the comparison of blurring by the first and second imaging systems comprises;

binarizing the same edge in corresponding first and second windows, and

determining a blur kernel that approximates blurring of the same binarized edge in corresponding first and second windows, wherein different blur kernels correspond to different object depths; and

generating depth information for the object based on said estimated comparisons comprises selecting the object depth that corresponds to the determined blur kernel.