SYSTEM AND METHOD FOR DISPARITY ESTIMATION USING CAMERAS WITH DIFFERENT FIELDS OF VIEW

• US 20200134848A1
• Filed: 03/26/2019
• Published: 04/30/2020
• Est. Priority Date: 10/29/2018
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: El-Khamy, Mostafa, Du, Xianzhi, Ren, Haoyu, Lee, Jungwon
• Granted Patent: US 11,055,866 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/133 Equalising the characteristics of different image components, e.g. their average brightness or colour balance
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images

Abstract

An electronic device and method are herein disclosed. The electronic device includes a first camera with a first field of view (FOV), a second camera with a second FOV that is narrower than the first FOV, and a processor configured to capture a first image with the first camera, the first image having a union FOV, capture a second image with the second camera, determine an overlapping FOV between the first image and the second image, generate a disparity estimate based on the overlapping FOV, generate a union FOV disparity estimate, and merge the union FOV disparity estimate with the overlapping FOV disparity estimate.

Claims

1. An electronic device, comprising:

a first camera with a first field of view (FOV);

a second camera with a second FOV that is narrower than the first FOV; and

a processor configured to;

capture a first image with the first camera, the first image having a union FOV;

capture a second image with the second camera,

determine an overlapping FOV between the first image and the second image;

generate a disparity estimate based on the overlapping FOV;

generate a union FOV disparity estimate; and

merge the union FOV disparity estimate with the overlapping FOV disparity estimate.

2. The electronic device of claim 1, wherein the overlapping FOV disparity estimation is generated using stereo matching network disparity estimation using the overlapping FOV.

3. The electronic device of claim 1, wherein the processor is further configured to crop the determined overlapping FOV from the first image and scale the overlapping FOV from the second image to a resolution corresponding to the first image.

4. The electronic device of claim 3, wherein the processor is further configured to enhance the union FOV disparity estimate using the overlapping FOV disparity estimate.

5. The electronic device of claim 1, wherein the union FOV disparity estimate is generated based on a depth estimation of the first FOV from the first image only and from information based on a camera baseline and a focal length of the first camera.

6. The electronic device of claim 1, wherein the processor is configured to merge the union FOV disparity estimate with the overlapping FOV disparity estimate based on a bias and scale calibration in the overlapping FOVs in the two disparity estimates.

7. The electronic device of claim 1, wherein the union FOV disparity estimate is generated based on a disparity estimate from a red-green-blue (RGB) single image depth estimation (SIDE) network.

8. The electronic device of claim 1, wherein the union FOV disparity estimate is further generated between a zero-padded overlapping FOV and the union FOV from a stereo matching disparity estimation (SMDE) network.

9. The electronic device of claim 1, wherein the union FOV disparity estimate is further generated between a zero-padded overlapping FOV and the union FOV from a multi-task stereo matching disparity and single-image depth estimation (MT-SMDE) network.

10. The electronic device of claim 1, further comprising:

a third camera with a third FOV that is wider than the first FOV,

wherein the processor is further configured to;

capture a third image with the third camera, the third image having a third FOV; and

generate a depth estimation of the third image based on a depth estimation of the union FOV of the first image.

11. A method, comprising:

capturing a first image with a first camera having a first field of view (FOV), the first image having a union FOV;

capturing a second image with a second camera having a second FOV that is narrower than the first FOV

determining an overlapping FOV between the first image and the second image;

generating a disparity estimate based on the overlapping FOV;

generating a union FOV disparity estimate; and

merging the union FOV disparity estimate with the overlapping FOV disparity estimate.

12. The method of claim 11, wherein the overlapping FOV disparity estimation is generated using stereo matching network disparity estimation using the overlapping FOV.

13. The method of claim 11, further comprising cropping the determined overlapping FOV from the first image and scaling the overlapping FOV from the second image to a resolution corresponding to the first image.

14. The method of claim 13, further comprising enhancing the union FOV disparity estimate using the overlapping FOV disparity estimate.

15. The method of claim 11, wherein the union FOV disparity estimate is generated based on a depth estimation of the first FOV of the first image only and from information based on a camera baseline and a focal length of the first camera.

16. The method of claim 11, wherein merging the union FOV disparity estimate with the overlapping FOV disparity estimate is performed based on a bias and scale calibration in the overlapping FOVs in two two disparity estimates.

17. The method of claim 11, wherein the union FOV disparity estimate is generated based on a disparity estimate from a red-green-blue (RGB) single image depth estimation (SIDE) network.

18. The method of claim 11, wherein the union FOV disparity estimate is further generated between a zero-padded overlapping FOV and the union FOV from a stereo matching disparity estimation (SMDE) network.

19. The method of claim 11, wherein the union FOV disparity estimate is further generated between a zero-padded overlapping FOV and the union FOV from a multi-task stereo matching disparity and single-image depth estimation (MT-SMDE).

20. The method of claim 11, further comprising:

capturing a third image with a third camera having a third FOV that is wider than the first FOV, and

generating a depth estimation of the third image based on a depth estimation of the union FOV of the first image.