One method of binocular depth perception based on active structured light

• US 10,142,612 B2
• Filed: 01/07/2015
• Issued: 11/27/2018
• Est. Priority Date: 02/13/2014
• Status: Active Grant

• Current Assignee: Chenyang Ge, Nanning Zheng
• Original Assignee: Chenyang Ge, Nanning Zheng
• Inventors: Ge, Chenyang, Zheng, Nanning, Yao, Huimin, Zhou, Yanhui, Lun, Jiankun
• Primary Examiner(s): Itskovich, Mikhail
• Application Number:
  US14/591,083
• Publication Number:
  US 20150229911A1
• Time in Patent Office: 1,420 Days
• Field of Search: 348 47
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

The present invention provides a method of binocular depth perception based on active structured light, adopting a coded pattern projector to project a coded pattern for structured light coding of the projective space or target object (characteristic calibration), then obtaining the coded pattern by means of two cameras on the same baseline and respectively located symmetrically on both sides of the coded pattern projector, after preprocessing and projection shadow detection, estimating the block matching movement in two modes based on the image blocks (binocular block matching and automatic matching) to obtain the offset of the optimal matching block, finally working out the depth value according to the formula for depth calculation and compensating the depth of the projection shadows to generate high-resolution and high-precision depth information.

Claims

1. A method of binocular depth perception based on active structured light, comprising the following steps of:

Step 1;

projecting coherent laser beams, by a coded pattern projector, with a coded pattern to carry out structured light coding for a target object with an unknown depth;

Step 2;

arranging a first camera and a second camera symmetrically at the same distances on the left side and right side of the coded pattern projector to acquire and fix their respective reference coded pattern R_l and reference coded pattern R_r, the first camera and the second camera being two separate and distinct components and each having the same or substantially the same optical lens and image sensor, and sharing the same baseline with the coded pattern projector and receiving the coded pattern within the range of a wavelength;

Step 3;

acquiring input image I_l, by the first camera, and acquiring input image I_r, by the second camera, each of the input image I_l and the input image I_r containing the coded pattern and the target object and preprocessing the input images I_l and I_r, wherein the preprocessing includes video format conversion, color space conversion, and grey image adaptive denoising and enhancement;

Step 4;

using the input image I_l and the input image l_r after being preprocessed to detect projection shadow areas A_l and A_r of the target object respectively, wherein projection shadow area A_r located behind the left side of the target object is detected in the input image I_l and projection shadow area A_l located behind the right side the target object is detected in the input image I_r;

Step 5;

performing two matching motion estimation;

a first block matching motion estimation based on the symmetric arrangements and equal distances of the first camera and the second camera from the coded pattern projector and a second of block matching motion estimation to generate the offset respectively, wherein the first block matching motion estimation is to perform a binocular block matching calculation between a first input image block of the input image I_l and a corresponding matching image block of the input image I_r based on the symmetric arrangements and equal distances of the first camera and the second camera from the projector and get an X-axis offset Δ

x_l,r or a Y-axis offset Δ

y _l, r; and

the second block matching motion estimation is to perform (1) a first block matching calculation between the first input image block of the input image I_l image and a corresponding matching image block with the reference coded pattern R_l to get an X-axis offset Δ

x_l and a Y-axis offset Δ

y_l and (2) a second block matching calculation between a second input image block of the input image I_r and a corresponding matching image block with the reference coded pattern R_r to get an X-axis offset Δ

xr or a Y-axis offset Δ

yr, wherein the block matching motion estimation is based on similarity values between input images and corresponding matching images;

Step 6;

carrying out depth calculation, including;

(6a) selecting the X-axis offset Δ

x_{l, r}or Δ

y _l,r and combining the focal length f of the image sensor, the baseline distance between the first camera and the second camera S and a dot pitch parameter μ

of the image sensor to obtain depth information d _{l ,r} for a central point 0 of an image block _mxn;

(6b) selecting the X-axis offset Δ

x_l and Δ

x_r or the Y-axis offset Δ

y_l and Δ

y_r and combining a given distance parameter d of the reference coded pattern R_l and reference coded pattern R_r, the focal length f of the image sensor, the baseline distance s between the first camera and the coded pattern projector, as well as the dot pitch parameter μ

of the image sensor to obtain depth information d_l and d _r respectively for the central point 0 of the image block_mxn corresponding to the same position in each of the input image I_l and the input image I_r;

Step 7;

performing depth compensation, including, using the depth information d_l and d _r, combining the projection shadow areas A_l and A_r detected in Step 4 to compensate and correct the depth information d_l,r , and outputting a final depth value d_out of the central point 0 on the image block _mxn;

Step 8;

moving the central point 0 of the image block_mxn to a next pixel in the same line, repeating the steps 5-7 to calculate a depth value corresponding to the next pixel and following such calculation sequence from left to right and from top to bottom line by line to obtain the depth information of the input image I_l and the input image I_reach comprising the target object based on point-by-point calculation.

2. The method according to claim 1, wherein the reference coded R_land reference coded pattern R_r, as mentioned in Step 2 are standard patterns stored and cured in the memory for matching benchmark and depth perception calculation after preprocessing of static images acquired by the first camera and the second camera when the coded pattern projector projects code patterns onto a plane vertical to the optical center axis of the coded pattern projector with the distance of d to the coded pattern projector.

3. The method according to claim 1, wherein the projection shadow area detection as mentioned in Step 4 is determined by detecting the number of the feature points contained in the image block_mxn.

4. The method according to claim 1, wherein in Step (6a), if the first camera and the second camera are arranged horizontally to the coded pattern projector, then selecting the X-axis offset Δ

x_l,r;

if the first camera and the second camera are arranged vertically to the coded pattern projector, then selecting the offset Δ

y_{l, r}.

5. The method according to claim 1, wherein in Step (6a), if Δ

x_{l, r} is selected, then the formula for depth calculation is as follows;

6. The method according to claim 1, wherein in Step (6b), if the first camera and the second camera are arranged horizontally to the coded pattern projector, then selecting the X-axis offset Δ

x_l and Δ

x_r;

if the first camera and the second camera are arranged vertically to the coded pattern projector, then selecting the Y-axis offset Y_l and Y_r.

7. The method according to claim 1, wherein in Step (6b), if Δ

x_l and Δ

x_r are selected, then the formula for depth calculation is as follows;

8. The method according to claim 1, wherein in Step (7), the specific method for depth compensation is as follows:

if the central point 0 of the projection image block falls within the left view of a non-cross area, then selecting d_l as the output d_out;

if the central point 0 of the projection image block falls within right view of the non-cross area, then selecting d_r as the output d_out;

if the central point 0 of the image block falls within the cross area, and for non-projection shadow areas, when there is no difference between d_l and d _rand there is a difference between d_l,r and the average of d _l and d_r,

then selecting d_l or d_r as the output d_out, otherwise selecting d_l,r, as the output; and

if the central point 0 of the image block falls within the cross area,

for the projection shadow area A_l, selecting d_r as the output d_out;

for the projection shadow area A_r, then selecting d_l as the output d_out, wherein th_lis a first threshold and the th₂ is a second threshold.