DEPTH MAP SUPER-RESOLUTION PROCESSING METHOD

• US 20160328828A1
• Filed: 07/21/2016
• Published: 11/10/2016
• Est. Priority Date: 02/25/2014
• Status: Active Grant

• Current Assignee: Tsinghua University
• Original Assignee: Tsinghua University
• Inventors: LI, Yangguang, ZHANG, Yongbing, WANG, Haoqian, ZHANG, Lei, JIN, Xin
• Granted Patent: US 10,115,182 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20212 Image combination
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

The present invention discloses a depth map super-resolution processing method, including: firstly, respectively acquiring a first original image (S1) and a second original image (S2) and a low resolution depth map (d) of the first original image (S1); secondly, 1) dividing the low resolution depth map (d) into multiple depth image blocks; 2) respectively performing the following processing on the depth image blocks obtained in step 1); 21) performing super-resolution processing on a current block with multiple super-resolution processing methods, to obtain multiple high resolution depth image blocks; 22) obtaining new synthesized image blocks by using an image synthesis technology; 23) upon matching and judgment, determining an ultimate high resolution depth image block; and 3) integrating the high resolution depth image blocks of the depth image blocks into one image according to positions of the depth image blocks in the low resolution depth map (d). Through the depth map super-resolution processing method of the present invention, depth information of the obtained high resolution depth maps is more accurate.

Claims

1. A depth map super-resolution processing method, comprising:

firstly, performing image acquisition on the same scene in a first position and a second position, and respectively acquiring a first original image (S1) and a second original image (S2);

acquiring a low resolution depth map (d) of the first original image (S1);

secondly, performing the following processing;

1) dividing the low resolution depth map (d) into multiple depth image blocks;

2) respectively performing the following processing on the depth image blocks obtained in step 1);

21) performing super-resolution processing on a current block with multiple super-resolution processing methods, to obtain multiple initial high resolution depth image blocks having the resolution the same as that of the first original image (S1);

22) traversing the multiple high resolution depth image blocks obtained in step 21), respectively combining corresponding image blocks in the first original image (S1) which correspond to the current block, and synthesizing multiple image blocks corresponding to the second original image (S2) by using an image synthesis technology according to a relative position relationship between the first position and the second position, which are defined as multiple synthesized image blocks;

23) traversing the multiple synthesized image blocks obtained in step 22), respectively calculating a matching degree between each synthesized image block and a corresponding block in the second original image (S2) which corresponds to the current block, determining the synthesized image block with the highest matching degree, and determining a high resolution depth image block corresponding to the synthesized image block with the highest matching degree as an ultimate high resolution depth image block of the current block; and

3) integrating the high resolution depth image blocks of the depth image blocks into one image according to positions of the depth image blocks in the low resolution depth map (d), to obtain a super-resolution processing map of the low resolution depth map (d).

2. The depth map super-resolution processing method according to claim 1, wherein, when the image synthesis technology is adopted, step 22) comprises the following steps:

a) projecting the corresponding image blocks in the first original image (S1) into a three-dimensional space by using a reference camera with a depth-image-based-rendering method according to depth information of the high resolution depth image blocks; and

b) setting the center of a virtual camera by making the center of the reference camera correspond to the first position and according to the relative position relationship of the second position relative to the first position, and imaging a scene of the three-dimensional space obtained in step a) into a two-dimensional plane by using the virtual camera, so as to obtain a synthesized image block.

3. The depth map super-resolution processing method according to claim 2, wherein, when projection is conducted in step a), the corresponding image block is projected into the three-dimensional space according to the following equation:

(X_W,Y_W,Z_W)^T=K₁<sup>−

1</sup>d₁p₁

wherein the center of the reference camera is the center of the world coordinate system, and a direction observed from the reference camera is a z-axis direction of the coordinate system;

p₁ indicates position information of a pixel point p₁ in the corresponding image block in the first original image (S1), in a homogeneous form;

d₁ is depth information of a corresponding pixel point p₁ in the high resolution depth image block, K₁ is a built-in parameter matrix of the reference camera, and (X_W,Y_W,Z_W) is a coordinate of a point in the three-dimensional space into which the pixel point p₁ is projected.

4. The depth map super-resolution processing method according to claim 2, wherein, when the scene of the three-dimensional space is imaged to the two-dimensional plane in step b), the scene of the three-dimensional space is imaged to the two-dimensional plane according to the following equation:

d₂p₂=k₂R₂P_W−

K₂R₂C₂

wherein the center of the reference camera is the center of the world coordinate system, and a direction observed from the reference camera is a z-axis direction of the coordinate system;

C₂ is a coordinate of the center of the virtual camera, R₂ is a rotation matrix of the virtual camera, K₂ is a built-in parameter matrix of the virtual camera, P_W is a coordinate of a point in the three-dimensional space obtained in step a), p₂ and d₂ are position and depth information of the corresponding pixel point in the synthesized image block obtained through imaging to the two-dimensional plane;

an operation result on the right of the equation is converted to a homogeneous form;

m(x, y,

     1), p₂ is (x, y), and d₂ is the coefficient m.

5. The depth map super-resolution processing method according to claim 1, wherein, after the high resolution depth image blocks are integrated into one image, step 3) further comprises smoothing the image.

6. The depth map super-resolution processing method according to claim 1, wherein, when the low resolution depth map (d) is acquired, image acquisition is performed on the scene in the first position by using a depth camera, so as to acquire the low resolution depth map (d).