Image processing method and apparatus for calibrating depth of depth sensor

• US 9,858,684 B2
• Filed: 07/28/2014
• Issued: 01/02/2018
• Est. Priority Date: 03/27/2014
• Status: Expired due to Fees

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: Kim, Jae-Hean, Chu, Chang-Woo, Cha, Young-Mi, Choi, Jin-Sung, Koo, Bon-Ki
• Primary Examiner(s): Potts, Ryan P
• Application Number:
  US14/444,030
• Publication Number:
  US 20150279016A1
• Time in Patent Office: 1,254 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2213/003 Aspects relating to the "2D+depth" image format

Abstract

An image processing apparatus and method for calibrating a depth of a depth sensor. The image processing method may include obtaining a depth image of a target object captured by a depth sensor and a color image of the target object captured by a color camera; and calibrating a depth of the depth sensor by calibrating a geometrical relation between a projector and a depth camera, which are included in the depth sensor, based the obtained depth and color images and calculating a correct feature point on an image plane of the depth camera that corresponds to a feature point of an image plane of the projector.

Claims

1. An image processing method for calibrating a depth of a depth sensor, the image processing method comprising:

obtaining a depth image of a target object captured by the depth sensor and a color image of the target object captured by a color camera;

calibrating a geometrical relation between a projector of the depth sensor and a depth camera based on the obtained depth image and the color image; and

calibrating a depth of the depth sensor by calculating a correct feature point on an image plane of the depth camera that corresponds to a feature point of an image plane of the projector,

wherein the calibrating of the depth of the depth sensor comprises;

calculating the feature point on the image plane of the projector that corresponds to a feature point with a predetermined depth value on the image plane of the depth camera,

calculating the correct feature point of the image plane of the depth camera that corresponds to the feature point of the image plane of the projector that is calculated by a first calculator when structured light is projected from an estimated correct position of the projector, and

calculating an actual depth value of the calculated correct feature point on the image plane of the depth camera, and wherein

the calculating of the correct feature point on the image plane of the depth camera comprises;

calculating a rotation matrix correction value and a translation vector correction value,

calculating a corrected epipolar line on the image plane of the depth camera using the calculated feature point on the image plane of the projector, the calculated rotation matrix correction value, and the calculated translation vector correction value, and

searching for a new feature point on the corrected epipolar line based on the rotation matrix correction value and the translation vector correction value, wherein the new feature point corresponds to the correct feature point on the image plane of the depth camera that corresponds to the feature point on the image plane of the projector.

2. The image processing method of claim 1, further comprising:

computing a homography, and

calculating the rotation matrix correction value and the translation vector correction value which minimize a cost function.

3. The image processing method of claim 2, wherein the homography is computed using the rotation matrix correction value and the translation vector correction value, and the correct feature point on the image plane of the depth camera is obtained by multiplying the calculated feature point on the image plane of the projector by the homography.

4. The image processing method of claim 2, wherein the calculating of the rotation matrix correction value and translation vector correction value which minimize the cost function comprises calculating an optimal rotation matrix correction value and an optimal translation vector correction value that minimize a cost function that represents a difference between a correct feature point on the corrected epipolar line on the image plane of the depth camera and a feature point on an epipolar line before correction.

5. The image processing method of claim 1, wherein the searching of the new feature point on the image plane of the depth camera comprises searching the corrected epipolar line on the image plane of the depth camera for a point to be found whose horizontal pixel coordinate is the same as a horizontal pixel coordinate of a feature point on an epipolar line before correction, and then determining the found point as the correct feature point;

and the found point is an intersection between the corrected epipolar line and a line including the feature point on the epipolar line and perpendicular to the epipolar line before the correction.

6. An image processing apparatus for calibrating a depth sensor, the image processing apparatus comprising:

a projector of the depth sensor configured to project structured light;

a depth camera comprising a color camera, the depth camera configured to capture the structured light and the color camera configured to capture a color image of a target object;

an image receiver, executed by a processor causing the processor to perform an image detector to obtain a depth image of the target object captured by the depth sensor and the color image of the target object captured by the color camera, wherein;

the processor is configured to calibrate a geometrical relation between the projector and the depth camera based on the obtained depth image and the color image, and to correct a depth of the depth sensor by calculating a correct feature point on an image plane of the depth camera that corresponds to a feature point of an image plane of the projector,

wherein the processor comprises a first calculator configured to calculate a feature point on an image plane of the projector that corresponds to a feature point with a predetermined depth value on an image plane of the depth camera,

a second calculator configured to calculate the correct feature point on the image plane of the depth camera that corresponds to the feature point on the image plane of the projector that is calculated by the first calculator when structured light is projected from an estimated correct position of the projector, and

a third calculator configured to calculate an actual depth value and actual three-dimensional (3D) coordinates of the correct feature point on the image plane of the depth camera that has been calculated by the second calculator, and wherein

the second calculator is configured to calculate a rotation matrix correction value and a translation vector correction value, calculate a corrected epipolar line on the image plane of the depth camera using the calculated rotation matrix correction value and translation vector correction value and the feature point on the image plane of the projector that has been calculated by the first calculator, and to search for a new feature point on the corrected epipolar line based on the rotation matrix correction value and the translation vector correction value, wherein the new feature point corresponds to the correct feature point on the image plane of the depth camera that corresponds to the feature point on the image plane of the projector.

7. The image processing apparatus of claim 6, wherein the second calculator is configured to calculate an optimal rotation matrix correction value and an optimal translation vector correction value that minimize a cost function that represents a difference between a correct feature point on the corrected epipolar line on the image plane of the depth camera and a feature point on an epipolar line before correction.

8. The image processing apparatus of claim 7, wherein the second calculator is configured to search the corrected epipolar line on the image plane of the depth camera for a point to be found whose horizontal pixel coordinate is the same as a horizontal pixel coordinate of the feature point on the epipolar line before the correction, and then determine the found point as the correct feature point on the corrected epipolar line;

and the found point is an intersection between the corrected epipolar line and a line including the feature point on the epipolar line and perpendicular to the epipolar line before the correction.