IMAGE PROCESSING METHOD AND APPARATUS FOR CALIBRATING DEPTH OF DEPTH SENSOR

• US 20150279016A1
• Filed: 07/28/2014
• Published: 10/01/2015
• Est. Priority Date: 03/27/2014
• Status: Active Grant

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: KIM, Jae-Hean, CHU, Chang-Woo, CHOI, Jin-Sung, KOO, Bon-Ki, CHA, Young-Mi
• Granted Patent: US 9,858,684 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2213/003 Aspects relating to the "2D+depth" image format

Abstract

An image processing apparatus and method for calibrating a depth of a depth sensor. The image processing method may include obtaining a depth image of a target object captured by a depth sensor and a color image of the target object captured by a color camera; and calibrating a depth of the depth sensor by calibrating a geometrical relation between a projector and a depth camera, which are included in the depth sensor, based the obtained depth and color images and calculating a correct feature point on an image plane of the depth camera that corresponds to a feature point of an image plane of the projector.

Claims

1. An image processing method for calibrating a depth of a depth sensor, the image processing method comprising:

obtaining a depth image of a target object captured by a depth sensor and a color image of the target object captured by a color camera; and

calibrating a depth of the depth sensor by calibrating a geometrical relation between a projector and a depth camera, which are included in the depth sensor, based on the obtained depth and color images, and calculating a correct feature point on an image plane of the depth camera that corresponds to a feature point of an image plane of the projector.

2. The image processing apparatus of claim 1, wherein the obtaining of the depth and color images comprises obtaining simultaneously both the depth image and the color image using the depth camera of the depth sensor and the color camera that simultaneously captures the images of the target object.

3. The image processing method of claim 1, wherein the calibrating of the depth of the depth sensor comprises:

calculating the feature point on the image plane of the projector that corresponds to a feature point with a predetermined depth value on the image plane of the depth camera,

calculating the correct feature point of the image plane of the depth camera that corresponds to the feature point of the image plane of the projector that is calculated by the first calculator when structured light is projected from an estimated correct position of the projector, and

calculating an actual depth value of the calculated correct feature point on the image plane of the depth camera.

4. The image processing method of claim 3, wherein the calculating of the feature point on the image plane of the projector comprises calculating 3-dimensional (3D) coordinates corresponding to the feature point on the image plane of the depth camera, and calculating a feature point on the image plane of the projector onto which the calculated 3D coordinates are projected.

5. The image processing method of claim 4, wherein the calculating of the 3D coordinates comprises calculating the 3D coordinates using a rotation matrix and a translation vector, which represent a relative position between the projector and the depth camera, and intrinsic parameters of the depth camera.

6. The image processing method of claim 3, wherein the calculating of the correct feature point on the image plane of the depth camera comprises:

calculating a rotation matrix correction value and a translation vector correction value,

calculating a corrected epipolar line on the image plane of the depth camera using the calculated feature point on the image plane of the projector, the calculated rotation matrix correction value, and the calculated translation vector correction value, and

searching for a new feature point on the corrected epipolar line based on the rotation matrix correction value and the translation vector correction value, wherein the new feature point corresponds to a correct feature point on the image plane of the depth camera that corresponds to the feature point on the image plane of the projector.

7. The image processing method of claim 6, wherein the calculating of the rotation matrix correction value and the translation vector correction value comprises:

computing a homography, and

calculating a rotation matrix correction value and a translation vector correction value which minimize a cost function.

8. The image processing method of claim 7, wherein the homography is computed using the rotation matrix correction value and the translation vector correction value, and the correct feature point on the image plane of the depth camera is obtained by multiplying the calculated feature point on the image plane of the projector by the homography.

9. The image processing method of claim 7, wherein the calculating of the rotation matrix correction value and translation vector correction value which minimize the cost function comprises calculating an optimal rotation matrix correction value and an optical translation vector correction value that minimize a cost function that represents a difference between a correct feature point on the corrected epipolar line on the image plane of the depth camera and a feature point on an epipolar line before correction.

10. The image processing method of claim 6, wherein the searching of the new feature point on the image plane of the depth camera comprises searching the corrected epipolar line on the image plane of the depth camera for a point whose horizontal pixel coordinate is the same as a horizontal pixel coordinate of a feature point on an epipolar line before correction, and then determining the found point as the correct feature point;

and the found point is an intersection between the corrected epipolar line and the perpendicular line of the feature point on the epipolar line before correction.

11. The image processing method of claim 3, wherein the calculating of the actual depth value of the correct feature point on the image plane of the depth camera comprises calculating the actual depth value of the image plane of the depth camera by triangulation using the calculated feature point on the image plane of the projector, the calculated correct feature point on the image plane of the depth camera that corresponds to the calculated feature point on the image plane of the projector, a rotation matrix correction value, a translation vector correction value, and intrinsic parameters of the depth camera.

12. The image processing method of claim 1, further comprising:

obtaining actual 3D coordinates by calibrating 3D coordinates measured by the depth camera, based on correction of depth error.

13. An image processing apparatus for calibrating a depth sensor, the image processing apparatus comprising:

a receiver configured to obtain a depth image of a target object captured by a depth sensor and a color image of the target object captured by a color camera; and

a processor configured to correct a depth of the depth sensor by calibrating a geometrical relation between a projector and a depth camera, which are included in the depth sensor, based the obtained depth and color images, and calculate a correct feature point on an image plane of the depth camera that corresponds to a feature point of an image plane of the projector.

14. The image processing apparatus of claim 13, wherein the processor comprises a first calculator configured to calculate a feature point on an image plane of the projector that corresponds to a feature point with a predetermined depth value on an image plane of the depth camera,

a second calculator configured to calculate a correct feature point on the image plane of the depth camera that corresponds to the feature point on the image plane of the projector that is calculated by the first calculator when structured light is projected from an estimated correct position of the projector, and

a third calculator configured to calculate an actual depth value and actual 3D coordinates of the correct feature point on the image plane of the depth camera that has been calculated by the second calculator.

15. The image processing apparatus of claim 14, wherein the first, calculator is configured to calculate 3D coordinates corresponding to a feature point on the image of the depth camera, and calculate a feature point on the image plane of the projector to which the calculated 3D coordinates are projected, wherein the first calculator uses a rotation matrix and a translation vector that represent a relative position between the projector and the depth camera, and intrinsic parameters of the depth camera when calculating the 3D coordinates.

16. The image processing apparatus of claim 14, wherein the second calculator is configured to calculate a rotation matrix correction value and a translation vector correction value, calculate a corrected epipolar line on the image plane of the depth camera using the calculated rotation matrix correction value and translation vector correction value and the feature point on the image plane of the projector that has been calculated by the first calculator, and to search for a new feature point on the corrected epipolar line based on the rotation matrix correction value and the translation vector correction value, wherein the new feature point corresponds to a correct feature point on the image plane of the depth camera that corresponds to the feature point on the image plane of the projector.

17. The image processing apparatus of claim 16, wherein the second calculator is configured to calculate an optimal rotation matrix correction value and an optical translation vector correction value that minimize a cost function that represents a difference between a correct feature point on the corrected epipolar line on the image plane of the depth camera and a feature point on an epipolar line before correction.

18. The image processing apparatus of claim 17, wherein the second calculator is configured to search the corrected epipolar line on the image plane of the depth camera for a point whose horizontal pixel coordinate is the same as a horizontal pixel coordinate of a feature point on an epipolar line before correction, and then determine the found point as the correct feature point;

and the found point is an intersection between the corrected epipolar line and the perpendicular line of the feature point on the epipolar line before correction.

19. The image processing apparatus of claim 14, wherein the third calculator is configured to calculate the actual depth value of the image plane of the depth camera by triangulation using the calculated feature point on the image plane of the projector, the calculated correct feature point on the image plane of the depth camera that corresponds to the calculated feature point on the image plane of the projector, a rotation matrix correction value, a translation vector correction value, and intrinsic parameters of the depth camera.