Deformation Detection and Automatic Calibration for a Depth Imaging System

• US 20180061034A1
• Filed: 08/30/2016
• Published: 03/01/2018
• Est. Priority Date: 08/30/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Zhao, Jian, Bleyer, Michael, Price, Raymond Kirk, Demandolx, Denis
• Granted Patent: US 10,560,679 B2
• Time in Patent Office: Days
• CPC Class Codes: G01B 11/2513 with several lines being projected in more than one direction, e.g. grids, patterns
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 2207/30208 Marker matrix
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays

Abstract

Disclosed are an apparatus and a method for detecting deformation of a structured light depth imaging system and automatic re-calibration for the depth imaging system. In one embodiment, a depth imaging device includes a light projector, a camera and a processor. The light projector emits light corresponding to a projected pattern image having a plurality of features with known locations in the projected pattern image. The camera captures the light reflected by an environment of the depth imaging device and generates a reflected pattern image as a two-dimensional (2D) projection of the environment. The reflected pattern image includes a plurality of features that correspond to the features of the projected pattern image. The processor detects a misalignment for a distance or an orientation between the light projector and the camera based on a relationship between the features in the projected pattern image and the features in the reflected pattern image.

Claims

1. A depth imaging device comprising:

a light projector that, when in operation, emits light corresponding to a projected pattern image having a plurality of features with known illumination pattern in the projected image;

a camera that, when in operation, captures the light after reflection of the light by an environment of the depth imaging device and generates a reflected pattern image as a two-dimensional (2D) projection of the environment, the reflected pattern image including a plurality of features that correspond to the features of the projected image; and

a processor to detect a misalignment of a distance or an orientation between the light projector and the camera based on a relationship between the features in the projected image and the features in the reflected pattern image.

2. The device of claim 1, wherein the processor corrects the misalignment by updating a set of calibration parameters indicative of the distance or the orientation between the light projector and the camera.

3. The device of claim 1, wherein the processor searches the projected image and the reflected pattern image for pairs of matching features, each pair including a feature from the projected image and a feature from the reflected pattern image, wherein a search result is indicative of a presence of a physical distortion to the depth imaging device.

4. The device of claim 3, wherein the processor generates a depth map including a plurality of pixel values corresponding to depths of surfaces in the environment relative to the depth imaging device, based on the pairs of matching features and a set of updated calibration parameters indicative of the distance or the orientation between the light projector and the camera.

5. The device of claim 1, wherein the processor detects the misalignment based on a determination that a spatial distribution of the features in the reflected pattern image is shifted above or below the horizontal epipolar scanline.

6. The device of claim 1, wherein the projected image includes a plurality of random dots that are evenly distributed in the projected image;

and

wherein the processor detects the misalignment based on a determination that a spatial distribution of the features in the reflected pattern image is shifted with respect to the known illumination pattern in the projected image.

7. The device of claim 1, wherein for an individual feature at a scanline in the projected image, the processor searches for a matching feature corresponding to the individual feature of the projected image at the scanline in the reflected pattern image;

and

wherein the processor detects the misalignment based on a determination that the individual feature at the scanline in the projected image does not have a matching feature at the scanline in the reflected pattern image.

8. The device of claim 1, wherein the processor searches the projected image and the reflected pattern image for pairs of matching features, each pair including a feature located at a scanline in the projected image and another feature located at the scanline in the reflected pattern image;

and

wherein the processor detects the misalignment based on a determination that an average of matching scores of the pairs of matching features is below a threshold value.

9. The device of claim 1, wherein for an individual feature at a scanline in the projected image, the processor searches for a first matching feature corresponding to the individual feature of the projected image at the scanline in the reflected pattern image;

and

wherein the processor detects the misalignment based on a determination that a second matching feature in the reflected pattern image is not at the scanline and is a better match for the individual feature than the first matching feature at the scanline in the reflected pattern image.

10. The device of claim 1, wherein the processor identifies a pair of matching features, the pair of matching features including a first feature from the projected image and a second feature from the reflected pattern image that is a match of the first feature;

and

wherein the processor corrects the misalignment by updating a set of calibration parameters indicative of the distance or the orientation between the light projector and the camera, based on the pair of matching features and an object in the environment corresponding to the matching features.

11. The device of claim 10, wherein the processor updates the set of calibration parameters by minimizing a function indicative of differences between known coordinates of the pair of matching features and calculated coordinates of projections of the object onto the projected image and the reflected pattern image.

12. A method for detecting a deformation of a depth imaging device, the method comprising:

emitting, by a light projector, light corresponding to a projected pattern image having a plurality of features with known locations in the projected pattern image;

capturing, by a camera, the light reflected by environment of the depth imaging device;

generating a reflected pattern image as a two-dimensional (2D) projection of the environment, the reflected pattern image including a plurality of features that correspond to the features of the projected pattern image; and

detecting a misalignment for a distance or an orientation between the light projector and the camera based on a relationship between the features in the projected pattern image and the features in the reflected pattern image.

13. The method of claim 12, further comprising:

performing a calibration process to update a set of calibration parameters indicative of the distance or the orientation between the light projector and the camera.

14. The method of claim 12, further comprising:

searching the projected pattern image and the reflected pattern image for pairs of matching features, each pair including a feature from the projected pattern image and another feature from the reflected pattern image.

15. The method of claim 14, further comprising:

generating a depth map including a plurality of pixel values corresponding to depths of the environment relative to the depth imaging device, based on the pairs of matching features and a set of calibration parameters indicative of the distance or the orientation between the light projector and the camera.

16. The method of claim 12, wherein the detecting comprises:

detecting the misalignment based on a determination that a spatial distribution of the features in the reflected pattern image is biased toward a direction relative to a spatial distribution of the features in the projected pattern image.

17. The method of claim 12, wherein the detecting comprises:

detecting the misalignment based on a determination that for an individual feature at a scanline in the projected pattern image, and best match of the individual feature is not at the same scanline in the reflected pattern image.

18. A head-mounted display device comprising:

a display that, when in operation, outputs an image to an eye of a user;

a light projector that, when in operation, emits light corresponding to a projected pattern image having a plurality of features with known locations in the projected pattern image;

a camera that, when in operation, captures the light after reflection of the light by an environment of the head-mounted display device and generates a reflected pattern image as a two-dimensional (2D) projection of the environment, the reflected pattern image including a plurality of features that correspond to the features of the projected pattern image; and

a processor, when in operation, performs a calibration process to update a set of calibration parameters indicative of a distance or an orientation between the light projector and the camera based on a relationship between the features in the projected pattern image and the features in the reflected pattern image.

19. The device of claim 18, wherein the processor, when in operation, generates a depth map including a plurality of pixel values corresponding to depths of objects of the environment relative to the head-mounted display device, based on the set of calibration parameters that has been updated and the features in the projected pattern image and the reflected pattern image.

20. The device of claim 18, wherein the processor, when in operation, minimizes a function indicative of differences between known coordinates of a pair of matching features corresponding to an object of the environment and calculated coordinates of projections of the object onto the projected pattern image and the reflected pattern image, wherein the function depends on the calibration parameters indicative of the distance or the orientation between the light projector and the camera.