Deformation detection and automatic calibration for a depth imaging system

• US 10,560,679 B2
• Filed: 08/30/2016
• Issued: 02/11/2020
• Est. Priority Date: 08/30/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Zhao, Jian, Bleyer, Michael, Price, Raymond Kirk, Demandolx, Denis
• Primary Examiner(s): Navas, Jr., Edemio
• Application Number:
  US15/252,129
• Publication Number:
  US 20180061034A1
• Time in Patent Office: 1,260 Days
• Field of Search: None
• CPC Class Codes: G01B 11/2513 with several lines being projected in more than one direction, e.g. grids, patterns
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 2207/30208 Marker matrix
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays

Abstract

Disclosed are an apparatus and a method for detecting deformation of a structured light depth imaging system and automatic re-calibration for the depth imaging system. In one embodiment, a depth imaging device includes a light projector, a camera and a processor. The light projector emits light corresponding to a projected pattern image having a plurality of features with known locations in the projected pattern image. The camera captures the light reflected by an environment of the depth imaging device and generates a reflected pattern image as a two-dimensional (2D) projection of the environment. The reflected pattern image includes a plurality of features that correspond to the features of the projected pattern image. The processor detects a misalignment for a distance or an orientation between the light projector and the camera based on a relationship between the features in the projected pattern image and the features in the reflected pattern image.

Claims

1. A depth imaging device comprising:

a light projector that, when in operation, emits light corresponding to a projected pattern image having a plurality of features with a known illumination pattern in the projected pattern image;

a camera that, when in operation, captures the light after reflection of the light by an environment of the depth imaging device and generates a reflected pattern image as a two-dimensional (2D) projection of the environment, the reflected pattern image including a plurality of second features that correspond to the plurality of features of the projected pattern image; and

a processor configured to;

determine a relationship of a distance or a relationship of an orientation between the plurality of features in the projected pattern image and the plurality of second features in the reflected pattern image based on an intensity dissimilarity function of the plurality of features in the projected pattern image and the plurality of second features in the reflected pattern image;

identify a set of matching features based on the relationship of the distance or the relationship of the orientation along a scanline according to a 1-dimensional (1D) scanline searching process;

identify a second set of matching features based on a second relationship of the distance or a second relationship of the orientation between the plurality of features in the projected pattern image and the plurality of second features in the reflected pattern image within a range of the scanline according to a 2-dimensional (2D) scanline searching process; and

detect a misalignment between the light projector and the camera based on the set of matching features and the second set of matching features.

2. The device of claim 1, wherein the processor corrects the misalignment by updating a set of calibration parameters indicative of the distance or the orientation between the light projector and the camera.

3. The device of claim 1, wherein the processor searches the projected pattern image and the reflected pattern image for pairs of matching features, each pair including a feature from the plurality of features of the projected pattern image and a second feature of the plurality of second features from the reflected pattern image, wherein a search result is indicative of a presence of a physical distortion to the depth imaging device.

4. The device of claim 3, wherein the processor generates a depth map including a plurality of pixel values corresponding to depths of surfaces in the environment relative to the depth imaging device, based on the pairs of matching features and a set of updated calibration parameters indicative of the distance or the orientation between the light projector and the camera.

5. The device of claim 1, wherein the processor detects the misalignment based on a determination that a spatial distribution of the plurality of second features of the reflected pattern image is shifted above or below a horizontal epipolar scanline.

6. The device of claim 1, wherein the projected pattern image includes a plurality of random dots that are distributed in the projected pattern image;

and

wherein the processor detects the misalignment based on a determination that a spatial distribution of the plurality of second features of the reflected pattern image is shifted with respect to the known illumination pattern in the projected pattern image.

7. The device of claim 1, wherein for an individual feature at a scanline in the projected pattern image, the processor searches for a matching feature corresponding to the individual feature of the projected pattern image at the scanline in the reflected pattern image;

and

wherein the processor detects the misalignment based on a determination that the individual feature at the scanline in the projected pattern image does not have the matching feature at the scanline in the reflected pattern image.

8. The device of claim 1, wherein the processor searches the projected pattern image and the reflected pattern image for pairs of matching features, each pair including a feature of the plurality of features of the projected pattern image located at a scanline in the projected pattern image and a second feature of the plurality of second features of the reflected pattern image located at the scanline in the reflected pattern image;

and

wherein the processor detects the misalignment based on a determination that an average of matching scores of the pairs of matching features is below a threshold value.

9. The device of claim 1, wherein for an individual feature of the plurality of features of the projected pattern image at a scanline in the projected pattern image, the processor searches for a first matching feature corresponding to the individual feature of the projected pattern image at the scanline in the reflected pattern image;

and

wherein the processor detects the misalignment based on a determination that a second matching feature of the plurality of second features of the reflected pattern image is not at the scanline and is a better match for the individual feature than the first matching feature at the scanline in the reflected pattern image.

10. The device of claim 1, wherein the processor identifies a pair of matching features, the pair of matching features including a first feature from the plurality of features of the projected pattern image and a second feature from the plurality of features of the reflected pattern image that is a match of the first feature;

and

wherein the processor corrects the misalignment by updating a set of calibration parameters indicative of the distance or the orientation between the light projector and the camera, based on the pair of matching features and an object in the environment corresponding to the pair of matching features.

11. The device of claim 10, wherein the processor updates the set of calibration parameters by minimizing a function indicative of differences between known coordinates of the pair of matching features and calculated coordinates of projections of the object onto the projected pattern image and the reflected pattern image.

21. The device of claim 1, wherein the processor is further configured to:

calculate the intensity dissimilarity function based on one or more candidate pixels of the reflected pattern image having an intensity dissimilarity score corresponding to one or more pixels of the projected pattern image.

24. The device of claim 1, wherein the processor is further configured to:

compare a dissimilarity score of the set of matching features with a dissimilarity score of the second set of matching features; and

detect the misalignment between the light projector and the camera further based on the comparison.

12. A method for detecting a deformation of a depth imaging device, the method comprising:

emitting, by a light projector, light corresponding to a projected pattern image having a plurality of features with known locations in the projected pattern image;

capturing, by a camera, the light reflected by environment of the depth imaging device;

generating a reflected pattern image as a two-dimensional (2D) projection of the environment, the reflected pattern image including a plurality of second features that correspond to the plurality of features of the projected pattern image;

determining a relationship of a distance or a relationship of an orientation between the plurality of features in the projected pattern image and the plurality of second features in the reflected pattern image based on an intensity dissimilarity function of the plurality of features in the projected pattern image and the plurality of second features in the reflected pattern image;

identifying a set of matching features based on the relationship of the distance or the relationship of the orientation along a scanline according to a 1-dimensional (1D) scanline searching process;

identifying a second set of matching features based on a second relationship of the distance or a second relationship of the orientation between the plurality of features in the projected pattern image and the plurality of second features in the reflected pattern image within a range of the scanline according to a 2-dimensional (2D) scanline searching process; and

detecting a misalignment between the light projector and the camera based on the set of matching features and the second set of matching features.

13. The method of claim 12, further comprising:

performing a calibration process to update a set of calibration parameters indicative of the distance or the orientation between the light projector and the camera.

14. The method of claim 12, further comprising:

searching the projected pattern image and the reflected pattern image for pairs of matching features, each pair including a feature from the plurality of features of the projected pattern image and a second feature from the plurality of second features of the reflected pattern image.

15. The method of claim 14, further comprising:

generating a depth map including a plurality of pixel values corresponding to depths of the environment relative to the depth imaging device, based on the pairs of matching features and a set of calibration parameters indicative of the distance or the orientation between the light projector and the camera.

16. The method of claim 12, wherein the detecting comprises:

detecting the misalignment based on a determination that a spatial distribution of the plurality of second features in the reflected pattern image is biased toward a direction relative to a spatial distribution of the plurality of features in the projected pattern image.

17. The method of claim 12, wherein the detecting comprises:

detecting the misalignment based on a determination that for an individual feature of the plurality of features of the projected pattern image at a scanline in the projected pattern image, and best match of the individual feature is not at a same scanline in the reflected pattern image.

22. The method of claim 12, further comprising:

calculating the intensity dissimilarity function based on one or more candidate pixels of the reflected pattern image having an intensity dissimilarity score corresponding to one or more pixels of the projected pattern image.

25. The method of claim 12, further comprising:

comparing a dissimilarity score of the set of matching features with a dissimilarity score of the second set of matching features; and

detecting the misalignment between the light projector and the camera further based on the comparison.

18. A head-mounted display device comprising:

a display that, when in operation, outputs an image to an eye of a user;

a light projector that, when in operation, emits light corresponding to a projected pattern image having a plurality of features with known locations in the projected pattern image;

a camera that, when in operation, captures the light after reflection of the light by an environment of the head-mounted display device and generates a reflected pattern image as a two-dimensional (2D) projection of the environment, the reflected pattern image including a plurality of second features that correspond to the plurality of features of the projected pattern image; and

a processor, when in operation;

determines a relationship of a distance or a relationship of an orientation between the plurality of features in the projected pattern image and the plurality of second features in the reflected pattern image based on an intensity dissimilarity function of the plurality of features in the projected pattern image and the plurality of second features in the reflected pattern image;

identifies a set of matching features based on the relationship of the distance or the relationship of the orientation along a scanline according to a 1-dimensional (1D) scanline searching process;

identifies a second set of matching features based on a second relationship of the distance or a second relationship of the orientation between the plurality of features in the projected pattern image and the plurality of second features in the reflected pattern image within a range of the scanline according to a 2-dimensional (2D) scanline searching process; and

performs a calibration process to update a set of calibration parameters based on the set of matching features and the second set of matching features.

19. The device of claim 18, wherein the processor, when in operation, generates a depth map including a plurality of pixel values corresponding to depths of objects of the environment relative to the head-mounted display device, based on the set of calibration parameters that has been updated and the plurality of features in the projected pattern image and the plurality of second features of the reflected pattern image.

20. The device of claim 18, wherein the processor, when in operation, minimizes a function indicative of differences between known coordinates of a pair of matching features corresponding to an object of the environment and calculated coordinates of projections of the object onto the projected pattern image and the reflected pattern image, wherein the function depends on the set of calibration parameters indicative of the distance or the orientation between the light projector and the camera.

23. The device of claim 18, wherein the processor, when in operation, calculates the intensity dissimilarity function based on one or more candidate pixels of the reflected pattern image having an intensity dissimilarity score corresponding to one or more pixels of the projected pattern image.

26. The device of claim 18, wherein the processor, when in operation,

compares a dissimilarity score of the set of matching features with a dissimilarity score of the second set of matching features;

and

performs the calibration process to update the set of calibration parameters further based on the comparison.