Temperature Compensation for Structured Light Depth Imaging System

• US 20180061056A1
• Filed: 08/30/2016
• Published: 03/01/2018
• Est. Priority Date: 08/30/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Zhao, Jian, Price, Raymond Kirk, Bleyer, Michael, Demandolx, Denis
• Granted Patent: US 10,089,738 B2
• Time in Patent Office: Days
• CPC Class Codes: G01B 11/22 for measuring depth
• CPC Class Codes: G01B 11/2513 with several lines being projected in more than one direction, e.g. grids, patterns
• CPC Class Codes: G01B 5/0014 due to temperature on machine tools B23Q11/0003
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 5/60 using machine learning, e.g. neural networks
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04N 13/20 Image signal generators
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/271 : wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 23/51 : Housings

Abstract

Disclosed are an apparatus and a method of compensating temperature shifts of a structured light pattern for a depth imaging system. In some embodiments, a depth imaging device includes a light source, an imaging sensor and a processor. The light source emits light corresponding to a pattern. A temperature drift of the light source can cause a shift of the pattern. The imaging sensor receives the light reflected by environment in front of the depth imaging device and generates a depth map including a plurality of pixel values corresponding to depths of the environment relative to the depth imaging device. The processor estimates the shift of the pattern based on a polynomial model depending on the temperature drift of the light source. The processor further adjusts the depth map based on the shift of the pattern.

Claims

1. A depth imaging device comprising:

a light source to emit light corresponding to a predetermined pattern;

an imaging sensor to receive the light reflected by the environment of the depth imaging device and that, when in operation, generates a depth map including a plurality of pixel values corresponding to depths of the environment relative to the imaging device; and

a processor that, when in operation, estimates a shift of the pattern by using a polynomial model depending on the temperature drift of the light source and adjusts the depth map based on the shift of the pattern.

2. The device of claim 1, wherein a temperature drift of the light source, an optical component of the depth imaging device, or an environment of the depth imaging device causes a shift of the pattern.

3. The device of claim 1, further comprising:

a temperature sensor to measure a temperature of the light source.

4. The device of claim 1, wherein the light source is a structured light source to emit light corresponding to a known pattern.

5. The device of claim 1, wherein the pattern is a speckle pattern corresponding to a reference image including a plurality of dots, each of the dots having known coordinates in the reference image.

6. The device of claim 1, wherein the polynomial model is used to compensate for shifts in the pattern for generating an accurate depth map.

7. The device of claim 1, where the polynomial model includes a first polynomial and a second polynomial, the first and second polynomials being functions of a plurality of variables, the variables including a temperature drift from a reference temperature and positional coordinates in the reference image.

8. The device of claim 7, wherein the pattern includes a plurality of objects and the polynomial model includes the first and second polynomials to estimate shifts of the objects of the pattern due to the temperature drift.

9. The device of claim 7, wherein each of the first and second polynomials includes a plurality of cubic terms to model interactions between the positional coordinates in the reference image and the temperature drift, each of the cubic terms including a product of multiplying three variables.

10. The device of claim 7, wherein each of the first and second polynomials is a weighted combination of terms depending on the temperature drift, a first positional coordinate in the reference image, or a second positional coordinate in the reference image;

and

wherein the processor trains weight coefficients of the first and second polynomials using a training data set, the training data set including values of the positional coordinates in the reference image, values of the temperature drifts, and corresponding known values of shifted positional coordinates in shifted images that correspond to shifts of the pattern due to the temperature drift.

11. The device of claim 1, wherein the pattern includes a plurality of objects;

and

wherein the polynomial model includes a plurality of function sets, each of the function sets is predictive of shifts of one of the objects of the pattern due to the temperature drift, each of the function sets including a function predictive of a shift along a first direction and another function predictive of a shift along a second direction.

12. The device of claim 11, wherein each function of the function sets includes a plurality of quadratic terms to model interactions between the positional coordinates in the reference image and the temperature drift, each of the quadratic terms including a product of multiplying two variables.

13. The device of claim 11, wherein each function of the function sets is a weighted combination of terms depending on the temperature drift, a first coordinate in the reference image, or a second coordinate in the reference image;

and

wherein weight coefficients of each function are trained based on a plurality of known shifts of the pattern due to the temperature drift.

14. The device of claim 11, wherein the plurality of function sets are predictive of shifts of a subset of sampling objects within the pattern;

and

wherein the polynomial model estimates a shift of a remaining object using one function set that corresponds to a sampling object within the subset, the sampling object in proximity to the remaining object in the reference image.

15. A method of compensating for temperature shifts of a structured light pattern of a depth imaging system, the method comprising:

emitting light corresponding to a known pattern by a structured light source, the known pattern including a plurality of objects that have shifts in the pattern due to a temperature drift of the structured light source;

receiving the light reflected by an environment of the depth imaging device;

generating, based on the light, a depth map including a plurality of pixel values corresponding to depths of objects in the environment relative to the depth imaging device;

predicting the temperature shifts of the objects of the pattern based on a polynomial model; and

adjusting the depth map based on the temperature shifts of the objects of the patterns.

16. The method of claim 15, further comprising:

identifying a location of an object in the environment by using the depth map that has been adjusted.

17. The method of claim 15, further comprising:

training the polynomial model by optimizing model parameters of the polynomial model using a training data set, the training data set including known shifts of the objects due to temperature drifts and corresponding temperature drifts from a reference temperature.

18. The method of claim 15, wherein the polynomial model includes a first polynomial predictive of shifts in a first direction and a second polynomial predictive of shifts in a second direction, each of the first and second polynomials including a plurality of cubic terms, each of the cubic terms including a product of multiplying three variables that include a temperature drift from a reference temperature, a x coordinate in a reference image corresponding to the pattern, or a y coordinate in the reference image;

and

wherein the predicting comprises;

predicting the temperature shifts of the objects of the pattern by applying the first and second polynomials to each of the objects of the pattern.

19. The method of claim 15, wherein the polynomial model includes a plurality of polynomial sets, each of the polynomial sets predictive of shifts of one of the objects of the pattern due to temperature drift, each of the polynomial sets including a polynomial to estimate a shift along a first direction and another polynomial to estimate a shift along a second direction;

and

wherein the predicting comprises;

predicting the temperature shifts of the objects of the pattern by applying each polynomial set of the plurality of polynomial sets to a corresponding object of the objects of the pattern.

20. A head-mounted display device comprising:

a display that, when in operation, outputs an image to an eye of a user;

a light source to emit light corresponding to a pattern including a plurality of features, wherein a temperature drift of the light source, an optical component of the head-mounted display device, or an environment of the head-mounted display device causes a shift of the pattern;

an imaging sensor to receive the light reflected by the environment of the head-mounted display device and that, when in operation, generates a depth map including a plurality of pixel values corresponding to depths of the environment relative to the head-mounted display device; and

a processor that, when in operation, estimates the shifts of the features of the pattern by using a polynomial model depending on the temperature drift of the light source and adjusts the depth map based on the shifts of the features.