Learning from high quality depth measurements

• US 9,470,778 B2
• Filed: 03/29/2011
• Issued: 10/18/2016
• Est. Priority Date: 03/29/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Gilboa, Guy
• Primary Examiner(s): Jebari, Mohammed
• Application Number:
  US13/075,060
• Publication Number:
  US 20120249738A1
• Time in Patent Office: 2,030 Days
• US Class Current: 1/1
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 7/497 Means for monitoring or calibrating
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 13/20 Image signal generators
• CPC Class Codes: H04N 17/002 for television cameras

Abstract

A depth camera computing device is provided, including a depth camera and a data-holding subsystem holding instructions executable by a logic subsystem. The instructions are configured to receive a raw image from the depth camera, convert the raw image into a processed image according to a weighting function, and output the processed image. The weighing function is configured to vary test light intensity information generated by the depth camera from a native image collected by the depth camera from a calibration scene toward calibration light intensity information of a reference image collected by a high-precision test source from the calibration scene.

Claims

1. At an image processing module of a computing device compatible with a depth camera, a method of calibrating the depth camera, the method comprising:

receiving from the depth camera a native image including test light intensity information useable to derive a native depth map for a calibration scene;

receiving from a high-precision test source a respective reference image taken from a same perspective as the native image and including calibration light intensity information useable to derive a reference depth map for the calibration scene independently from the native depth map; and

fitting a weighting function to vary test light intensity information toward the calibration light intensity information for the calibration scene, the weighting function configured to scale light intensity information for a plurality of raw images received by the depth camera during a subsequent operation mode wherein the depth camera is operated independently from the high-precision test source.

2. The method of claim 1, wherein the calibration scene is one of a plurality of calibration scenes, and wherein the weighting function is one of a plurality of local weighting functions, each local weighting function being fit to vary test light intensity information toward the calibration light intensity information for a particular calibration scene of the plurality of calibration scenes.

3. The method of claim 2, further comprising generating a global weighting function for the depth camera from the plurality of local weighting functions.

4. The method of claim 1, wherein receiving the respective reference image from the high-precision test source includes receiving the respective reference image from a high-resolution time-of-flight camera system.

5. The method of claim 1, wherein receiving the respective reference image includes receiving the respective reference image from a 3-D triangulation system.

6. The method of claim 1, wherein receiving the respective reference image includes receiving the respective reference image from a laser light detection and ranging system.

7. The method of claim 1, wherein the weighting function is a function of one or more of a variance, a moment, and a total variation between a pixel included in the native image and a respective pixel included in the respective reference image.

8. The method of claim 1, wherein fitting the weighting function includes minimizing a difference between the test light intensity information and the calibration light intensity information.

9. The method of claim 8, wherein minimizing the difference includes minimizing a difference between the native depth map for a particular pixel of the native image and the reference depth map for a respective pixel included in the respective reference image.

10. A computing device configured to calibrate a depth camera, the computing device comprising:

a depth camera input configured to receive an image collected by the depth camera;

a logic subsystem; and

a data-holding subsystem holding instructions executable by the logic subsystem to;

receive from the depth camera a native image including test light intensity information useable to derive a native depth map for each of a plurality of calibration scenes,

receive from a high-precision test source that is different from the depth camera, a respective reference image taken from the same perspective as the native image including calibration light intensity information useable to derive a reference depth map for each of the plurality of calibration scenes,

for each of the plurality of calibration scenes, fit a local weighting function to vary test light intensity information toward the calibration light intensity information, and

generate a global weighting function for the depth camera from the local weighting functions.

11. The computing device of claim 10, wherein the global weighting function is a function of one or more of a variance, a moment, and a total variation between a pixel included in the native image and a respective pixel included in the respective reference image.

12. The computing device of claim 10, wherein fitting the local weighting function includes minimizing a difference between the test light intensity information and the calibration light intensity information.

13. The computing device of claim 12, wherein minimizing the difference includes minimizing a difference between the native depth map for a particular pixel of the native image and the reference depth map for a respective pixel included in the respective reference image.

14. The computing device of claim 10, wherein the high-precision test source includes one of a high-resolution time-of-flight camera system, a 3-D triangulation system, and a laser light detection and ranging system, and wherein the high-precision test source is configured to generate depth information for a particular calibration scene that is reliably more accurate than depth information generated by the depth camera.

15. A depth camera computing device including an image processing module configured to output a processed image by application of a global weighting function to a raw image, the depth camera computing device comprising:

a depth camera; and

a data-holding subsystem holding instructions executable by a logic subsystem to;

receive a raw image of a use environment from the depth camera,

convert the raw image into a processed image according to a weighting function and without receiving a corresponding reference image of the use environment, the weighting function generated by varying test light intensity information generated by the depth camera from a native image previously collected by the depth camera from a calibration scene toward calibration light intensity information of a reference image collected by a high-precision test source, different than the depth camera, from a same perspective as the native image, and

output the processed image.

16. The depth camera of claim 15, wherein the test light intensity information is useable to derive a native depth map for the native image, and wherein the calibration light intensity information is useable to derive a reference depth map for the reference image.

17. The depth camera of claim 16, wherein the weighting function is configured to adjust depth information for a particular pixel of the native image toward depth information for a respective pixel of the reference image included in the reference depth map.

18. The depth camera of claim 16, wherein the calibration scene is one of a plurality of calibration scenes for which a native image and a respective reference image are collected, wherein the weighting function is a global weighting function generated from a plurality of local weighting functions, each local weighting function being fit to vary test light intensity information of a particular native image toward the calibration light intensity information of a particular reference image for a particular calibration scene of the plurality of calibration scenes.

19. The depth camera of claim 18, wherein the instructions are further configured to select a particular local weighting function to be used instead of the global weighting function to process the native image according to an image quality of the native image prior to the native image being so processed.

20. The depth camera of claim 15, wherein the weighting function is a function of one or more of a variance, a moment, and a total variation between a pixel included in the native image and a respective pixel included in the respective reference image.