IN-HOME DEPTH CAMERA CALIBRATION

• US 20120105585A1
• Filed: 11/03/2010
• Published: 05/03/2012
• Est. Priority Date: 11/03/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Leyvand, Tommer, Zhang, Zhengyou, Masalkar, Prafulla J., Stachniak, Szymon P., Del Castillo, Leonardo, Mathe, Zsolt
• Granted Patent: US 8,866,889 B2
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/22 Setup operations, e.g. calibration, key configuration or button assignment
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/327 Calibration thereof

Abstract

A system and method are disclosed for calibrating a depth camera in a natural user interface. The system in general obtains an objective measurement of true distance between a capture device and one or more objects in a scene. The system then compares the true depth measurement to the depth measurement provided by the depth camera at one or more points and determines an error function describing an error in the depth camera measurement. The depth camera may then be recalibrated to correct for the error. The objective measurement of distance to one or more objects in a scene may be accomplished by a variety of systems and methods.

Claims

1. A method of recalibrating a depth image sensor, the depth sensor capturing image data from a 3-D space, the method comprising:

(a) measuring a depth value of at least one sample point in the 3-D space using the depth sensor;

(b) determining a true depth value of the at least one sample point independently of the measurement of said step (a);

(c) modeling a depth error resulting from a deviation in the true depth determined in said step (b) and the depth measured in said step (a); and

(d) correcting the depth measurements made by the depth sensor using the depth error modeled in said step (c).

2. The method of claim 1, said step (b) comprising the step of imaging a rigid calibration object presented to the depth image sensor, the size of the calibration object being known to a computing device performing said step (b).

3. The method of claim 2, further comprising the step (e) of translating the position of the calibration object into a reference coordinate system of an RGB sensor associated with the depth sensor using homography.

4. The method of claim 1, said step (b) comprising the steps of receiving a user indication of an x-axis or y-axis dimensional size of an object, and correlating the input size to a distance from the depth sensor based on a known correlation between size of an object and number of pixels depicting that object at given distances between the object and the depth sensor.

5. The method of claim 4, wherein the size of the object is a user'"'"'s height.

6. The method of claim 4, wherein the size of the object is a user'"'"'s arm span.

7. The method of claim 1, said steps (a) and (b) comprising the step of measuring at least two sample points on a calibration object at least two different locations in the 3-D space.

8. The method of claim 7, said step (c) comprising the step of determining a deviation in a position of the sample points on a calibration object as measured by the depth sensor in said step (a) and the true position of the sample points determined in said step (b), and finding an error model which explains the deviation between the depth measured in said step (a) and the true depth determined in said step (b).

9. The method of claim 7, wherein the size of the calibration object is unknown to a computing device performing said step (b).

10. The method of claim 9, said step (c) comprising the steps of defining a first line between a set of points measured at a first location in the 3-D space, defining a second line between the same set of points measured at a second location in the 3-D space, defining an equation where the first and second lines are proportional to each other by an error coefficient, and solving for the error coefficient.

11. The method of claim 1, wherein a depth error which is linear at different points in the 3-d space is modeled using two sample points in steps (a) and (b).

12. The method of claim 1, wherein a depth error which is non-linear at different points in the 3-d space is modeled using more than two sample points in steps (a) and (b).

13. A computer-readable storage medium including instructions for programming a processor to perform a method of recalibrating a depth sensor of a natural user interface, the depth sensor capturing image data from a 3-D space, the method comprising:

(a) measuring the depth of two or more points on a calibration object within the 3-D space using the depth sensor;

(b) determining the true depth of the two or more points on the calibration object independently of the depth sensor;

(c) modeling a depth error resulting from a deviation in the true depth determined in said step (b) and the depth measured in said step (a); and

(d) compensating depth measurements made by the depth sensor using the depth error modeled in said step (c).

14. The computer-readable medium of claim 13, where, for linear depth errors, said step (c) comprises the step of modeling the error in the form of measured depth of said step (a)=α

(true depth of said step (b))−

α

, where the β

component represents the depth at which the sensor and the true depth are equal to each other, and α

represents the slope of a line defined between the true depth and depth measured by the depth image sensor.

15. The computer-readable medium of claim 13, where, for non-linear depth errors, said steps (a) and (b) comprise taking true and measured depth samples of the object at points distributed around the 3-D space.

16. A system for recalibrating a depth sensor of a natural user interface, the depth sensor capturing image data from a 3-D space, the system comprising:

a calibration object positioned within the 3-D space; and

a computing device associated with the depth sensor, the computing device including a recalibration engine for comparing a true depth values of points on the calibration object in the 3-D space against depth values measured by the depth sensor, and deriving an error model which describes a deviation between the true depth values and the depth values measured by the depth sensor.

17. The system of claim 16, wherein the calibration object is a rigid object having dimension known to the computing device.

18. The system of claim 16, wherein the calibration object is a calibration card having different regions having horizontal and vertical components, the regions defining a plurality of points at intersections of the horizontal and vertical components.

19. The system of claim 18, the system further comprising an RGB sensor, the computing device including an algorithm for determining the locations of the plurality of points in a reference plane in a coordinate system of the RGB sensor.

20. The system of claim 19, the computing device prompting a user to position the calibration card at different positions in the 3-D space, the computing device determining the locations of the plurality of points at each position in reference planes of the RGB coordinate system.