Automatic Scene Calibration

• US 20150181198A1
• Filed: 01/14/2013
• Published: 06/25/2015
• Est. Priority Date: 01/13/2012
• Status: Active Grant

• Current Assignee: SoftKinetic Software
• Original Assignee: SoftKinetic Software
• Inventors: Baele, Xavier, Martinez Gonzalez, Javier
• Granted Patent: US 9,578,310 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/246 Calibration of cameras

Abstract

Described herein is a method of calibrating a three-dimensional imaging system. During calibration, a position and an orientation of the three-dimensional imaging system is determined with respect to a first parameter comprising a real world vertical direction (V_w) and to a second parameter comprising an origin of a three-dimensional scene captured by the imaging system. The first and second parameters are used to derive a calibration matrix (M_C2w) which is used to convert measurements from a virtual coordinate system (M_c) of the three-dimensional imaging system into a real coordinate system (M_w) related to the real world. The calibration matrix (M_C2w) is used to rectify measurements prior to signal processing. An inverse calibration matrix (M_w2c) is also determined. Continuous monitoring and adjustment of the setup of the three-dimensional imaging system is carried out and the calibration matrix (M_c2w) and its inverse (M_w2c) are adjusted accordingly.

Claims

1. A method of calibrating a three-dimensional time-of-flight imaging system in a three-dimensional environment, the method comprising the steps of:

a) determining a reference orthogonal virtual three-dimensional coordinate system (M_c) for the three-dimensional imaging system, the reference orthogonal virtual three-dimensional coordinate system (M_c) having horizontal, vertical and depth axes in which the horizontal axis and the vertical axis are respectively aligned with the horizontal and vertical axes of a sensor in the three-dimensional imaging system with the depth axis being orthonormal to the plane of the sensor defined by its horizontal and vertical axes;

b) obtaining a vertical direction (V_w) of the real world in the virtual coordinate system (M_c);

c) determining with respect to the reference coordinate system a real world three-dimensional orthonormal coordinate system (M_w) having horizontal, vertical and depth axes in which the vertical axis is rotated to align it with respect to the vertical direction (V_w);

d) determining a point in the scene as a new origin for the real world three-dimensional orthonormal coordinate system (M_w);

e) deriving a translation vector (T_v) from the origin of the virtual three-dimensional coordinate system (M_c) to the point defined as the new origin of the scene;

f) deriving a rotation matrix (M_r) for transforming the virtual three-dimensional coordinate system into the real world three-dimensional coordinate system; and

g) deriving a calibration matrix (M_c2w) for the three-dimensional imaging system as the rotation matrix translated by the translation vector;

characterised in that the method further comprises the step of aligning a plane defined by the vertical and depth axes of the real world three-dimensional coordinate system to be coplanar with a plane defined by the virtual vertical axis and the virtual depth axis of the virtual three-dimensional coordinate system (M_c).

2. A method according to claim 1, wherein step g) further comprises deriving an inverse calibration matrix (M_w2c) from the calibration matrix (M_c2w) for transforming the real world three-dimensional coordinate system into the virtual three-dimensional coordinate system.

3. A method according to claim 1, wherein step b) comprises deriving the vertical direction (V_w) of the real world as the opposite vector of a gravity vector determined using a measurement device, the measurement device comprising at least one inertial measurement unit.

4. A method according to claim 1, wherein step b) comprises deriving the vertical direction (V_w) of the real world from a normal to a plane in the scene, the plane in the scene being determined by the steps of:

i) capturing the scene with the three-dimensional imaging system in a first orientation;

ii) determining a plurality of planes within the scene using a plane fitting algorithm; and

iii) determining a reference plane within the scene as being the ground.

5. A method according to claim 4, wherein step iii) comprises determining the reference plane as one best satisfying:

a statistical mode;

the largest surface; and

a minimum surface area.

6. A method according to claim 4, wherein step iii) comprises determining the reference plane as a combination of:

a statistical mode of the principal component analysis of the scene;

the largest surface; and

a minimum surface area.

7. A method according to claim 1, wherein step b) comprises deriving the vertical direction (V_w) of the real world from a specific user stance within the scene, the vertical direction (V_w) being the direction that is aligned with the vertical direction of a user standing in a predefined calibration posture.

8. A method according to claim 7, further comprising the steps of deriving a horizontal axis and a vertical axis from the predefined calibration posture, and aligning the real world three-dimensional coordinate system (M_w) with the derived horizontal and vertical axes.

9. A method according to claim 1, wherein step b) comprises deriving the vertical direction (V_w) of the real world from edges detected within the scene.

10. A method according to claim 1, wherein step b) comprises refining the vertical direction (V_w) by combining two or more of the steps of:

deriving the vertical direction (V_w) of the real world as the opposite vector of a gravity vector determined using a measurement device;

deriving the vertical direction (V_w) of the real world from a normal to a plane in the scene;

deriving the vertical direction (V_w) of the real world from a specific user stance within the scene, the vertical direction (V_w) being the direction that is aligned with the vertical direction of a user standing in a predefined calibration posture; and

deriving the vertical direction (V_w) of the real world from edges detected within the scene.

11. A method according to claim 1, wherein step d) comprises defining the point for the new origin using one of:

a predefined point in space;

the lowest location of the points defining a user;

a point on a detected plane; and

the location of a predetermined object within the scene.

12. A method according to claim 1, further comprising the step of automatically refining the calibration matrix (M_e2w) if changes are detected in at least one of:

the position and the orientation of the three-dimensional imaging system with respect to the three-dimensional environment, at least one axis of the virtual coordinate system (M_c) being aligned with a plane defined by two axes in the real world coordinate system (M_w).

13. A method according to claim 12, further comprising the step of aligning at least the vertical axis of the virtual coordinate system (M_c) with the Y-Z plane of the real world coordinate system (M_w) by controlling a motorised system supporting the three-dimensional imaging system.

14. A depth sensing imaging system having inertial measurement unit, the depth sensing imaging system operating in accordance with the method according to claim 1.

15. A depth sensing imaging system having motorised means for adjusting at least one of:

position and orientation of the depth sensing imaging system in accordance with operation of the method according to claim 1.

16. A three-dimensional time of flight imaging system having calibration means operating in accordance with the method of claim 1.