Automated camera array calibration

• US 9,846,960 B2
• Filed: 08/03/2012
• Issued: 12/19/2017
• Est. Priority Date: 05/31/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kirk, Adam G., Eshet, Yaron, Eraker, David
• Primary Examiner(s): Teitelbaum, Michael
• Application Number:
  US13/566,877
• Publication Number:
  US 20130321589A1
• Time in Patent Office: 1,964 Days
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/08 Volume rendering
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2210/56 Particle system, point based geometry or rendering
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827
• CPC Class Codes: H04R 2227/005 Audio distribution systems for home, i.e. multi-room use
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction

Abstract

The automated camera array calibration technique described herein pertains to a technique for automating camera array calibration. The technique can leverage corresponding depth and single or multi-spectral intensity data (e.g., RGB (Red Green Blue) data) captured by hybrid capture devices to automatically determine camera geometry. In one embodiment it does this by finding common features in the depth maps between two hybrid capture devices and derives a rough extrinsic calibration based on shared depth map features. It then uses the intensity (e.g., RGB) data corresponding to the depth maps and uses the features of the intensity (e.g., RGB) data to refine the rough extrinsic calibration.

Claims

1. A computer-implemented process for calibrating an array of capture devices, comprising the process actions of:

using hybrid capture devices, which capture both intensity data and depth data, to simultaneously capture depth maps and corresponding intensity images of a scene;

finding common features in the depth maps from two hybrid capture devices;

automatically computing a rough calibration of extrinsic parameters of the hybrid capture devices, comprising both rotations and translations which detail the spatial location of each of the hybrid capture devices as well as the direction that the hybrid capture device is pointing using the shared depth map features.

2. The computer-implemented process of claim 1, further comprising the process actions of:

if the hybrid capture devices are not temporally synchronized,

separating the moving and non-moving data of the scene in the depth images and intensity images of the scene; and

using only the non-moving data of the scene for finding the common features.

3. The computer-implemented process of claim 1 wherein the depth maps are down sampled prior to finding the common features.

4. The computer-implemented process of claim 1, wherein once the rough calibration is found, the intensity data is used to refine the rough calibration.

5. The computer-implemented process of claim 4 wherein the intensity data is downsampled prior to using the RGB data to refine the rough calibration.

6. The computer-implemented process of claim 3, wherein the relationship between the corresponding depth map and intensity image data is used to refine the rough calibration.

7. The computer-implemented process of claim 1, wherein if a rough calibration cannot be computed,

manually scanning the scene with one capture device by hand to capture the scene,

building a model of the scene using images captured of the scene captured by the one capture device,

computing a rough calibration by comparing the depth map images captured by each camera against the created model.

8. The computer-implemented process of claim 1 wherein separating the moving and non-moving data further comprises employing an optical flow diagram.

9. A computer-implemented process for calibrating an array of capture devices, comprising the process actions of:

using hybrid capture devices, which capture both intensity data and depth data, to simultaneously capture depth maps and corresponding intensity images of a scene;

finding common features in the depth maps from two hybrid capture devices;

automatically computing a rough calibration of extrinsic parameters of the hybrid capture devices; and

using the intensity data to refine the rough calibration by,

for an intensity image captured using a current hybrid capture device,

identifying features in the intensity image;

projecting each feature onto an intensity image of another hybrid capture device using the rough calibration data obtained from matching the depth maps; and

refining the rough calibration by matching the intensity features in the intensity image captured by the current hybrid capture device to intensity features in the intensity image captured by the other hybrid capture device using feature or window based matching.

10. The computer-implemented process of claim 9 further comprising checking the refined calibration by verifying matching intensity feature points and discarding non-matching feature points.

11. The computer-implemented process of claim 10 further comprising sending the non-discarded features to a bundle adjustment algorithm to calibrate the hybrid capture devices.

12. The computer-implemented process of claim 9 wherein the depth map data is obtained by using two IR cameras in a given hybrid capture device, and the intensity data is obtained by a RGB camera in the same given hybrid capture device.

13. The computer-implemented process of claim 9 wherein the rough calibration and the refined calibration are repeated using progressively less down sampled depth maps and RGB images.