Calibration and registration of camera arrays using a single circular grid optical target

• US 9,210,404 B2
• Filed: 12/14/2012
• Issued: 12/08/2015
• Est. Priority Date: 12/14/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Hall, Michael
• Primary Examiner(s): Nguyen, Huy T
• Application Number:
  US13/715,723
• Publication Number:
  US 20140168378A1
• Time in Patent Office: 1,089 Days
• US Class Current: 1/1
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30208 Marker matrix
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps

Abstract

Methods for determining intrinsic parameters associated with a camera and for registering cameras within a camera array are described. In some embodiments, a camera may be calibrated using an optical target that is positioned at a depth outside the depth of focus of the camera. The optical target may include a grid of symmetrical shapes (e.g., a rectilinear grid of circles or tilted squares). The intrinsic parameters for the camera may be determined using an iterative calibration process in which a cost function is used to evaluate the straightness of lines intersecting rows and columns associated with centroids of the symmetrical shapes. In some embodiments, the registration of a color camera with a depth camera may include mapping centroids identified within a first color image captured by the color camera with corresponding centroids identified within an undistorted intensity image captured by the depth camera.

Claims

1. A method for calibrating a camera array, comprising:

capturing a first image of an environment including an optical target using a first camera, the optical target includes a plurality of symmetrical shapes;

determining a plurality of centroid locations within the first image corresponding with the plurality of symmetrical shapes;

determining one or more distortion parameters associated with the first camera based on the plurality of centroid locations;

generating a reduced distortion image using the first image and the one or more distortion parameters;

outputting the reduced distortion image;

capturing a second image of the optical target using a second camera different from the first camera;

determining a registration mapping for the second image onto the reduced distortion image;

generating a registered image based on the second image and the registration mapping; and

outputting the registered image.

2. The method of claim 1, wherein:

the optical target is positioned within the environment at a depth outside a depth of focus of the first camera.

3. The method of claim 1, wherein:

the plurality of symmetrical shapes includes a plurality of circles.

4. The method of claim 1, wherein:

the determining a plurality of centroid locations includes thresholding the first image and determining a center of mass for each of the plurality of symmetrical shapes.

5. The method of claim 1, further comprising:

determining a center of distortion based on the one or more distortion parameters; and

determining a principal point based on the center of distortion.

6. The method of claim 1, further comprising:

determining an average distance between neighboring centroids within the reduced distortion image; and

determining a focal length associated with the first camera based on the average distance between neighboring centroids.

7. The method of claim 1, wherein:

the determining one or more distortion parameters associated with the first camera includes optimizing the one or more distortion parameters using a cost function that evaluates the straightness of lines intersecting rows and columns associated with the plurality of centroids.

8. The method of claim 1, further comprising:

determining depth information associated with the reduced distortion image, the first camera comprises a depth camera, the registration mapping includes one or more registration mapping coefficients; and

adjusting the one or more registration mapping coefficients based on the depth information.

9. The method of claim 1, wherein:

the outputting the reduced distortion image includes displaying the reduced distortion image;

the first camera comprises a depth camera; and

the second camera comprises a color camera.

10. A system for calibrating a camera array, comprising:

a depth camera, the depth camera to capture a first image of an environment including an optical target, the optical target includes a plurality of symmetrical shapes;

one or more processors, the one or more processors to communicate with the depth camera, the one or more processors to determine a plurality of centroid locations within the first image corresponding with the plurality of symmetrical shapes, the one or more processors to determine one or more distortion parameters associated with the depth camera based on the plurality of centroid locations, the one or more processors to generate a reduced distortion image using the first image and the one or more distortion parameters; and

a color camera to communicate with the one or more processors, the color camera to capture a second image of the environment including the optical target, the one or more processors to determine a registration mapping for the second image onto the reduced distortion image, the one or more processors to generate a registered image by applying the registration mapping to the second image.

11. The system of claim 10, wherein:

the optical target is positioned within the environment at a depth outside a depth of focus of the depth camera.

12. The system of claim 10, wherein:

the plurality of symmetrical shapes includes a plurality of circles.

13. One or more storage devices containing processor readable code for programming one or more processors to perform a method for calibrating a camera array comprising the steps of:

capturing a first image of an environment including an optical target using a depth camera, the optical target includes a plurality of symmetrical shapes;

determining a plurality of centroid locations within the first image corresponding with the plurality of symmetrical shapes;

determining one or more intrinsic parameters associated with the depth camera based on the plurality of centroid locations;

generating a reduced distortion image using the first image and the one or more intrinsic parameters;

capturing a second image of the optical target using a color camera;

determining a registration mapping for the second image onto the reduced distortion image;

generating a registered image using the second image and the registration mapping; and

outputting the reduced distortion image and the registered image.

14. The one or more storage devices of claim 13, wherein:

the optical target is positioned within the environment at a depth outside a depth of focus of the depth camera.

15. The one or more storage devices of claim 13, wherein:

the plurality of symmetrical shapes includes a plurality of circles.

16. The one or more storage devices of claim 13, wherein:

the determining a plurality of centroid locations includes thresholding the first image and determining a center of mass for each of the plurality of symmetrical shapes.

17. The one or more storage devices of claim 13, wherein:

the outputting the undistorted image and the registered image includes displaying the reduced distortion image.

18. The one or more storage devices of claim 13, further comprising:

determining an average distance between neighboring centroids within the reduced distortion image; and

determining a focal length associated with the depth camera based on the average distance between neighboring centroids.