Registering cameras in a multi-camera imager

• US 10,027,954 B2
• Filed: 05/23/2016
• Issued: 07/17/2018
• Est. Priority Date: 05/23/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Hall, Michael Anthony
• Primary Examiner(s): Teitelbaum, Michael
• Assistant Examiner(s): DESIR, JEAN WICEL
• Application Number:
  US15/161,310
• Publication Number:
  US 20170339400A1
• Time in Patent Office: 785 Days
• Field of Search: 348188, 348187, 348180, 34833308, 348139
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A method of registering first and second cameras in a multi-camera imager comprising imaging a calibration target having rows and columns of fiducials with both the first and second cameras and using convergence angles of images of the rows and columns in the images to register the cameras.

Claims

1. A method of registering a first camera to a second camera of a multi-camera imager (MCI), each camera modeled as a pin hole camera having a center of projection, an optical axis extending from the center of projection, a camera coordinate system having an origin at the center of projection and a Z axis coincident with the optical axis, an image plane intersected by the optical axis at a principal point, and a photosensor comprising pixels on which the camera images scenes, the method comprising:

operating the first and second cameras to acquire first and second images respectively of a calibration target located at a known position relative to the MCI and comprising fiducials that define at least two lines in the calibration target;

determining at least one angle between two lines defined by images of the fiducials in a projection of the image of the calibration target acquired by at least one of the first and second cameras on the image plane of the camera; and

using the at least one angle to register the first and second cameras to each other.

2. The method according to claim 1 wherein using the at least one angle comprises determining an orientation angle between the optical axis of the at least one camera and a normal to a plane including the at least two lines in the calibration target.

3. The method according to claim 1 wherein the at least two lines in the calibration target comprise at least two parallel first lines, and wherein determining at least one angle comprises determining a convergence angle α

between two lines defined by the images of the fiducials in the projection of the image.

4. The method according to claim 3 wherein determining the orientation angle comprises determining a pan angle γ

of rotation of the camera based on the convergence angle α

.

5. The method according to claim 4 and comprising determining the pan angle γ

based on the expression tan α

=(Δ

Y_o/Z_o)tan γ

where Δ

Y_o is a distance between the two parallel first lines in the calibration target and Z_o is a distance between the projection point of the at least one camera and an intersection of the plane of the two parallel first lines in the calibration target and the optical axis of the at least one camera.

6. The method according to claim 4 and comprising determining the pan angle γ

based on an expression γ

=arc tan(f/x) where f represents the focal length of the camera and x represents a distance from the principal point of the at least one camera of an intersection of the lines that determine the angle α

.

7. The method according to claim 6 wherein the at least two lines in the calibration target comprise at least two parallel second lines that are not parallel to the first lines and wherein determining at least one angle comprises determining a convergence angle β

between two lines defined by images of the fiducials that define the second lines in the projection of the image.

8. The method according to claim 7 wherein the at least two second parallel lines are perpendicular to the at least two first parallel lines.

9. The method according to claim 7 wherein determining the orientation angle comprises determining a tilt angle δ

of rotation of the camera based on the convergence angle β

.

10. The method according to claim 9 and comprising determining the tilt angle δ

based on the expression tan β

=(Δ

X_o/Z_o)tan δ

where Δ

X_o is a distance between two parallel second lines of the at least two second lines and Z_o is a distance between the projection point of the at least one camera and an intersection of the plane of the two parallel second lines and the optical axis of the at least one camera.

11. The method according to claim 8 and comprising determining the tilt angle δ

based on an expression δ

=arc tan(f/y) where f represents the focal length of the camera and y represents a distance from the principal point of the at least one camera of an intersection between images of lines defined by images of the fiducials in the projected image that define the second lines.

12. The method according to claim 11 and comprising comparing the convergence angles α

and β

to values in a lookup table that tabulates pan and tilt angles of a camera as functions of convergence angles α and

β

to determining pan and tilt angels for the at least one camera.

13. The method according to claim 11 wherein using the at least one convergence angle comprises generating the lookup table.

14. The method according to claim 1 wherein the optical axis of each of the first and second cameras is defined by an optical system that projects the image of the calibration target acquired by the camera and determining the at least one angle comprises processing the projected image to remove distortion in the projected image introduced by the optical system of the at least one of the first and second camera.

15. The method according to claim 13 wherein generating the lookup table comprises:

imaging a calibration target comprising at least two parallel rows of fiducials and at least two parallel columns of fiducials that are perpendicular to the rows with a camera oriented at a plurality of different known pan angles and a plurality of different known tilt angles;

determining for each pair of a known pan angle of the plurality of known pan angles and a known tilt angle of the plurality of tilt angles a convergence angle for images of the at least two rows of fiducials and images of the at least two columns of fiducials in the image of the calibration target acquired by the camera at the known pan and tilt angles in the pair; and

tabulating the known pan and tilt angles as functions of the convergence angles.

16. The method according to claim 11 and comprising using the pan and tilt angles to determine a relative rotation between the camera coordinate systems of the first and second cameras and using the determined relative rotation to register the first and second cameras to each other.

17. The method according to claim 1 wherein the MCI comprises a plurality of cameras greater than two.

18. The method according to claim 17 and comprising calibrating each of the plurality of cameras to another of the plurality of cameras so that all of the plurality of cameras are registered one to the other.

19. A multi-camera imager (MCI) comprising:

first and second cameras each having a center of projection, an optical axis extending from the center of projection and an image plane intersected by the optical axis at a principal point; and

a controller configured to;

determine a first angle between lines defined by images of features of a scene in a first image that the first camera projects onto the image plane of the first camera and a second angle between lines defined by images of the same features of the scene in a second image that the second camera projects onto the image plane of the second camera;

use the first and second angles to determine a relative rotation between the camera coordinate systems of the first and second cameras; and

use the relative rotation to register images acquired by one of the first and second cameras to the other of the first and second cameras.

20. The MCI according to claim 19 wherein the features in the scene that define the first and second lines in the in the first and second images define parallel lines in the scene and determining the first and second angles comprises determining a first convergence angle between the first line in the first image and a second convergence angle between the second lines in the second image.