Registering cameras with virtual fiducials

• US 10,339,662 B2
• Filed: 05/23/2016
• Issued: 07/02/2019
• Est. Priority Date: 05/23/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wilson, Andrew D., Hall, Michael Anthony
• Primary Examiner(s): Jones, Heather R
• Application Number:
  US15/161,314
• Publication Number:
  US 20170337700A1
• Time in Patent Office: 1,135 Days
• Field of Search: None
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/262 using transform domain methods, e.g. Fourier domain methods
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/344 involving models
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A method of registering first and second cameras in a multi-camera imager comprising generating virtual fiducials at different locations relative to the multi camera imager and using coordinates of the virtual fiducials to determine a fundamental matrix for the cameras.

Claims

1. A multi-camera imager (MCI) comprising:

first and second cameras each having a center of projection, an optical axis extending from the center of projection and an image plane intersected by the optical axis at a principal point, wherein the first camera comprises a time of flight (TOF) range camera and the second camera comprises an RGB picture camera;

memory having stored therein;

a fundamental matrix based on a polynomial transform configured to map image plane coordinates of images of features of a calibration target imaged by the TOF range camera on the image plane of the TOF range camera to image plane coordinates of images of the same features of the calibration target as imaged by the RGB picture camera on the image plane of the RGB picture camera; and

instructions executable to register images of a scene acquired by the first and second cameras to each other based on the fundamental matrix; and

a processor configured to execute the instructions to register the images of the scene based on the fundamental matrix.

2. The MCI according to claim 1 wherein the memory comprises the polynomial transform.

3. The MCI according to claim 2 wherein the instructions comprise instructions executable to transform image plane coordinates of features of the scene imaged by the first camera to image plane coordinates of the same features of the scene as imaged by the second camera.

4. The MCI according to claim 1 wherein the polynomial transform is determined from first and second images respectively acquired by the first and second cameras of the calibration target located at a same, only one, known position, and at a same, only one, known orientation relative to the MCI.

5. The MCI according to claim 4 wherein the features of the calibration target comprise an array of fiducials.

6. The MCI according to claim 1 wherein the fundamental matrix is determined based on real spatial coordinates selected for a plurality of “

N”

virtual fiducials in a field of view (FOV) of the MCI.

7. The MCI according to claim 6 wherein components of the fundamental matrix are functions of image plane coordinates of a virtual image of each of the plurality of virtual fiducials in the image plane of the first camera and matching image plane coordinates of a virtual image of the virtual fiducial in the image plane of the second camera.

8. The MCI according to claim 7 wherein the coordinates of the virtual image of a virtual fiducial in the image plane of the first or second camera are image plane coordinates of an intersection of a line that connects the real spatial coordinates of the virtual fiducial and real spatial coordinates of the center of projection of the camera.

9. The MCI according to claim 8 wherein if a vector x(n) represents the image coordinates for the virtual image of an n-th virtual fiducial in the image plane of the first camera and a vector x′

(n) represents the image coordinates of the virtual image of the n-th virtual fiducial in the image plane of the second camera the components of the fundamental matrix are determined based on N equations x′

(n)^TFx(n)=0, 1≤

n≤

N, where x′

(n)^T is a transpose of the vector x′

(n).

10. The MCI according to claim 1 wherein the optical axes of the first and second camera are parallel.

11. A method of registering first and second cameras comprised in a multi-camera imager (MCI), each camera having a center of projection, an optical axis extending from the center of projection and an image plane intersected by the optical axis, the method comprising:

operating the MCI to image a calibration target and acquire first and second images of the calibration target by the first and second cameras respectively, wherein the first camera comprises a time of flight (TOF) range camera and the second camera comprises an RGB picture camera;

determining a polynomial transform configured to map image plane coordinates of images of features of the calibration target in the first image on the image plane of the TOF range camera to image plane coordinates of images of the same features in the second image on the image plane of the RGB picture camera; and

using the polynomial transform to determine a fundamental matrix for the first and second cameras.

12. The method according to claim 11 wherein operating the MCI comprises operating the MCI to image the calibration target located at a same, only one, known position, and at a same, only one, known orientation relative to the MCI.

13. The method according to claim 11 and comprising selecting real space coordinates for each of a plurality of N virtual fiducials in the field of view of the first and second cameras that are not real spatial coordinates of a feature of the calibration target as imaged by the MCI.

14. The method according to claim 13 wherein selecting real space coordinates comprises selecting coordinates of a three dimensional point lattice.

15. The method according to claim 13 wherein selecting comprises selecting real space coordinates for which distances from the center of projection of the first or second camera to the real space coordinates have different projections on the optical axis of the camera.

16. The method according to claim 13 and comprising for each of the N virtual fiducials determining image plane coordinates of a virtual image of the virtual fiducial in the image plane of the first camera and matching image plane coordinates of a virtual image of the virtual fiducial in the image plane of the second camera.

17. The method according to claim 16 wherein determining image plane coordinates for the virtual image in the image plane of the first or second camera comprises determining image plane coordinates of an intersection of a line that connects the real spatial coordinates of the virtual fiducial and real spatial coordinates of the center of projection of the camera.

18. The method according to claim 16 wherein if a vector x(n) represents the image coordinates for the virtual image of an n-th virtual fiducial in the image plane of the first camera and a vector x′

(n) represents the image coordinates of the virtual image of the n-th virtual fiducial in the image plane of the second camera determining the fundamental matrix comprises determining components of the fundamental matrix that satisfy N equations x′

(n)^TFx(n)=0, 1≤

n≤

N, where x′

(n)^T is a transpose of the vector x′

(n).

19. The method according to claim 11 wherein the optical axes of the first and second cameras are parallel.

20. Computer memory embodied with instructions executable by one or more processors for registering first and second cameras comprised in a multi-camera imager (MCI), the first and second cameras having a center of projection, an optical axis extending from the center of projection and an image plane intersected by the optical axis, comprising:

operating the MCI to image a calibration target and acquire first and second images of the calibration target by the first and second cameras respectively, wherein the first camera comprises a time of flight (TOF) range camera and the second camera comprises an RGB picture camera;

determining a polynomial transform configured to map image plane coordinates of images of features of the calibration target in the first image on the image plane of the TOF range camera to image plane coordinates of images of the same features in the second image on the image plane of the RGB picture camera; and

using the polynomial transform to determine a fundamental matrix for the first and second cameras.