Imaging system comprising real-time image registration

• US 10,326,979 B2
• Filed: 05/23/2016
• Issued: 06/18/2019
• Est. Priority Date: 05/23/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Hall, Michael Anthony
• Primary Examiner(s): Jones, Heather R
• Application Number:
  US15/161,309
• Publication Number:
  US 20170339395A1
• Time in Patent Office: 1,121 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/344 involving models
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/257 Colour aspects

Abstract

A multi-camera imager (MCI) configured to register a first image of a scene acquired by a first camera comprised in the MCI to a second image of the same scene acquired by a second camera in the MCI based on adjusting a transform configured to register images of a calibration target acquired by the first and second cameras for a same fixed position and orientation of the calibration target using epipolar line segments in one of the cameras associated with image plane coordinates in the other of the cameras.

Claims

1. A multi-camera imager (MCI) comprising:

first and second cameras each having a center of projection, an optical axis extending from the center of projection, an image plane intersected by the optical axis at a principal point and a photosensor comprising pixels on which the camera images scenes;

a memory having stored therein;

instructions executable to implement a first transform configured to map image plane coordinates of features of a scene located at a same predetermined distance from the MCI imaged by the first camera to image plane coordinates of the same features imaged by the second camera;

data defining an epipolar line segment in the image plane of the second camera that corresponds to a projection line in the first camera, for each of a plurality of image plane coordinates in the image plane of the first camera; and

instructions executable to adjust the first transform based on the epipolar line segments to provide a second transform configured to map image plane coordinates of features of a scene located at a distance from the MCI different from the predetermined distance that are imaged by the first camera to image coordinates of the same features imaged by the second camera; and

a processor configured to register images of a scene acquired by the first and second cameras to each other based on the instructions and data stored in the memory.

2. The MCI according to claim 1 wherein the first transform comprises a polynomial transform.

3. The MCI according to claim 1 wherein the first transform is determined based on first and second images of fiducials in a calibration target respectively acquired by the first and second cameras from imaging the target at a same single position and orientation relative to the MCI.

4. The MCI according to claim 3 wherein each epipolar line segment in the image plane of the second camera corresponds to a segment of a different projection line that extends from a fiducial in the calibration target to the center of projection of the first camera along which the first camera projects an image of a feature in the calibration target to the image plane of the first camera.

5. The MCI according to claim 3 wherein the target is a planar target and the optical axis of at least one of the cameras is perpendicular to the target.

6. The MCI according to claim 1 wherein the segment of the projection line has ends that are located substantially at lower and upper bound limits of an operating distance range of the MCI.

7. The MCI according to claim 1 wherein the processor is configured to register a first image of a scene acquired by the first camera to a second image of the same scene acquired by the second camera based on the processor identifying a feature in the first image and searching for the same feature in the second image in a region of the second image based on a neighborhood of an epipolar line segment associated with a projection line along which the first camera projects an image of the feature.

8. The MCI according to claim 7 wherein the neighborhood of the epipolar line comprises a region of the image plane of the second camera that extends from the epipolar line segment to a distance less than or equal to about five characteristic dimensions of a photosensor pixel of the second camera.

9. The MCI according to claim 7 wherein the instructions to adjust the first transform to provide the second transform comprises instructions to adjust the first transform to transform image plane coordinates of the feature in the first image to image plane coordinates of the feature in the second image.

10. The MCI according to claim 1, wherein the optical axes of the first and second cameras are parallel.

11. A method of registering a first camera to a second camera of a multi-camera imager (MCI), each camera having a center of projection, an optical axis extending from the center of projection, an image plane intersected by the optical axis at a principal point and a photosensor comprising pixels on which the camera images scenes, the method comprising:

providing a calibration target at a known position and known orientation relative to the MCI, the calibration target having fiducials;

operating the first and second cameras to acquire first and second images of the calibration target at the known position and orientation;

using the first and second images to determine a first transform configured to map image plane coordinates in the image plane of the first camera of images of fiducials in the calibration target imaged in the first image to image plane coordinates in the image plane of the second camera of images of the same fiducials imaged by the second camera in the second image; and

associating an epipolar line segment in the image plane of the second camera that corresponds to a projection line in the first camera for each of a plurality of image plane coordinates in the image plane of the first camera.

12. The method according to claim 11 and comprising configuring the MCI to adjust the first transform to generate a second transform based on the epipolar line segments.

13. The method according to claim 12 wherein configuring the MCI to adjust the first transform to generate a second transform function based on the epipolar line segments comprises configuring the MCI to identify a feature in a first image of a scene acquired by the first camera and search for the same feature in a second image of the same scene acquired by the second camera in a region of the second image based on a neighborhood of an epipolar line segment associated with a projection line along which the first camera projects an image of the feature.

14. The method according to claim 13 wherein the neighborhood of the epipolar line comprises a region of the image plane of the second camera that extends from the epipolar line segment to a distance less than or equal to about five characteristic dimensions of a photosensor pixel of the second camera.

15. The method according to claim 13 wherein configuring the MCI to adjust the first transform comprises configuring the MCI to adjust the first transform to transform image plane coordinates of the feature in the first image to image plane coordinates of the feature in the second image.

16. The method according to claim 11 wherein the first transform comprises a polynomial transform.

17. The method according to claim 11 wherein each epipolar line segment corresponds to a segment of a different projection line that extends from a fiducial in the calibration target to the center of projection of the first camera along which the first camera projects an image of a feature in the calibration target to the image plane of the first camera.

18. The method according to claim 17 wherein the segment of the projection line has ends that are located substantially at lower and upper bound limits of an operating distance range of the MCI.

19. The method according to claim 11 wherein the target is a planar target and the optical axis of at least one of the cameras is perpendicular to the target.

20. The method according to claim 11 wherein the optical axes of the first and second cameras are parallel.