Infrastructure free intrinsic calibration

• US 10,298,910 B1
• Filed: 06/29/2018
• Issued: 05/21/2019
• Est. Priority Date: 06/29/2018
• Status: Active Grant

• Current Assignee: Zoox Inc. (Amazon.com, Inc.)
• Original Assignee: Zoox Inc. (Amazon.com, Inc.)
• Inventors: Kroeger, Till
• Primary Examiner(s): Natnael, Paulos M
• Application Number:
  US16/023,507
• Time in Patent Office: 326 Days
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras

Abstract

This disclosure is directed to calibrating sensors mounted on an autonomous vehicle. First image data and second image data representing an environment can be captured by first and second cameras, respectively (and or a single camera at different points in time). Point pairs comprising a first point in the first image data and a second point in the second image data can be determined and projection errors associated with the points can be determined. A subset of point pairs can be determined, e.g., by excluding point pairs with the highest projection error. Calibration data associated with the subset of points can be determined and used to calibrate the cameras without the need for calibration infrastructure.

Claims

1. A system comprising:

a vehicle configured to traverse through an environment, the vehicle comprising;

a plurality of cameras disposed on a body of the vehicle such that fields of view of at least two of the cameras at least partially overlap;

one or more processors; and

computer-readable media storing instructions executable by the one or more processors, wherein the instructions cause the system to perform operations comprising;

receiving a set of images from the plurality of cameras, the set of images comprising images representing the environment at substantially a first time and captured by the plurality of cameras,

determining, for the set of images, a plurality of point pairs, each of the point pairs comprising a first point in one of the images of the set of images and a second point in another of the images of the set of images, the first point and the second point corresponding to a same image feature;

determining first errors associated with the plurality of point pairs, the first errors comprising, at least in part, a first distance between the first point and an epipolar line corresponding to the second point;

determining, based at least in part on the first errors, a first subset of the plurality of point pairs;

determining, from the first subset of the plurality of point pairs, a first correction function representative of a misalignment of an estimated relative pose of the cameras;

determining second errors associated with the plurality of point pairs, the second errors comprising, at least in part, a second distance between the first point and a re-projection of the second point estimating a point depth;

determining, based at least in part on the second errors, a second subset of the plurality of point pairs;

determining, from the second subset of the plurality of point pairs, a second correction function representative of a misalignment of an estimated relative pose of the cameras;

calibrating the plurality of cameras, based at least in part on the first correction function and the second correction function.

2. The system of claim 1, wherein the determining the first errors comprises:

determining a fundamental matrix which describes a relationship between the one of the images and the other of the images;

determining, for individual point pairs of the plurality of point pairs and based at least in part on the fundamental matrix, the epipolar line corresponding to the second point in the first image, and

comparing the first point to the epipolar line in the first image.

3. The system of claim 1, wherein determining the second error comprises undistorting the first point according to a camera intrinsic.

4. The system of claim 1, wherein at least one of the determining the first subset of the plurality of point pairs or the determining the second subset of the plurality of point pairs comprises:

dividing an image of the set of images containing the first points and at least one of the epipolar lines or the re-projections into a plurality of regions; and

determining a subset of point pairs for individual of the plurality of regions,

wherein the first subset of the plurality of point pairs or the second subset of the plurality of point pairs comprises the subset of point pairs for the plurality of regions.

5. The system of claim 1, wherein the vehicle is an autonomous vehicle, the one or more computer-readable media storing further instructions executable by the one or more processors, wherein the further instructions cause the system to perform operations comprising:

calibrating, based at least in part on the calibration function, additional image data received from the plurality of cameras as calibrated data;

generated, based at least in part on the calibrated data, a trajectory to control the autonomous vehicle; and

controlling the autonomous vehicle based at least in part on the trajectory.

6. A method comprising:

receiving first image data comprising a first image of an environment captured by a first camera disposed on a system, the first camera having a first field of view;

receiving second image data comprising a second image of the environment captured by a second camera disposed on the system, the second camera having a second field of view at least partially overlapping the first field of view;

determining a plurality of point pairs, a point pair of the plurality of point pairs comprising a first point in the first image and a second point in the second image corresponding to the first point;

determining first errors for individual point pairs of the plurality of point pairs, the first errors comprising, at least in part, a distance between the first point and an epipolar line corresponding to the second point in the first image;

determining second errors for individual point pairs of the plurality of point pairs, the second errors comprising, at least in part, a distance between the first point and a reprojection of the corresponding second point, the reprojection based at least in part on an estimated depth of the first point;

determining, based at least in part on at least one of the first errors or the second errors, a subset of the plurality of point pairs; and

determining, from the subset of the plurality of point pairs, a calibration function representative of at least one of an estimated relative pose of the first camera and the second camera or an estimated camera intrinsic of the first camera or the second camera.

7. The method of claim 6, wherein the determining the first errors comprises:

determining a fundamental matrix;

determining, for the point pair of the plurality of point pairs and based at least in part on the fundamental matrix, the epipolar line corresponding to the second point,

projecting the epipolar line in the first image, and

determining a distance between the first point and the epipolar line in the first image.

8. The method of claim 6, wherein the determining the plurality of point pairs comprises:

extracting a plurality of first features from the first image;

extracting a plurality of second features from the second image;

matching, as matched features, first features from the first plurality of features and second features of the second plurality of features; and

determining, as the plurality of point pairs, a plurality of pixels associated with the matched features,

wherein the plurality of first features and the plurality of second features comprise one or more of FREAK, AKAZE, FAST, SURF, SIFT, ORB, or BRIEF features.

9. The method of claim 6, wherein the reprojection is further based on undistorting the first point based, at least in part, on the estimated camera intrinsic of the first camera.

10. The method of claim 6, wherein the determining the subset of the plurality of point pairs comprises:

dividing the first image containing the first points into a plurality of regions; and

determining, based at least in part on the first errors or the second errors, a subset of point pairs for a region of the plurality of regions,

wherein the first subset of the plurality of point pairs or the second subset of the plurality of point pairs comprises the subset of point pairs for the plurality of regions.

11. The method of claim 10, wherein the determining the calibration function comprises applying, based on an association between the first points and the plurality of regions, a weighting factor to individual of the plurality of point pairs.

12. The method of claim 11, wherein the first points in a first region of the plurality of regions are assigned a first weight based on the number of the first points in the first region.

13. The method of claim 6, wherein:

at least one of the first image data comprises a third image of the environment captured by the first camera or the second image data comprises a fourth image of the environment captured by the second camera; and

the point pair further comprises at least one of a third point in the third image or a fourth point in the fourth image.

14. The method of claim 6, further comprising:

detecting edges in the first image or the second image;

receiving lidar data comprising a point cloud of the environment captured by a lidar sensor disposed on the system, the point cloud comprising data about a portion of the second field of view overlapping the first field of view;

determining a plurality of depth discontinuities in the lidar data;

comparing the plurality of depth discontinuities to the edges by projecting the plurality of depth discontinuities into the first image or the second image as a plurality of projected points; and

determining distances between the edges and the plurality of projected points; and

determining, based at least in part on the distances, a correction function representative of a misalignment of at least one of the first camera or the second camera and the lidar sensor.

15. A non-transitory computer-readable medium storing instructions that, when executed, cause one or more processors to perform operations comprising:

receiving first image data comprising a first image of an environment captured by a first camera disposed on a system, the first camera having a first field of view;

receiving second image data comprising a second image of the environment captured by a second camera disposed on the system, the second camera having a second field of view at least partially overlapping the first field of view;

determining a plurality of point pairs, individual of the point pairs comprising a first point in the first image and a second point in the second image corresponding to the first point;

determining first errors for individual point pairs of the plurality of point pairs, the first errors comprising, at least in part, a distance between the first point and an epipolar line corresponding to the second point in the first image;

determining second errors for individual point pairs of the plurality of point pairs, the second errors comprising, at least in part, a distance between the first point and a reprojection of the corresponding second point estimating a depth of the first point;

determining, based at least in part on at least one of the first errors or the second errors, a subset of the plurality of point pairs;

determining, from the subset of the plurality of point pairs, a calibration function representative of at least one of an estimated relative pose of the first camera and the second camera or an estimated camera intrinsic.

16. The non-transitory computer-readable medium of claim 15, wherein the determining the plurality of point pairs comprises:

extracting a plurality of first features from the first image;

extracting a plurality of second features from the second image;

matching, as matched features, first features from the first plurality of features and second features of the second plurality of features; and

determining, as the plurality of point pairs, a plurality of pixels associated with the matched features,

wherein the plurality of first features and the plurality of second features comprise one or more of FREAK, AKAZE, FAST, SURF, SIFT, ORB, or BRIEF features.

17. The non-transitory computer-readable medium of claim 15, wherein the determining the subset of the plurality of point pairs comprises:

determining point pairs having projection errors equal to or below a threshold error.

18. The non-transitory computer-readable medium of claim 15, wherein the determining the subset of the plurality of point pairs comprises:

dividing the first image containing the first points into a plurality of regions; and

determining, based at least in part on the first errors or the second errors, a subset of point pairs of the plurality of regions,

wherein the subset of the plurality of point pairs comprises the subset of point pairs for the plurality of regions.

19. The method of claim 18, wherein the determining the calibration function comprises applying, based on an association between the first points and the plurality of regions, a weighting factor to individual of the plurality of point pairs.

20. The method of claim 19, wherein the first points in a first region of the plurality of regions are assigned a first weight based on the number of the first points in the first region.