Lidar and camera synchronization

• US 10,469,753 B2
• Filed: 10/17/2018
• Issued: 11/05/2019
• Est. Priority Date: 10/19/2017
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: DeepMap, Inc. (NVIDIA Corporation)
• Inventors: Yang, Lin, Wheeler, Mark Damon
• Primary Examiner(s): Rahaman, Mohammed S
• Application Number:
  US16/163,463
• Publication Number:
  US 20190120948A1
• Time in Patent Office: 384 Days
• Field of Search: 348118
• CPC Class Codes: G01C 21/1652 with ranging devices, e.g. LIDAR or RADAR
• CPC Class Codes: G01C 21/3602 Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G01C 25/00 Manufacturing, calibrating, cleaning, or repairing instruments or devices referred to in the other groups of this subclass testing, calibrating or compensating compasses G01C17/38
• CPC Class Codes: G01S 17/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S17/46
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 7/4817 relating to scanning
• CPC Class Codes: G01S 7/497 Means for monitoring or calibrating
• CPC Class Codes: G01S 7/4972 Alignment of sensor
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0231 using optical position detecting means position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16
• CPC Class Codes: G05D 1/0248 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0287 involving a plurality of land vehicles, e.g. fleet or convoy travelling traffic control systems for road vehicles G08G1/00, particularly anticollision systems G08G1/16
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20092 Interactive image processing based on input by user
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30242 : Counting objects in image
• CPC Class Codes: G06T 2207/30252 : Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/13 : Edge detection
• CPC Class Codes: G06T 7/33 : using feature-based methods
• CPC Class Codes: G06T 7/55 : from multiple images
• CPC Class Codes: G06T 7/80 : Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 13/106 : Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 23/54 : Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/60 : Control of cameras or camera modules
• CPC Class Codes: H04N 23/689 : Motion occurring during a rolling shutter mode
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/04 : Synchronising for television systems using pulse code modulation H04N7/56

Abstract

A method and system for synchronizing a lidar and a camera on an autonomous vehicle. The system selects a plurality of track samples for a route including a lidar scan and an image. For each track sample, the system calculates a time shift by iterating many time deltas. For each time delta, the system adjusts a camera timestamp by that time delta, projects a lidar scan onto the image as a lidar projection according to the adjusted camera timestamp, and calculates an alignment score of the lidar projection for that time delta. The system defines the time shift for each track sample as the time delta with the highest alignment score. The system then models time drift of the camera compared to the lidar based on the calculated time shifts for the track samples and synchronizes the lidar and the camera according to the modeled time drift.

Claims

1. A method for synchronizing a light detection and ranging sensor (lidar) and a camera on an autonomous vehicle, comprising:

selecting a plurality of track samples for a route, each track sample comprising an image captured at a camera timestamp by the camera and a lidar scan captured by the lidar;

for each track sample, calculating a time shift comprising;

for each time delta of a plurality of time deltas;

adjusting the camera timestamp for the image by the time delta,

projecting the lidar scan into image coordinates of the image as a lidar projection according to the adjusted camera timestamp, and

calculating an alignment score of the lidar projection indicative of alignment of the lidar projection to the image;

defining time shift of the track sample as the time delta with an optimal alignment score;

modeling time drift of the camera compared to the lidar over the route based on the calculated time shifts for the track samples; and

synchronizing the lidar and the camera according to the modeled time drift.

2. The method of claim 1, wherein the plurality of track samples comprise at least a first track sample within a threshold time of a start of the route and a second track sample within the threshold time of an end of the route.

3. The method of claim 1, wherein each of the track samples are selected according to any combination of following conditions:

speed of the autonomous vehicle during recording of the track sample is greater than or equal to thirty miles per hour;

movement of the autonomous vehicle during recording of the track sample is less than or equal to a threshold curvature; and

one or more reflective features are detected in the track sample.

4. The method of claim 1, wherein for each track sample, for each time delta, projecting the lidar scan onto the image as a lidar projection according to the adjusted camera timestamp, further comprising:

computing an unwinding transform of the track sample based on two consecutive poses that encompass a duration of the track sample;

unwinding the lidar scan according to the unwinding transform at the adjusted camera timestamp; and

projecting the unwound lidar data onto the image as the lidar projection.

5. The method of claim 1, wherein for each track sample, for each time delta, adjusting the camera timestamp for the image by the time delta, further comprising one of:

adding the time delta to the camera timestamp as the adjusted camera timestamp; and

subtracting the time delta from the camera timestamp as the adjusted camera timestamp.

6. The method of claim 1, wherein the plurality of time deltas comprises any combination of 0 milliseconds (ms), 5 ms, 10 ms, 15 ms, 20 ms, 25 ms, and 30 ms.

7. The method of claim 1, wherein for each track sample, for each time delta, calculating the alignment score of the lidar projection, further comprising:

identifying an informative feature in the lidar projection comprising a plurality of lidar point clouds in the lidar projection;

identifying the informative feature in the image comprising a plurality of pixels in the image; and

calculating an alignment edge score at each image coordinate by comparing the lidar point cloud and the pixel at each image coordinate,

wherein the alignment score is dependent on the alignment edge score.

8. The method of claim 7, wherein identifying the informative feature in the lidar projection comprising a plurality of point clouds in the lidar projection, further comprising:

fitting a ground plane in the lidar projection using Random Sample Consensus (RANSAC);

removing lidar point clouds in the lidar data not within the fitted ground plane;

identifying one or more lidar point clouds above a threshold reflective intensity; and

determining a shape of the informative feature in the lidar projection according to the identified one or more lidar point clouds;

wherein the informative feature comprising the plurality of lidar point clouds is based on the determined shape and the identified one or more lidar point clouds.

9. The method of claim 7, wherein identifying the informative feature in the image comprising a plurality of pixels in the image, further comprising:

transforming each pixel in the image into an intensity value;

calculating a gradient strength for each pixel based on intensity values of neighboring pixels;

identifying one or more pixels above a threshold gradient strength; and

determining a shape of the informative feature in the image according to the identified one or more pixels;

wherein the informative feature comprising the plurality of pixels is based on the determined shape and the identified one or more pixels.

10. The method of claim 1, wherein for each track sample, for each time delta, calculating the alignment score of the lidar projection, further comprising:

displaying the lidar projection superimposed on the image; and

receiving an input in response to the superimposed lidar projection on the image,

wherein the alignment score is dependent on the input.

11. The method of claim 1, wherein modeling time drift of the camera compared to the lidar over the route is based on a linear regression of the track samples and the calculated time shifts.

12. The method of claim 1, further comprising:

estimating a time shift for another track sample along the route based on the modeled time drift.

13. The method of claim 1, wherein synchronizing the lidar and the camera according to the modeled time drift, further comprising:

adjusting the camera timestamp according to the modeled time drift for subsequent track samples.

14. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform steps for synchronizing a lidar and a camera on an autonomous vehicle, the steps comprising:

selecting a plurality of track samples for a route, each track sample comprising an image captured at a camera timestamp by the camera and a lidar scan captured by the lidar;

for each track sample, calculating a time shift comprising;

for each time delta of a plurality of time deltas;

adjusting the camera timestamp for the image by the time delta,

projecting the lidar scan into image coordinates of the image as a lidar projection according to the adjusted camera timestamp, and

calculating an alignment score of the lidar projection indicative of alignment of the lidar projection to the image;

defining time shift of the track sample as the time delta with an optimal alignment score;

modeling time drift of the camera compared to the lidar over the route based on the calculated time shifts for the track samples; and

synchronizing the lidar and the camera according to the modeled time drift.

15. The non-transitory computer-readable storage medium of claim 14, wherein the plurality of track samples comprise at least a first track sample within a threshold time of a start of the route and a second track sample within the threshold time of an end of the route.

16. The non-transitory computer-readable storage medium of claim 14, wherein each of the track samples are selected according to any combination of following conditions:

speed of the autonomous vehicle during recording of the track sample is greater than or equal to thirty miles per hour;

movement of the autonomous vehicle during recording of the track sample is less than or equal to a threshold curvature; and

one or more reflective features are detected in the track sample.

17. The non-transitory computer-readable storage medium of claim 14, wherein for each track sample, for each time delta, projecting the lidar scan onto the image as a lidar projection according to the adjusted camera timestamp, further comprising:

computing an unwinding transform of the track sample based on two consecutive poses that encompass a duration of the track sample;

unwinding the lidar scan according to the unwinding transform at the adjusted camera timestamp; and

projecting the unwound lidar data onto the image as the lidar projection.

18. The non-transitory computer-readable storage medium of claim 14, wherein for each track sample, for each time delta, adjusting the camera timestamp for the image by the time delta, further comprising one of:

adding the time delta to the camera timestamp as the adjusted camera timestamp; and

subtracting the time delta from the camera timestamp as the adjusted camera timestamp.

19. The non-transitory computer-readable storage medium of claim 14, wherein for each track sample, for each time delta, calculating the alignment score of the lidar projection, further comprising:

identifying an informative feature in the lidar projection comprising a plurality of lidar point clouds in the lidar projection;

identifying the informative feature in the image comprising a plurality of pixels in the image; and

calculating an alignment edge score at each image coordinate by comparing the lidar point cloud and the pixel at each image coordinate,

wherein the alignment score is dependent on the alignment edge score.

20. The non-transitory computer-readable storage medium of claim 14, wherein modeling time drift of the camera compared to the lidar over the route is based on a linear regression of the track samples and the calculated time shifts.

21. The non-transitory computer-readable storage medium of claim 14, the steps further comprising:

estimating a time shift for another track sample along the route based on the modeled time drift.

22. The non-transitory computer-readable storage medium of claim 14, wherein synchronizing the lidar and the camera according to the modeled time drift, further comprising:

adjusting the camera timestamp according to the modeled time drift for subsequent track samples.

23. A system comprising:

a lidar on an autonomous vehicle;

a camera on the autonomous vehicle;

a processor; and

a non-transitory computer-readable storage medium storing instructions that, when executed by the processor, cause the processor to perform steps for synchronizing the lidar and the camera, the steps comprising;

selecting a plurality of track samples for a route, each track sample comprising an image captured at a camera timestamp by the camera and a lidar scan captured by the lidar;

for each track sample, calculating a time shift comprising;

for each time delta of a plurality of time deltas;

adjusting the camera timestamp for the image by the time delta,

projecting the lidar scan into image coordinates of the image as a lidar projection according to the adjusted camera timestamp, and

calculating an alignment score of the lidar projection indicative of alignment of the lidar projection to the image;

defining time shift of the track sample as the time delta with an optimal alignment score;

modeling time drift of the camera compared to the lidar over the route based on the calculated time shifts for the track samples; and

synchronizing the lidar and the camera according to the modeled time drift.

24. The system of claim 23, wherein the plurality of track samples comprise at least a first track sample within a threshold time of a start of the route and a second track sample within the threshold time of an end of the route.

25. The system of claim 23, wherein each of the track samples are selected according to any combination of following conditions:

speed of the autonomous vehicle during recording of the track sample is greater than or equal to thirty miles per hour;

movement of the autonomous vehicle during recording of the track sample is less than or equal to a threshold curvature; and

one or more reflective features are detected in the track sample.

26. The system of claim 23, wherein for each track sample, for each time delta, projecting the lidar scan onto the image as a lidar projection according to the adjusted camera timestamp, further comprising:

computing an unwinding transform of the track sample based on two consecutive poses that encompass a duration of the track sample;

unwinding the lidar scan according to the unwinding transform at the adjusted camera timestamp; and

projecting the unwound lidar data onto the image as the lidar projection.

27. The system of claim 23, wherein for each track sample, for each time delta, adjusting the camera timestamp for the image by the time delta, further comprising one of:

adding the time delta to the camera timestamp as the adjusted camera timestamp; and

subtracting the time delta from the camera timestamp as the adjusted camera timestamp.

28. The system of claim 23, wherein for each track sample, for each time delta, calculating the alignment score of the lidar projection, further comprising:

identifying an informative feature in the lidar projection comprising a plurality of lidar point clouds in the lidar projection;

identifying the informative feature in the image comprising a plurality of pixels in the image; and

calculating an alignment edge score at each image coordinate by comparing the lidar point cloud and the pixel at each image coordinate,

wherein the alignment score is dependent on the alignment edge score.

29. The system of claim 23, wherein modeling time drift of the camera compared to the lidar over the route is based on a linear regression of the track samples and the calculated time shifts.

30. The system of claim 23, the steps further comprising:

estimating a time shift for another track sample along the route based on the modeled time drift.

31. The system of claim 23, wherein synchronizing the lidar and the camera according to the modeled time drift, further comprising:

adjusting the camera timestamp according to the modeled time drift for subsequent track samples.