Calibrating sensors mounted on an autonomous vehicle

• US 10,436,885 B2
• Filed: 10/15/2018
• Issued: 10/08/2019
• Est. Priority Date: 10/19/2017
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: DeepMap, Inc. (NVIDIA Corporation)
• Inventors: Wheeler, Mark Damon, Yang, Lin
• Primary Examiner(s): Zanelli, Michael J
• Application Number:
  US16/161,035
• Publication Number:
  US 20190120946A1
• Time in Patent Office: 358 Days
• CPC Class Codes: G01C 21/1652 with ranging devices, e.g. LIDAR or RADAR
• CPC Class Codes: G01C 21/3602 Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G01C 25/00 Manufacturing, calibrating, cleaning, or repairing instruments or devices referred to in the other groups of this subclass testing, calibrating or compensating compasses G01C17/38
• CPC Class Codes: G01S 17/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S17/46
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 7/4817 relating to scanning
• CPC Class Codes: G01S 7/497 Means for monitoring or calibrating
• CPC Class Codes: G01S 7/4972 Alignment of sensor
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0231 using optical position detecting means position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16
• CPC Class Codes: G05D 1/0248 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0287 involving a plurality of land vehicles, e.g. fleet or convoy travelling traffic control systems for road vehicles G08G1/00, particularly anticollision systems G08G1/16
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20092 Interactive image processing based on input by user
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30242 : Counting objects in image
• CPC Class Codes: G06T 2207/30252 : Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/13 : Edge detection
• CPC Class Codes: G06T 7/33 : using feature-based methods
• CPC Class Codes: G06T 7/55 : from multiple images
• CPC Class Codes: G06T 7/80 : Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 13/106 : Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 23/54 : Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/60 : Control of cameras or camera modules
• CPC Class Codes: H04N 23/689 : Motion occurring during a rolling shutter mode
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/04 : Synchronising for television systems using pulse code modulation H04N7/56

Abstract

A system calibrates one or more sensors mounted to an autonomous vehicle. From the one or more sensors, the system identifies a primary sensor and a secondary sensor. The system determines a reference angle for the primary sensor, and based on that reference angle for the primary sensor, a scan-start time representing a start of a scan and a scan-end time representing an end of a scan. The system receives, from the primary sensor, a primary set of scan data recorded from the scan-start time to the scan-end time. The system receives, from the secondary sensor, a secondary set of sensor data recorded from the scan-start time to the scan-end time. The system calibrates the primary and secondary sensors by determining a relative transform for transforming points between the first set of scan data and the second set of scan data.

Claims

1. A non-transitory computer readable storage medium having instructions for calibrating light detection and ranging (LIDAR) sensors mounted on an autonomous vehicle encoded thereon that, when executed by a processor, cause the processor to:

identify from a plurality of LIDAR sensors mounted on the autonomous vehicle, a primary LIDAR sensor and a secondary LIDAR sensor;

determine a reference angle for the primary LIDAR sensor;

determine, based on the reference angle of the primary LIDAR sensor, a scan-start time representing a start of a scan and a scan-end time representing an end of a scan;

receive, from the primary LIDAR sensor, a primary set of scan data recorded from the scan-start time to the scan-end time;

receive, from the secondary LIDAR sensor, a secondary set of scan data, recorded from the scan-start time to the scan-end time;

calibrate the primary and secondary LIDAR sensors by determining a relative transform for transforming points between the primary set of scan data and the secondary set of scan data;

generate a high definition map based on scan data recorded by the plurality of LIDAR sensors, the scan data aggregated using the relative transform; and

send signals to the controls of the autonomous vehicle based on the high-definition map.

2. The non-transitory computer readable storage medium of claim 1, wherein identifying the primary LIDAR sensor comprises:

selecting a LIDAR sensor from the plurality of LIDAR sensors that has a rotational frequency closest to a rate of performing localization for the autonomous vehicle.

3. The non-transitory computer readable storage medium of claim 1, wherein determining the scan-start time and scan-end time comprises:

detecting that the primary LIDAR sensor rotates past the reference angle a first time; and

determining the scan-start time as the time at which primary LIDAR sensor rotates past the reference angle;

detecting that the primary LIDAR sensor rotates past the reference angle a second time, the second time immediately subsequent to the first time; and

determining the scan-end time as the time at which primary LIDAR sensor rotates past the reference angle a second time.

4. The non-transitory computer readable storage medium of claim 1, wherein the scan-end time is a scan-start time for a second primary set of scan data recorded by the primary LIDAR sensor.

5. The non-transitory computer readable storage medium of claim 1, further comprising:

accessing, from a computer memory, a history of scan data recorded by the secondary LIDAR sensor as the primary LIDAR sensor periodically rotates past the reference angle; and

at each scan-start time and scan-end time received from the primary LIDAR sensor, segmenting the history of scan data.

6. The non-transitory computer readable storage medium of claim 5, wherein segmenting the history of scan data comprises:

identifying a timestamp assigned to portions of scan data recorded by the secondary LIDAR sensor within a threshold of the scan-start time received from the primary LIDAR sensor.

7. The non-transitory computer readable storage medium of claim 1, wherein the relative transform between the scans of the primary LIDAR sensor and each secondary LIDAR sensor is determined using an iterative closest point technique.

13. The computer-implemented method of claim 5, wherein segmenting the history of scan data comprises:

identifying a timestamp assigned to portions of scan data recorded by the secondary sensor within a threshold of the scan-start time received from the primary sensor.

8. A computer-implemented method for calibrating sensors mounted on a vehicle, the method comprising:

identifying from a plurality of sensors mounted on the vehicle, a primary sensor and a secondary sensor;

determining a reference angle for the primary sensor;

determining, based on the reference angle of the primary sensor, a scan-start time representing a start of a scan and a scan-end time representing an end of a scan;

receiving, from the primary sensor, a primary set of scan data recorded from the scan-start time to the scan-end time;

receiving, from the secondary sensor, a secondary set of scan data, recorded from the scan-start time to the scan-end time;

calibrating the primary and secondary sensors by determining a relative transform for transforming points between the primary set of scan data and the secondary set of scan data;

generating a high definition map based on scan data recorded by the plurality of sensors, the scan data aggregated using the relative transform; and

storing the high definition map in a computer readable storage medium.

9. The computer-implemented method of claim 8, wherein identifying the primary sensor comprises:

selecting a sensor from the plurality of sensors that has a rotational frequency closest to a rate of performing localization for the vehicle.

10. The computer-implemented method of claim 8, wherein determining the scan-start time and scan-end time comprises:

detecting that the primary sensor rotates past the reference angle a first time; and

determining the scan-start time as the time at which primary sensor rotates past the reference angle;

detecting that the primary sensor rotates past the reference angle a second time, the second time immediately subsequent to the first time; and

determining the scan-end time as the time at which sensor rotates past the reference angle a second time.

11. The computer-implemented method of claim 8, wherein the scan-end time is a scan-start time for a second primary set of scan data recorded by the primary sensor.

12. The computer-implemented method of claim 8, further comprising:

accessing, from a computer memory, a history of scan data recorded by the secondary sensor as the primary sensor periodically rotates past the reference angle; and

at each scan-start time and scan-end time received from the primary sensor, segmenting the history of scan data.

14. The computer-implemented method of claim 8, wherein the relative transform between the scans of the primary sensor and the secondary sensor is determined using an iterative closest point technique.

15. A computer-implemented method for calibrating light detection and ranging (LIDAR) sensors mounted on an autonomous vehicle, the method comprising:

receiving a reference angle for a primary LIDAR sensor;

determining, based on the reference angle of the primary LIDAR sensor, a scan-start time representing a start of a scan and a scan-end time representing an end of a scan;

receiving, from the primary LIDAR sensor, a primary set of scan data recorded from the scan-start time to the scan-end time;

receiving, from a secondary LIDAR sensor, a secondary set of scan data, recorded from the scan-start time to the scan-end time;

calibrating the primary and secondary LIDAR sensors by determining a relative transform for transforming points between the set of scan data and the secondary set of scan data;

generating a high definition map based on scan data recorded by the plurality of LIDAR sensors, the scan data aggregated using the relative transform; and

sending signals to the controls of the autonomous vehicle based on the high-definition map.

16. The computer-implemented method of claim 15, wherein, determining the scan-start time and scan-end time comprises:

detecting that the primary LIDAR sensor rotates past the reference angle a first time; and

determining the scan-start time as the time at which primary LIDAR sensor rotates past the reference angle;

detecting that the primary LIDAR sensor rotates past the reference angle a second time, the second time immediately subsequent to the first time; and

determining the scan-end time as the time at which primary LIDAR sensor rotates past the reference angle a second time.

17. The computer-implemented method of claim 15, wherein the scan-end time is a scan-start time for a second primary set of scan data recorded by the primary LIDAR sensor.

18. The computer-implemented method of claim 15, further comprising:

accessing, from a computer memory, a history of scan data recorded by the secondary LIDAR sensor as the primary LIDAR sensor periodically rotates past the reference angle; and

at each scan-start time and scan-end time received from the primary LIDAR sensor, segmenting the history of scan data.

19. The computer-implemented method of claim 18, wherein segmenting the history of scan data comprises:

identifying a timestamp assigned to portions of scan data recorded by the secondary LIDAR sensor within a threshold of the scan-start time received from the primary LIDAR sensor.

20. The computer-implemented method of claim 15, wherein the relative transform between the scans of the primary LIDAR sensor and each secondary LIDAR sensor is determined using an iterative closest point technique.

21. A non-transitory computer readable storage medium storing instructions for calibrating sensors mounted on a vehicle encoded thereon that, when executed by a processor, cause the processor to:

identify from a plurality of sensors mounted on the vehicle, a primary sensor and a secondary sensor;

determine a reference angle for the primary sensor;

determine, based on the reference angle of the primary sensor, a scan-start time representing a start of a scan and a scan-end time representing an end of a scan;

receive, from the primary sensor, a primary set of scan data recorded from the scan-start time to the scan-end time;

receive, from the secondary sensor, a secondary set of scan data, recorded from the scan-start time to the scan-end time;

calibrate the primary and secondary sensors by determining a relative transform for transforming points between the primary set of scan data and the secondary set of scan data;

generating a high definition map based on scan data recorded by the plurality of sensors, the scan data aggregated using the relative transform; and

store the high definition map in a computer readable storage medium.

22. The non-transitory computer readable storage medium of claim 21, wherein instructions for identifying the primary sensor cause the processor to:

select a sensor from the plurality of sensors that has a rotational frequency closest to a rate of performing localization for the vehicle.

23. The non-transitory computer readable storage medium of claim 21, wherein instructions for determining the scan-start time and scan-end time cause the processor to:

detect that the primary sensor rotates past the reference angle a first time; and

determine the scan-start time as the time at which primary sensor rotates past the reference angle;

detect that the primary sensor rotates past the reference angle a second time, the second time immediately subsequent to the first time; and

determine the scan-end time as the time at which sensor rotates past the reference angle a second time.

24. The non-transitory computer readable storage medium of claim 21, wherein the scan-end time is a scan-start time for a second primary set of scan data recorded by the primary sensor.

25. The non-transitory computer readable storage medium of claim 21, further comprising instructions encoded thereon that, when executed, further cause the processor to:

access, from a computer memory, a history of scan data recorded by the secondary sensor as the primary sensor periodically rotates past the reference angle; and

at each scan-start time and scan-end time received from the primary sensor, segment the history of scan data.

26. The non-transitory computer readable storage medium of claim 25, wherein instructions for segmenting the history of scan data cause the processor to:

identify a timestamp assigned to portions of scan data recorded by the secondary sensor within a threshold of the scan-start time received from the primary sensor.

27. The non-transitory computer readable storage medium of claim 21, wherein the relative transform between the scans of the primary sensor and the secondary sensor is determined using an iterative closest point technique.

28. A computer system for calibrating sensors mounted on a vehicle, the computer system comprising:

a processor; and

a computer-readable storage medium containing computer program code that when executed, causes the processor to;

identify from a plurality of sensors mounted on the vehicle, a primary sensor and a secondary sensor;

determine a reference angle for the primary sensor;

determine, based on the reference angle of the primary sensor, a scan-start time representing a start of a scan and a scan-end time representing an end of a scan;

receive, from the primary sensor, a primary set of scan data recorded from the scan-start time to the scan-end time;

receive, from the secondary sensor, a secondary set of scan data, recorded from the scan-start time to the scan-end time;

calibrate the primary and secondary sensors by determining a relative transform for transforming points between the primary set of scan data and the secondary set of scan data;

generate a high definition map based on scan data recorded by the plurality of sensors, the scan data aggregated using the relative transform; and

store the high definition map in a computer readable storage medium.

29. The computer system of claim 28, wherein instructions for identifying the primary sensor cause the processor to:

select a sensor from the plurality of sensors that has a rotational frequency closest to a rate of performing localization for the vehicle.

30. The computer system of claim 28, wherein instructions for determining the scan-start time and scan-end time cause the processor to:

detect that the primary sensor rotates past the reference angle a first time; and

determine the scan-start time as the time at which primary sensor rotates past the reference angle;

detect that the primary sensor rotates past the reference angle a second time, the second time immediately subsequent to the first time; and

determine the scan-end time as the time at which sensor rotates past the reference angle a second time.

31. The computer system of claim 30, wherein the relative transform between the scans of the primary sensor and the secondary sensor is determined using an iterative closest point technique.

32. The computer system of claim 28, wherein the scan-end time is a scan-start time for a second primary set of scan data recorded by the primary sensor.

33. The computer system of claim 28, further comprising instructions encoded thereon that, when executed, further cause the one or more processors to:

access, from a computer memory, a history of scan data recorded by the secondary sensor as the primary sensor periodically rotates past the reference angle; and

at each scan-start time and scan-end time received from the primary sensor, segment the history of scan data.

34. The computer system of claim 33, wherein instructions for segmenting the history of scan data cause the processor to:

identify a timestamp assigned to portions of scan data recorded by the secondary sensor within a threshold of the scan-start time received from the primary sensor.