Rolling shutter correction for images captured by a camera mounted on a moving vehicle

• US 10,498,966 B2
• Filed: 10/16/2018
• Issued: 12/03/2019
• Est. Priority Date: 10/19/2017
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: DeepMap, Inc. (NVIDIA Corporation)
• Inventors: Wheeler, Mark Damon, Yang, Lin
• Primary Examiner(s): Ren, Zhubing
• Application Number:
  US16/162,224
• Publication Number:
  US 20190222736A1
• Time in Patent Office: 413 Days
• CPC Class Codes: G01C 21/1652 with ranging devices, e.g. LIDAR or RADAR
• CPC Class Codes: G01C 21/3602 Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G01C 25/00 Manufacturing, calibrating, cleaning, or repairing instruments or devices referred to in the other groups of this subclass testing, calibrating or compensating compasses G01C17/38
• CPC Class Codes: G01S 17/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S17/46
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 7/4817 relating to scanning
• CPC Class Codes: G01S 7/497 Means for monitoring or calibrating
• CPC Class Codes: G01S 7/4972 Alignment of sensor
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0231 using optical position detecting means position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16
• CPC Class Codes: G05D 1/0248 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0287 involving a plurality of land vehicles, e.g. fleet or convoy travelling traffic control systems for road vehicles G08G1/00, particularly anticollision systems G08G1/16
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20092 Interactive image processing based on input by user
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30242 : Counting objects in image
• CPC Class Codes: G06T 2207/30252 : Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/13 : Edge detection
• CPC Class Codes: G06T 7/33 : using feature-based methods
• CPC Class Codes: G06T 7/55 : from multiple images
• CPC Class Codes: G06T 7/80 : Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 13/106 : Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 23/54 : Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/60 : Control of cameras or camera modules
• CPC Class Codes: H04N 23/689 : Motion occurring during a rolling shutter mode
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/04 : Synchronising for television systems using pulse code modulation H04N7/56

Abstract

A system performs rolling shutter correction to transform data captured by sensors of a vehicle, for example, cameras or lidar mounted on a vehicle, for example, an autonomous vehicle. The images captured by a rolling shutter camera mounted on a moving vehicle show rolling shutter distortion. The rolling shutter correction transforms the data representing points of scenes to perform rolling shutter compensation, i.e., compensation for the rolling shutter distortion. The rolling shutter compensation ensures that data representing as three dimensional points, for example, data captured by lidar is consistent with data represented in images captured by a rolling shutter camera. The system performs rolling shutter compensation by estimating a distance travelled by the vehicle between the time that the camera captured a point and the time that the image scan was completed and translating the 3D points by the estimated distance.

Claims

1. A computer implemented method for performing rolling shutter correction of data captured by sensors of a vehicle, comprising:

receiving an image captured by a rolling shutter camera mounted on a vehicle, the rolling shutter camera capturing the image via an image scan, the image comprising a plurality of scan-lines, the image associated with a distortion axis, wherein each scan-line of the image maps to a distinct point along the distortion axis;

identifying a plurality of three-dimensional (3D) data points corresponding to the scene captured by the image, each 3D data point having a location in a 3D space;

for each of the plurality of 3D points;

projecting the 3D point to the image coordinates to obtain a projected point;

determining based on the position of the projected point, an estimate of a distance travelled by the vehicle between the time of capture of the 3D point by the rolling shutter camera and the time of completion of the image scan;

translating the 3D point to location along a direction of movement of the vehicle by the estimate of the distance;

storing the translated 3D points for displaying in conjunction with the image captured by the rolling shutter camera.

2. The method of claim 1, further comprising:

project the plurality of translated 3D points on the image to obtain an overlapping image wherein the projection of the 3D points is aligned with the pixels of the image; and

configuring a user interface for displaying the overlapping image.

3. The method of claim 2, wherein the user interface is displayed by a toolkit for development of a high definition map.

4. The method of claim 3, wherein the vehicle is an autonomous vehicle, further comprising:

sending signals to the controls of the autonomous vehicle based on the high definition map.

5. The method of claim 1, wherein image is associated with a reference point, wherein determining the estimate of a distance travelled by the vehicle comprises:

determining a distance of the projected point from the reference point along the distortion axis; and

determining an estimate of a distance travelled by the vehicle between the time that the 3D point was captured by the rolling shutter camera and the time that the image scan was completed, the estimate of the distance travelled determined based on the distance of the projected point from the reference point along the distortion axis.

6. The method of claim 1, wherein the plurality of 3D points are determined using a lidar scan.

7. The method of claim 1, wherein the plurality of 3D points are determined using sensor data captured by one or more vehicles previously travelling along the path of the moving vehicle.

8. The method of claim 1, further comprising, mapping a shape displayed in the image to 3D space, comprising:

projecting a set of the translated 3D points to the image;

selecting a subset of the set of translated 3D points, the subset comprising translated 3D points whose projection falls within the shape displayed in the image;

fitting a plane through the selected subset of translated 3D points; and

determining a depth of a 2D point within the shape displayed in the image by projecting the 2D point to the fitted plane.

9. A non-transitory computer readable storage medium storing instructions for performing rolling shutter correction of data captured by sensors of a vehicle, wherein the instructions when executed by a processor, cause the processor to perform the steps comprising:

receiving an image captured by a rolling shutter camera mounted on a vehicle, the rolling shutter camera capturing the image via an image scan, the image comprising a plurality of scan-lines, the image associated with a distortion axis, wherein each scan-line of the image maps to a distinct point along the distortion axis;

identifying a plurality of three-dimensional (3D) data points corresponding to the scene captured by the image, each 3D data point having a location in a 3D space;

for each of the plurality of 3D points;

projecting the 3D point to the image coordinates to obtain a projected point;

determining based on the position of the projected point, an estimate of a distance travelled by the vehicle between the time of capture of the 3D point by the rolling shutter camera and the time of completion of the image scan;

translating the 3D point to location along a direction of movement of the vehicle by the estimate of the distance;

storing the translated 3D points for displaying in conjunction with the image captured by the rolling shutter camera.

10. The non-transitory computer readable medium of claim 9, wherein the stored instructions further cause the processor to perform the steps comprising:

projecting the plurality of translated 3D points on the image to obtain an overlapping image wherein the projection of the 3D points is aligned with the pixels of the image; and

configuring a user interface for displaying the overlapping image.

11. The non-transitory computer readable medium of claim 10, wherein the user interface is displayed by a toolkit for development of a high definition map.

12. The non-transitory computer readable medium of claim 11, wherein the vehicle is an autonomous vehicle, wherein the stored instructions further cause the processor to perform the steps comprising:

sending signals to the controls of the autonomous vehicle based on the high definition map.

13. The non-transitory computer readable medium of claim 9, wherein image is associated with a reference point, wherein instructions for determining the estimate of a distance travelled by the vehicle comprise instructions:

determining a distance of the projected point from the reference point along the distortion axis; and

determining an estimate of a distance travelled by the vehicle between the time that the 3D point was captured by the rolling shutter camera and the time that the image scan was completed, the estimate of the distance travelled determined based on the distance of the projected point from the reference point along the distortion axis.

14. The non-transitory computer readable medium of claim 9, wherein the plurality of 3D points are determined using a lidar scan.

15. The non-transitory computer readable medium of claim 9, wherein the plurality of 3D points are determined using sensor data captured by one or more vehicles previously travelling along the path of the moving vehicle.

16. The non-transitory computer readable medium of claim 9, wherein the stored instructions further cause the processor to map a shape displayed in the image to 3D space by performing the steps of:

projecting a set of the translated 3D points to the image;

selecting a subset of the set of translated 3D points, the subset comprising translated 3D points whose projection falls within the shape displayed in the image;

fitting a plane through the selected subset of translated 3D points; and

determining a depth of a 2D point within the shape displayed in the image by projecting the 2D point to the fitted plane.

17. A computer system comprising:

a processor; and

a non-transitory computer readable storage medium storing instructions for performing rolling shutter correction of data captured by sensors of a vehicle, wherein the instructions when executed by a processor, cause the processor to perform the steps comprising;

receiving an image captured by a rolling shutter camera mounted on a vehicle, the rolling shutter camera capturing the image via an image scan, the image comprising a plurality of scan-lines, the image associated with a distortion axis, wherein each scan-line of the image maps to a distinct point along the distortion axis;

identifying a plurality of three-dimensional (3D) data points corresponding to the scene captured by the image, each 3D data point having a location in a 3D space;

for each of the plurality of 3D points;

projecting the 3D point to the image coordinates to obtain a projected point;

determining based on the position of the projected point, an estimate of a distance travelled by the vehicle between the time of capture of the 3D point by the rolling shutter camera and the time of completion of the image scan;

translating the 3D point to location along a direction of movement of the vehicle by the estimate of the distance;

storing the translated 3D points for displaying in conjunction with the image captured by the rolling shutter camera.

18. The computer system of claim 17, wherein the stored instructions further cause the processor to perform the steps comprising:

projecting the plurality of translated 3D points on the image to obtain an overlapping image wherein the projection of the 3D points is aligned with the pixels of the image; and

configuring a user interface for displaying the overlapping image.

19. The computer system of claim 17, wherein image is associated with a reference point, wherein instructions for determining the estimate of a distance travelled by the vehicle comprise instructions:

determining a distance of the projected point from the reference point along the distortion axis; and

determining an estimate of a distance travelled by the vehicle between the time that the 3D point was captured by the rolling shutter camera and the time that the image scan was completed, the estimate of the distance travelled determined based on the distance of the projected point from the reference point along the distortion axis.

20. The computer system of claim 17, wherein the stored instructions further cause the processor to map a shape displayed in the image to 3D space by performing the steps of:

projecting a set of the translated 3D points to the image;

selecting a subset of the set of translated 3D points, the subset comprising translated 3D points whose projection falls within the shape displayed in the image;

fitting a plane through the selected subset of translated 3D points; and

determining a depth of a 2D point within the shape displayed in the image by projecting the 2D point to the fitted plane.