System and method for large-scale lane marking detection using multimodal sensor data

• US 10,528,823 B2
• Filed: 11/27/2017
• Issued: 01/07/2020
• Est. Priority Date: 11/27/2017
• Status: Active Grant

• Current Assignee: TuSimple, Inc. (TuSimple Holdings, Inc.)
• Original Assignee: TuSimple, Inc. (TuSimple Holdings, Inc.)
• Inventors: Guo, Dazhou, Wei, Yujie, Mei, Xue, Hou, Xiaodi
• Primary Examiner(s): Mistry, Oneal R
• Application Number:
  US15/822,597
• Publication Number:
  US 20190163989A1
• Time in Patent Office: 771 Days
• CPC Class Codes: G05D 1/0248 in combination with a laser lasers per se H01S
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 7/10 Segmentation; Edge detection motion-based segmentation G06T7/215
• CPC Class Codes: G06V 10/803 of input or preprocessed data
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G08G 1/167 Driving aids for lane monitoring, lane changing, e.g. blind spot detection

Abstract

A system and method for large-scale lane marking detection using multimodal sensor data are disclosed. A particular embodiment includes: receiving image data from an image generating device mounted on a vehicle; receiving point cloud data from a distance and intensity measuring device mounted on the vehicle; fusing the image data and the point cloud data to produce a set of lane marking points in three-dimensional (3D) space that correlate to the image data and the point cloud data; and generating a lane marking map from the set of lane marking points.

Claims

1. A system comprising:

a data processor; and

a multimodal lane detection module, executable by the data processor, the multimodal lane detection module being configured to perform a multimodal lane detection operation configured to;

receive image data from an image generating device mounted on a vehicle;

fit piecewise lines for each lane marking object detected in the received image data;

receive point cloud data from a distance and intensity measuring device mounted on the vehicle;

fuse the image data and the point cloud data to produce a set of lane marking points in three-dimensional (3D) space that correlate to the image data and the point cloud data, wherein the multimodal lane detection operation is configured to project 3D point cloud data on to two-dimensional (2D) image data, and to add a 3D point cloud point to the set of lane marking points if a distance between the position of the projected 3D point cloud point in 2D space and the position of at least one of the piecewise lines is within a pre-determined threshold, the pre-determined threshold being linearly decreased based on perspective depth; and

generate a lane marking map from the set of lane marking points.

2. The system of claim 1 being configured to perform a semantic segmentation operation on the received image data to identify and label objects in the image data with object category labels on a per-pixel basis.

3. The system of claim 2 being configured to train a neural network to perform the semantic segmentation operation.

4. The system of claim 1 wherein the image generating device is one or more cameras.

5. The system of claim 1 wherein the distance and intensity measuring device is one or more laser light detection and ranging (LIDAR) devices.

6. The system of claim 1 being configured to receive vehicle metrics from a vehicle subsystem.

7. The system of claim 1 being further configured to output the lane marking map to a vehicle control subsystem of the vehicle.

8. A method comprising:

receiving image data from an image generating device mounted on a vehicle;

fitting piecewise lines for each lane marking object detected in the received image data;

receiving point cloud data from a distance and intensity measuring device mounted on the vehicle;

fusing the image data and the point cloud data to produce a set of lane marking points in three-dimensional (3D) space that correlate to the image data and the point cloud data, wherein the fusing includes projecting 3D point cloud data on to two-dimensional (2D) image data, and adding a 3D point cloud point to the set of lane marking points if a distance between the position of the projected 3D point cloud point in 2D space and the position of at least one of the piecewise lines is within a pre-determined threshold, the pre-determined threshold being linearly decreased based on perspective depth; and

generating a lane marking map from the set of lane marking points.

9. The method of claim 8 including performing a semantic segmentation operation on the received image data to identify and label objects in the image data with object category labels on a per-pixel basis.

10. The method of claim 9 including training a neural network to perform the semantic segmentation operation.

11. The method of claim 8 wherein the image generating device is one or more cameras.

12. The method of claim 8 wherein the distance and intensity measuring device is one or more laser light detection and ranging (LIDAR) devices.

13. The method of claim 8 including receiving vehicle metrics from a vehicle subsystem.

14. The method of claim 8 including outputting the lane marking map to a vehicle control subsystem of the vehicle.

15. A non-transitory machine-useable storage medium embodying instructions which, when executed by a machine, cause the machine to:

receive image data from an image generating device mounted on a vehicle;

fit piecewise lines for each lane marking object detected in the received image data;

receive point cloud data from a distance and intensity measuring device mounted on the vehicle;

fuse the image data and the point cloud data to produce a set of lane marking points in three-dimensional (3D) space that correlate to the image data and the point cloud data, wherein the instructions are configured to project 3D point cloud data on to two-dimensional (2D) image data, and to add a 3D point cloud point to the set of lane marking points if a distance between the position of the projected 3D point cloud point in 2D space and the position of at least one of the piecewise lines is within a pre-determined threshold, the pre-determined threshold being linearly decreased based on perspective depth; and

generate a lane marking map from the set of lane marking points.

16. The non-transitory machine-useable storage medium of claim 15 being configured to perform a semantic segmentation operation on the received image data to identify and label objects in the image data with object category labels on a per-pixel basis.

17. The non-transitory machine-useable storage medium of claim 15 wherein the image generating device is one or more cameras.

18. The non-transitory machine-useable storage medium of claim 15 wherein the distance and intensity measuring device is one or more laser light detection and ranging (LIDAR) devices.