System and method for large-scale lane marking detection using multimodal sensor data

• US 10,657,390 B2
• Filed: 11/27/2017
• Issued: 05/19/2020
• Est. Priority Date: 11/27/2017
• Status: Active Grant

• Current Assignee: TuSimple, Inc. (TuSimple Holdings, Inc.)
• Original Assignee: TuSimple, Inc. (TuSimple Holdings, Inc.)
• Inventors: Mei, Xue, Hou, Xiaodi, Guo, Dazhou, Wei, Yujie
• Primary Examiner(s): Mistry, Oneal R
• Application Number:
  US15/822,689
• Publication Number:
  US 20190163990A1
• Time in Patent Office: 904 Days
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: G01C 21/3602 Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G01C 21/3826 Terrain data
• CPC Class Codes: G01C 21/3833 characterised by the source of data
• CPC Class Codes: G01C 21/3867 Geometry of map features, e.g. shape points, polygons or for simplified maps
• CPC Class Codes: G01S 17/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S17/46
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 7/4802 using analysis of echo signal for target characterisation; Target signature; Target cross-section
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/10 Segmentation; Edge detection motion-based segmentation G06T7/215
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G07C 5/008 communicating information to a remotely located station transmission systems for measured values G08C

Abstract

A system and method for large-scale lane marking detection using multimodal sensor data are disclosed. A particular embodiment includes: receiving image data from an image generating device mounted on a vehicle; receiving point cloud data from a distance and intensity measuring device mounted on the vehicle; fusing the image data and the point cloud data to produce a set of lane marking points in three-dimensional (3D) space that correlate to the image data and the point cloud data; and generating a lane marking map from the set of lane marking points.

Claims

1. A system comprising:

a data processor; and

a multimodal lane detection module, executable by the data processor, the multimodal lane detection module being configured to perform a multimodal lane detection operation configured to;

receive image data from an image generating device mounted on a vehicle, the received image data corresponding to a particular location;

receive point cloud data from a distance and intensity measuring device mounted on the vehicle;

fuse the image data and the point cloud data to produce a set of lane marking points in three-dimensional (3D) space that correlate to the image data and the point cloud data, the fusion including aligning and orienting the image data with a terrain map corresponding to the particular location and using terrain map elevation data to transform the image data to the 3D space; and

generate a lane marking map from the set of lane marking points.

2. The system of claim 1 being configured to perform a semantic segmentation operation on the received image data to identify and label objects in the image data with object category labels on a per-pixel basis.

3. The system of claim 2 being configured to train a neural network to perform the semantic segmentation operation.

4. The system of claim 1 wherein the image generating device is one or more cameras.

5. The system of claim 1 wherein the distance and intensity measuring device is one or more laser light detection and ranging (LIDAR) devices.

6. The system of claim 1 being configured to receive vehicle metrics from a vehicle subsystem.

7. The system of claim 1 being configured to back-project the image data on the terrain map with the terrain map elevation data.

8. The system of claim 1 being further configured to output the lane marking map to a vehicle control subsystem of the vehicle.

9. A method comprising:

receiving image data from an image generating device mounted on a vehicle, the received image data corresponding to a particular location;

receiving point cloud data from a distance and intensity measuring device mounted on the vehicle;

fusing the image data and the point cloud data to produce a set of lane marking points in three-dimensional (3D) space that correlate to the image data and the point cloud data the fusing including aligning and orienting the image data with a terrain map corresponding to the particular location and using terrain map elevation data to transform the image data to the 3D space; and

generating a lane marking map from the set of lane marking points.

10. The method of claim 9 including performing a semantic segmentation operation on the received image data to identify and label objects in the image data with object category labels on a per-pixel basis.

11. The method of claim 10 including training a neural network to perform the semantic segmentation operation.

12. The method of claim 9 wherein the image generating device is one or more cameras.

13. The method of claim 9 wherein the distance and intensity measuring device is one or more laser light detection and ranging (LIDAR) devices.

14. The method of claim 9 including receiving vehicle metrics from a vehicle subsystem.

15. The method of claim 9 including back-projecting the image data on the terrain map with the terrain map elevation data.

16. The method of claim 9 including outputting the lane marking map to a vehicle control subsystem of the vehicle.

17. A non-transitory machine-useable storage medium embodying instructions which, when executed by a machine, cause the machine to:

receive image data from an image generating device mounted on a vehicle, the received image data corresponding to a particular location;

receive point cloud data from a distance and intensity measuring device mounted on the vehicle;

fuse the image data and the point cloud data to produce a set of lane marking points in three-dimensional (3D) space that correlate to the image data and the point cloud data, the fusion including aligning and orienting the image data with a terrain map corresponding to the particular location and using terrain map elevation data to transform the image data to the 3D space; and

generate a lane marking map from the set of lane marking points.

18. The non-transitory machine-useable storage medium of claim 17 being configured to perform a semantic segmentation operation on the received image data to identify and label objects in the image data with object category labels on a per-pixel basis.

19. The non-transitory machine-useable storage medium of claim 17 being further configured to fit piecewise lines for each lane marking object detected in the received image data.

20. The non-transitory machine-useable storage medium of claim 17 wherein the distance and intensity measuring device is one or more laser light detection and ranging (LIDAR) devices.