FUSED RAISED PAVEMENT MARKER DETECTION FOR AUTONOMOUS DRIVING USING LIDAR AND CAMERA

• US 20160180177A1
• Filed: 12/23/2014
• Published: 06/23/2016
• Est. Priority Date: 12/23/2014
• Status: Active Grant

• Current Assignee: Volkswagen AG
• Original Assignee: Volkswagen AG
• Inventors: NGUYEN, Tuan Tran, LIPINSKI, Daniel, SPEHR, Jens
• Granted Patent: US 9,530,062 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06V 10/42 Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
• CPC Class Codes: G06V 10/60 relating to illumination properties, e.g. using a reflectance or lighting model
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/803 of input or preprocessed data
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road

Abstract

An in-vehicle system for estimating a lane boundary based on raised pavement markers that mark the boundary. The in-vehicle system includes a camera for obtaining image data regarding reflective raised pavement markers and non-reflective raised pavement markers, an image processor for processing frames of image data captured by the camera, a lidar detector for obtaining lidar data regarding reflective raised pavement markers, and a lidar processor for processing frames of lidar data captured by the lidar detector. The image processor generates a first probabilistic model for the lane boundary and the lidar processor generates a second probabilistic model for the lane boundary. The in-vehicle system fuses the first probabilistic model and the second probabilistic model to generate a fused probabilistic model and estimates the lane boundary based on the fused probabilistic model.

Claims

1. An in-vehicle system for estimating a lane boundary formed of raised pavement markers disposed on a roadway, the system comprising:

a camera for obtaining image data regarding reflective raised pavement markers and non-reflective raised pavement markers;

an image processor for processing frames of image data captured by the camera and generating a first probabilistic model for the lane boundary;

a lidar detector for obtaining lidar data regarding reflective raised pavement markers;

a lidar processor for processing frames of lidar data captured by the lidar detector and generating a second probabilistic model for the lane boundary; and

means for fusing the first probabilistic model and the second probabilistic model to generate a fused probabilistic model and for estimating the lane boundary based on the fused probabilistic model.

2. The in-vehicle system of claim 1, wherein

the first probabilistic model includes a first mixture of Gaussian distributions;

the second probabilistic model includes a second mixture of Gaussian distributions; and

the means for fusing the first probabilistic model and the second probabilistic model includes a processor and a non-transitory data storage on which is stored computer code which, when executed on the processor, causes the in-vehicle system to compute a product of the first probabilistic model and the second probabilistic model.

3. The in-vehicle system of claim 1, wherein the image processor is in communication with a first non-transitory data storage on which is stored computer code which, when executed on the image processor, causes the image processor to generate the first probabilistic model by:

identifying candidate reflective and non-reflective raised pavement markers;

fitting a first spline to the candidate reflective and non-reflective raised pavement markers;

segmenting the first spline; and

mixing a first set of probability distributions generated based on respective segments of the first spline.

4. The in-vehicle system of claim 3, wherein the lidar processor is in communication with a second non-transitory data storage on which is stored computer code which, when executed on the lidar processor, causes the lidar processor to generate the second probabilistic model by:

identifying candidate reflective raised pavement markers;

fitting a second spline to the candidate reflective raised pavement markers;

segmenting the second spline; and

mixing a second set of probability distributions generated based on respective segments of the second spline.

5. The in-vehicle system of claim 4, further comprising a database containing statistical information regarding angular trajectories of the roadway, wherein the first non-transitory data storage and the second non-transitory data storage include respective computer code which, when executed, causes the image processor and the lidar processor to fit the first and second splines based on the statistical information stored in the database.

6. An in-vehicle system for estimating a lane boundary on a roadway, the system comprising:

one or more detectors that capture image data and lidar data of raised pavement markers on the roadway;

a processor and a non-transitory data storage on which is stored computer code which, when executed on the processor, causes the in-vehicle system to;

generate a first probabilistic model for the lane boundary based on the image data;

generate a second probabilistic model for the lane boundary based on the lidar data;

fuse the first probabilistic model and the second probabilistic model to generate a fused probabilistic model; and

estimate the lane boundary based on the fused probabilistic model.

7. The in-vehicle system of claim 6, wherein the computer code, when executed on the processor, causes the in-vehicle system to generate the first probabilistic model by:

detecting candidate raised pavement markers;

tracking the candidate raised pavement markers over time;

eliminating false positives among the candidate raised pavement markers;

fitting one or more splines to the candidate raised pavement markers;

segmenting the one or more splines; and

generating a mixture of probability distributions corresponding to respective segments of the one or more splines.

8. The in-vehicle system of claim 6, wherein the computer code, when executed on the processor, causes the in-vehicle system to generate the second probabilistic model by:

detecting candidate raised pavement markers;

tracking the candidate raised pavement markers over time;

eliminating false positives among the candidate raised pavement markers;

fitting one or more splines to the candidate raised pavement markers;

segmenting the one or more splines; and

generating a mixture of probability distributions corresponding to respective segments of the one or more splines.

9. The in-vehicle system of claim 6, further comprising

a database containing statistical information regarding angular trajectories of painted lane markings or roadway boundaries, wherein

the computer code, when executed on the processor, causes the in-vehicle system to consult the database when generating the first and second probabilistic models.

10. The in-vehicle system of claim 6, wherein the computer code, when executed on the processor, causes the in-vehicle system to fuse the first and second probabilistic models through a multiplication operation.

12. The in-vehicle system of claim 6, wherein

the one or more detectors include a camera and a lidar detector;

and

the computer code, when executed on the processor, causes the in-vehicle system to

detect candidate raised pavement markers;

determine a dynamic detection range for the lidar detector; and

identify candidate raised pavement markers falling outside the dynamic detection range as false positives.

13. The vehicle system of claim 6, wherein the dynamic detection range excludes areas where beams of the lidar detector are obstructed.

11. The in-vehicle system of claim 11, wherein the first probabilistic model and the second probabilistic model comprise respective mixtures of Gaussian distributions.

14. A method for estimating a lane boundary on a roadway, the method comprising:

receiving image data;

receiving lidar data;

generating a first probabilistic model for the lane boundary based on the image data;

generating a second probabilistic model for the lane boundary based on the lidar data;

fusing the first probabilistic model and the second probabilistic model to generate a fused probabilistic model; and

estimating the lane boundary based on the fused probabilistic model.

15. The method of claim 14, wherein fusing the first probabilistic model and the second probabilistic model comprises multiplying the first probabilistic model and the second probabilistic model.

16. The method of claim 14, further comprising:

detecting candidate reflective and non-reflective raised pavement markers in the image data;

tracking the candidate reflective and non-reflective raised pavement markers over time;

eliminating false positives among the candidate reflective and non-reflective raised pavement markers;

fitting one or more splines to the candidate reflective and non-reflective raised pavement markers;

segmenting the one or more splines; and

generating the first probabilistic model based on the segmented one or more splines.

17. The method of claim 14, further comprising:

detecting candidate reflective raised pavement markers in the lidar data;

tracking the candidate reflective raised pavement markers over time;

eliminating false positives among the reflective raised pavement markers;

fitting one or more splines to the candidate reflective raised pavement markers;

segmenting the one or more splines; and

generating the second probabilistic model based on the segmented one or more splines.

18. The method of claim 14, further comprising querying a database containing statistical information regarding angular trajectories of painted lane markings or roadway boundaries.

19. The method of claim 14, further comprising:

determine a dynamic detection range for the lidar detector; and

identifying candidate raised pavement markers falling outside the dynamic detection range as false positives.

20. The method of claim 19, further comprising determining the dynamic detection range by excluding areas where beams of the lidar detector are obstructed.