Fused raised pavement marker detection for autonomous driving using lidar and camera

• US 9,530,062 B2
• Filed: 12/23/2014
• Issued: 12/27/2016
• Est. Priority Date: 12/23/2014
• Status: Active Grant

• Current Assignee: Volkswagen AG
• Original Assignee: Volkswagen AG
• Inventors: Nguyen, Tuan Tran, Lipinski, Daniel, Spehr, Jens
• Primary Examiner(s): Bitar, Nancy
• Application Number:
  US14/580,287
• Publication Number:
  US 20160180177A1
• Time in Patent Office: 735 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06V 10/42 Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
• CPC Class Codes: G06V 10/60 relating to illumination properties, e.g. using a reflectance or lighting model
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/803 of input or preprocessed data
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road

Abstract

An in-vehicle system for estimating a lane boundary based on raised pavement markers that mark the boundary. The in-vehicle system includes a camera for obtaining image data regarding reflective raised pavement markers and non-reflective raised pavement markers, an image processor for processing frames of image data captured by the camera, a lidar detector for obtaining lidar data regarding reflective raised pavement markers, and a lidar processor for processing frames of lidar data captured by the lidar detector. The image processor generates a first probabilistic model for the lane boundary and the lidar processor generates a second probabilistic model for the lane boundary. The in-vehicle system fuses the first probabilistic model and the second probabilistic model to generate a fused probabilistic model and estimates the lane boundary based on the fused probabilistic model.

Claims

1. An in-vehicle system for estimating and tracking a lane boundary formed of pavement markers disposed on a roadway, the system comprising:

a camera located on a vehicle for obtaining image data along the roadway regarding stationary reflective raised pavement markers and non-reflective raised pavement markers as the vehicle advances along the roadway;

an image processor for processing frames of image data captured along the roadway over time by the camera and generating a first probabilistic model for the lane boundary;

a lidar detector located on the vehicle for obtaining lidar data along the roadway regarding stationary reflective raised pavement markers as the vehicle advances along the roadway;

a lidar processor for processing frames of lidar data captured along the roadway over time by the lidar detector and generating a second probabilistic model for the lane boundary; and

means for fusing the first probabilistic model and the second probabilistic model to generate a fused probabilistic model and for estimating and tracking the lane boundary based on the fused probabilistic model;

wherein the first probabilistic model includes a first mixture of Gaussian distributions;

the second probabilistic model includes a second mixture of Gaussian distributions; and

the means for fusing the first probabilistic model and the second probabilistic model includes a processor and a non-transitory data storage on which is stored computer code which, when executed on the processor, causes the in-vehicle system to compute a product of the first probabilistic model and the second probabilistic model.

2. The in-vehicle system of claim 1, wherein the image processor is in communication with a first non-transitory data storage on which is stored computer code which, when executed on the image processor, causes the image processor to generate the first probabilistic model by:

identifying candidate reflective and non-reflective raised pavement markers;

fitting a first spline to the candidate reflective and non-reflective raised pavement markers;

segmenting the first spline; and

mixing a first set of probability distributions generated based on respective segments of the first spline.

3. The in-vehicle system of claim 2, wherein the lidar processor is in communication with a second non-transitory data storage on which is stored computer code which, when executed on the lidar processor, causes the lidar processor to generate the second probabilistic model by:

identifying candidate reflective raised pavement markers;

fitting a second spline to the candidate reflective raised pavement markers;

segmenting the second spline; and

mixing a second set of probability distributions generated based on respective segments of the second spline.

4. The in-vehicle system of claim 3, further comprising a database containing statistical information regarding angular trajectories of the roadway, wherein the first non-transitory data storage and the second non-transitory data storage include respective computer code which, when executed, causes the image processor and the lidar processor to fit the first and second splines based on the statistical information stored in the database.

5. An in-vehicle system for estimating a lane boundary on a roadway, the system comprising:

one or more detectors located on a vehicle that capture image data and lidar data of raised pavement markers along the roadway over time as the vehicle advances along the roadway;

a processor and a non-transitory data storage on which is stored computer code which, when executed on the processor, causes the in-vehicle system to;

generate a first probabilistic model for the lane boundary based on the image data;

generate a second probabilistic model for the lane boundary based on the lidar data;

fuse the first probabilistic model and the second probabilistic model to generate a fused probabilistic model; and

estimate and track the lane boundary based on the fused probabilistic model;

wherein the first probabilistic model includes a first mixture of Gaussian distributions;

the second probabilistic model includes a second mixture of Gaussian distributions; and

the means for fusing the first probabilistic model and the second probabilistic model includes a processor and a non-transitory data storage on which is stored computer code which, when executed on the processor, causes the in-vehicle system to compute a product of the first probabilistic model and the second probabilistic model; and

wherein the computer code, when executed on the processor, causes the in-vehicle system to generate the first probabilistic model by;

detecting candidate raised pavement markers;

tracking the candidate raised pavement markers over time;

eliminating false positives among the candidate raised pavement markers;

fitting one or more splines to the candidate raised pavement markers;

segmenting the one or more splines; and

generating a mixture of probability distributions corresponding to respective segments of the one or more splines.

6. The in-vehicle system of claim 5, wherein the computer code, when executed on the processor, causes the in-vehicle system to generate the second probabilistic model by:

detecting candidate raised pavement markers;

tracking the candidate raised pavement markers over time;

eliminating false positives among the candidate raised pavement markers;

fitting one or more splines to the candidate raised pavement markers;

segmenting the one or more splines; and

generating a mixture of probability distributions corresponding to respective segments of the one or more splines.

7. The in-vehicle system of claim 5, further comprising

a database containing statistical information regarding angular trajectories of painted lane markings or roadway boundaries, wherein

the computer code, when executed on the processor, causes the in-vehicle system to consult the database when generating the first and second probabilistic models.

8. The in-vehicle system of claim 5, wherein the computer code, when executed on the processor, causes the in-vehicle system to fuse the first and second probabilistic models through a multiplication operation.

9. The in-vehicle system of claim 5, wherein

the one or more detectors include a camera and a lidar detector;

and

the computer code, when executed on the processor, causes the in-vehicle system to

detect candidate raised pavement markers;

determine a dynamic detection range for the lidar detector; and

identify candidate raised pavement markers falling outside the dynamic detection range as false positives.

10. The vehicle system of claim 5, wherein the dynamic detection range excludes areas where beams of the lidar detector are obstructed.

11. A computer-implemented method for estimating and tracking a lane boundary on a roadway, the method comprising:

capturing image data along the roadway by an image capture device located on a vehicle;

receiving the image data by at least one processor;

obtaining lidar data along the roadway by a lidar device located on the vehicle;

receiving lidar data by at least one processor;

generating a first probabilistic model for the lane boundary based on the image data;

generating a second probabilistic model for the lane boundary based on the lidar data;

fusing the first probabilistic model and the second probabilistic model to generate a fused probabilistic model;

estimating and tracking the lane boundary based on the fused probabilistic model;

detecting candidate reflective and non-reflective raised pavement markers in the image data;

tracking the candidate reflective and non-reflective raised pavement markers over time;

eliminating false positives among the candidate reflective and non-reflective raised pavement markers;

fitting one or more splines to the candidate reflective and non-reflective raised pavement markers;

segmenting the one or more splines; and

generating the first probabilistic model based on the segmented one or more splines;

wherein the first probabilistic model includes a first mixture of Gaussian distributions;

the second probabilistic model includes a second mixture of Gaussian distributions; and

the means for fusing the first probabilistic model and the second probabilistic model includes a processor and a non-transitory data storage on which is stored computer code which, when executed on the processor, causes the in-vehicle system to compute a product of the first probabilistic model and the second probabilistic model.

12. The method of claim 11, wherein fusing the first probabilistic model and the second probabilistic model comprises multiplying the first probabilistic model and the second probabilistic model.

13. The method of claim 11, further comprising:

detecting candidate reflective raised pavement markers in the lidar data;

tracking the candidate reflective raised pavement markers over time;

eliminating false positives among the reflective raised pavement markers;

fitting one or more splines to the candidate reflective raised pavement markers;

segmenting the one or more splines; and

generating the second probabilistic model based on the segmented one or more splines.

14. The method of claim 11, further comprising querying a database containing statistical information regarding angular trajectories of painted lane markings or roadway boundaries.

15. The method of claim 11, further comprising:

determine a dynamic detection range for the lidar detector; and

identifying candidate raised pavement markers falling outside the dynamic detection range as false positives.

16. The method of claim 15, further comprising determining the dynamic detection range by excluding areas where beams of the lidar detector are obstructed.