Lane fusion system using forward-view and rear-view cameras

• US 9,090,263 B2
• Filed: 07/20/2010
• Issued: 07/28/2015
• Est. Priority Date: 07/20/2010
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: Zeng, Shuqing
• Primary Examiner(s): Nguyen, Angela
• Application Number:
  US12/840,069
• Publication Number:
  US 20120062747A1
• Time in Patent Office: 1,834 Days
• Field of Search: 348/149
• US Class Current: 1/1
• CPC Class Codes: B60W 2552/30 Road curve radius
• CPC Class Codes: B60W 40/06 Road conditions
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road

Abstract

A method and system for computing lane curvature and a host vehicle'"'"'s position and orientation relative to lane boundaries, using image data from forward-view and rear-view cameras and vehicle dynamics sensors as input. A host vehicle includes cameras at the front and rear, which can be used to detect lane boundaries such as curbs and lane stripes, among other purposes. The host vehicle also includes vehicle dynamics sensors including vehicle speed and yaw rate. A method is developed which computes lane curvature and the host vehicle'"'"'s position relative to a lane reference path, where the lane reference path is derived from the lane boundaries extracted from a fusion of the front and rear camera images. Mathematical models provided in the disclosure include a Kalman filter tracking routine and a particle filter tracking routine.

Claims

1. A method for determining a vehicle'"'"'s position in a lane, said method comprising:

providing data from a forward-view camera module onboard the vehicle;

providing data from a rear-view camera module onboard the vehicle;

providing data from vehicle dynamics sensors onboard the vehicle; and

fusing the data from the forward-view camera module, the rear-view camera module, and the vehicle dynamics sensors into a combined set of lane geometry information to determine the vehicle'"'"'s position in the lane, including using a Kalman filter routine to jointly estimate the lane geometry information and misalignment angles of the forward-view camera module and the rear-view camera module, where the Kalman filter routine includes lane curvature, vehicle tangent angle to the lane, front bumper displacement to a lane boundary, and azimuth misalignment angles of the forward-view camera module and the rear-view camera module as state vector variables and simultaneously solves for all of the state vector variables.

2. The method of claim 1 wherein the lane geometry information includes lane curvature, vehicle orientation with respect to a tangent to the lane, and displacements to left and right lane boundaries.

3. The method of claim 1 wherein providing data from a forward-view camera module includes providing front lane geometry information from a front full-fledged lane sensing system, and providing data from a rear-view camera module includes providing rear lane geometry information from a rear full-fledged lane sensing system.

4. The method of claim 1 wherein providing data from a forward-view camera module includes providing forward image data from a front camera, and providing data from a rear-view camera module includes providing rear image data from a rear camera.

5. The method of claim 4 wherein fusing the data from the forward-view camera module, the rear-view camera module, and the vehicle dynamics sensors includes finding high intensity pixels in the forward image data and the rear image data, projecting the high intensity pixels onto a vehicle coordinate frame, detecting curbs and lane stripes from the high intensity pixels, linking the curbs and lane stripes into the lane, and tracking the lane geometry information.

6. The method of claim 4 wherein fusing the data from the forward-view camera module, the rear-view camera module, and the vehicle dynamics sensors includes using a Kalman filter.

7. The method of claim 4 wherein fusing the data from the forward-view camera module, the rear-view camera module, and the vehicle dynamics sensors includes using a particle filter.

8. The method of claim 5 wherein finding high intensity pixels in the forward image data and the rear image data includes using a Gaussian pyramid routine, where binary images at different spatial scales are subtracted from each other.

9. The method of claim 5 wherein projecting the high intensity pixels onto a vehicle coordinate frame includes using an iterative procedure to remove radial and tangential distortion, and a rotational and translational transformation procedure, to produce a plurality of high intensity pixels in the vehicle coordinate frame.

10. The method of claim 5 wherein detecting curbs and lane stripes from the high intensity pixels includes using a similarity graph and depth-first search routine, which clusters neighboring high intensity pixels into stripes.

11. The method of claim 10 wherein detecting curbs and lane stripes from the high intensity pixels also includes fitting the stripes with a line segment or an arc using a least-squares routine.

12. The method of claim 5 wherein linking the curbs and lane stripes into the lane includes finding a center of curvature of the lane using a least-squares routine applied to the curbs and lane stripes.

13. The method of claim 12 wherein tracking the lane geometry information includes computing displacements to left and right lane boundaries using a histogram routine, and computing lane curvature and vehicle orientation with respect to a tangent to the lane using the center of curvature.

14. A method for determining a vehicle'"'"'s position in a lane, said method comprising:

providing image data from a forward-view camera onboard the vehicle;

providing image data from a rear-view camera onboard the vehicle; and

fusing the image data from the forward-view camera and the rear-view camera into a combined set of lane geometry information to determine the vehicle'"'"'s position in the lane, including using a Kalman filter routine to jointly estimate the lane geometry information and misalignment angles of the forward-view camera and the rear-view camera, where the Kalman filter routine includes lane curvature, vehicle tangent angle to the lane, front bumper displacement to a lane boundary, and azimuth misalignment angles of the forward-view camera and the rear-view camera as state vector variables and simultaneously solves for all of the state vector variables.

15. The method of claim 14 wherein fusing the data from the forward-view camera and the rear-view camera includes finding high intensity pixels in the image data, projecting the high intensity pixels onto a vehicle coordinate frame, detecting curbs and lane stripes from the high intensity pixels, linking the curbs and lane stripes into the lane, and tracking the lane geometry information.

16. The method of claim 14 wherein the lane geometry information includes lane curvature, vehicle orientation with respect to a tangent to the lane, and displacements to left and right lane boundaries.

17. A system for determining a vehicle'"'"'s position in a lane, said system comprising:

a first camera for capturing images of a forward view from the vehicle;

a second camera for capturing images of a rear view from the vehicle; and

a processor configured to receive the images from the first and second cameras and compute lane geometry information using a fusion of the images from the first and second cameras, including using a Kalman filter routine to jointly estimate the lane geometry information and misalignment angles of the first camera and the second camera, where the Kalman filter routine includes lane curvature, vehicle tangent angle to the lane, front bumper displacement to a lane boundary, and azimuth misalignment angles of the first camera and the second camera as state vector variables and simultaneously solves for all of the state vector variables.

18. The system of claim 17 wherein the processor finds high intensity pixels in the images from the first and second cameras, projects the high intensity pixels onto a vehicle coordinate frame, detects curbs and lane stripes from the high intensity pixels, links the curbs and lane stripes into the lane, and tracks lane geometry information.

19. The method of claim 17 wherein the lane geometry information includes lane curvature, vehicle orientation with respect to a tangent to the lane, and displacements to left and right lane boundaries.