Radar guided vision system for vehicle validation and vehicle motion characterization

• US 8,355,539 B2
• Filed: 06/26/2008
• Issued: 01/15/2013
• Est. Priority Date: 09/07/2007
• Status: Expired due to Fees

• Current Assignee: SRI International, Inc., Visteon Corporation
• Original Assignee: SRI International, Inc., Visteon Global Technologies Incorporated (Visteon Corporation)
• Inventors: Tan, Yi, Han, Feng, Eledath, Jayan, Kumar, Rakesh, Ibrahim, Faroog Abdel-kareem
• Primary Examiner(s): Werner, Brian P
• Assistant Examiner(s): GORADIA, SHEFALI DINESH
• Application Number:
  US12/146,897
• Publication Number:
  US 20090067675A1
• Time in Patent Office: 1,664 Days
• Field of Search: 382/100, 382/104
• US Class Current: 382/104
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 2013/93271 in the front of the vehicles
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road

Abstract

A method for determining whether a target vehicle in front of a host vehicle intends to change lanes using radar data and image data is disclosed, comprising the steps of processing the image data to detect the boundaries of the lane of the host vehicle; estimating a ground plane by determining a projected vanishing point of the detected lane boundaries; using a camera projection matrix to map the target vehicle from the radar data to image coordinates; and determining lane change intentions of the target vehicle based on a moving trajectory and an appearance change of the target vehicle. Determining lane change intentions based on a moving trajectory of the target vehicle is based on vehicle motion trajectory relative to the center of the lane such that the relative distance of the target vehicle from the center of the lane follows a predetermined trend. Determining lane change intentions based on an appearance change of the target vehicle is based on a template that tracks changes to the appearance of the rear part of the target vehicle due to rotation.

Claims

1. A method for determining whether a target vehicle in front of a host vehicle intends to change lanes using radar data and image data, comprising the steps of:

processing the image data to detect boundaries of a lane of the host vehicle;

estimating a ground plane by determining a projected vanishing point of the detected lane boundaries;

using a camera projection matrix to map the target vehicle from the radar data to image coordinates; and

determining lane change intentions of the target vehicle based on the target vehicle motion trajectory relative to the center of the lane such that the relative distance of the target vehicle from the center of the lane follows a predetermined trend.

2. The method of claim 1, further comprising the step of tracking the target vehicle, wherein said step of determining lane change intentions based on an appearance change of the target vehicle is based on a template that tracks changes to the appearance of the rear part of the target vehicle due to rotation.

3. The method of claim 2, wherein the Kanade-Lucas-Tomasi Feature Tracker (KLT) is used for tracking the target vehicle.

4. The method of claim 2, wherein said step of determining lane change intentions based on an appearance change of the target vehicle further comprises the steps of:

encoding an appearance relationship between the tracked target vehicle and the template in warping parameters;

feeding the warping parameters to a support vehicle machine (SVM); and

using the SVM to classify the target vehicle as intending to change lanes or not intending to change lanes.

5. The method of claim 1, further comprising the step of creating a virtual lane and virtual lane boundaries when real lane boundaries cannot be obtained.

6. The method of claim 1, wherein said step of processing the image data to detect the boundaries of a lane of the host vehicle further comprises the steps of:

(a) extracting features from at least one image of the image data that represent potential lane boundaries;

(b) calculating the Laplacian of the at least one image and applying multiple thresholds to horizontal gradients to generate tri-value outputs, wherein high positive gradients are mapped to white, high negative gradients are mapped to black, and remaining pixels are mapped to gray;

(c) computing a search region in the image by projecting two elongated rectangular areas one the two sides of the host vehicle through a camera calibration matrix;

(d) along every row in each of the two rectangular areas, correlating a transition with the pixels in the row;

(e) selecting peaks possessing a predetermined strength as boundary feature points;

(f) using a Hough transform to obtain a proposed fit of the lane boundaries to a straight line;

(g) determined whether a fitness metric for a correct lane boundaries estimation exceeds a predetermined threshold; and

(h) repeating steps (a)-(g) if the predetermined threshold is not exceeded, otherwise a accepting the proposed fit as genuine lane boundaries.

7. The method of claim 6, wherein the projected vanishing point of the detected lane boundaries is determined from the projected intersections of the accepted lane boundaries.

8. The method of claim 1, further comprising the step of tracking the boundaries of the lane.

9. The method of claim 8, wherein said step of tracking the boundaries of the lane further comprise the steps of:

using an Intraframe-Kalman filter to predict the position of the target vehicle in a current frame from estimates of lane boundaries in a previous frame; and

applying an Interframe-Kalman filter to incorporate features within the current frame into the Intraframe-Kalman filter'"'"'s estimate.

10. The method of claim 1, further comprising the step of determining whether the target vehicle is a valid vehicle.

11. The method of claim 10, further comprising the steps of:

representing the target vehicle using a histogram of oriented gradient (HOG) representation;

feeding the HOG representation to a support vehicle machine (SVM); and

using the SVM to classify whether the target vehicle is a valid target vehicle.

12. The method of claim 11, wherein said step of representing the target vehicle using a histogram of oriented gradient (HOG) representation further comprises the steps of:

computing the gradient magnitude and orientation at each image sample point in a region around an anchor point;

splitting the region into N×

N sub-regions;

forming an orientation histogram for each sub-region by accumulating samples within the sub-region, weighted by gradient magnitudes;

concatenating the histograms from all the sub-regions into a final HOG feature vector; and

extending the HOG by incorporating spatial locality.

13. The method of claim 12, wherein spatial locality is incorporated by the steps of:

adding one distance dimension to an angle dimension in the binning of all the pixels within each sub-region;

accumulating a local I-D histogram of gradient directions over the pixels of a cell; and

combining the histogram entries to form an extended HOG representation.

14. A system for detecting whether a target vehicle in front of a host vehicle intends to change lanes, comprising:

a camera for receiving image data;

a radar system for receiving radar data; and

a processor communicatively connected to said camera and said radar system, said processor being configured for;

processing the image data to detect boundaries of a lane of the host vehicle;

estimating a ground plane by determining a projected vanishing point of the detected lane boundaries;

using a camera projection matrix to map the target vehicle from the radar data to image coordinates; and

determining lane change intentions of the target vehicle based on the target vehicle motion trajectory relative to the center of the lane such that the relative distance of the target vehicle from the center of the lane follows a predetermined trend.

15. The system of claim 14, further comprising the step of tracking the target vehicle, wherein said step of determining lane change intentions based on an appearance change of the target vehicle is based on a template that tracks changes to the appearance of the rear part of the target vehicle due to rotation.

16. The system of claim 14, wherein the step of determining lane change intentions based on an appearance change of the target vehicle further comprises the steps of:

encoding an appearance relationship between the tracked target vehicle and the template in warping parameters;

feeding the warping parameters to a support vehicle machine (SVM); and

using the SVM to classify the target vehicle as intending to change lanes or not intending to change lanes.

17. A non-transitory computer-readable medium carrying one or more sequences for determining whether a target vehicle in front of a host. vehicle intends to change lanes using radar data and image data, wherein execution of the one of more sequences of instructions by one or more processors causes the one or more processors to perform the steps of:

processing the image data to detect boundaries of the lane of the host vehicle;

estimating a ground plane by determining a projected vanishing point of the detected lane boundaries;

using a camera projection matrix to map the target vehicle from the radar data to image coordinates; and

determining lane change intentions of the target vehicle based on a template that tracks changes to the appearance of the rear part of the target vehicle due to rotation.

18. The computer readable medium of claim 17, wherein the step of determining lane change intentions based on a moving trajectory of the target vehicle is based on vehicle motion trajectory relative to the center of the lane such that the relative distance of the target vehicle from the center of the lane follows a predetermined trend.

19. The computer readable medium of claim 17, wherein the step of determining lane change intentions based on an appearance change of the target vehicle further comprises the steps of:

encoding an appearance relationship between the tracked target vehicle and the template in warping parameters;

feeding the warping parameters to a support vehicle machine (SVM); and

using the SVM to classify the target vehicle as intending to change lanes or not intending to change lanes.