Apparatus and method for object detection and tracking and roadway awareness using stereo cameras

• US 8,108,119 B2
• Filed: 04/23/2007
• Issued: 01/31/2012
• Est. Priority Date: 04/21/2006
• Status: Active Grant

• Current Assignee: ZAMA Innovations LLC
• Original Assignee: SRI International, Inc.
• Inventors: Southall, John Benjamin, Bansal, Mayank, Jain, Aastha, Kumar, Manish, Camus, Theodore Armand, Fields, John Richard, Eledath, Jayakrishnan, Das, Aveek Kumar, Greene, Gary Alan
• Primary Examiner(s): Black, Thomas
• Assistant Examiner(s): Nolan, Peter D
• Application Number:
  US11/739,068
• Publication Number:
  US 20070255480A1
• Time in Patent Office: 1,744 Days
• Field of Search: 701/93, 701/96, 701/300, 701/301, 382/102, 382/104, 382/106, 382/107, 382/168, 382/170, 382/171, 382/172
• US Class Current: 701/96
• CPC Class Codes: B60T 2201/089 using optical detection
• CPC Class Codes: B60T 7/22 initiated by contact of vehicle, e.g. bumper, with an external object, e.g. another vehicle , or by means of contactless obstacle detectors mounted on the vehicle
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 30/08 Active safety systems predicting or avoiding probable or impending collision or attempting to minimise its consequences
• CPC Class Codes: B60W 30/095 Predicting travel path or likelihood of collision
• CPC Class Codes: B60W 30/16 Control of distance between vehicles, e.g. keeping a distance to preceding vehicle
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G08G 1/167 Driving aids for lane monitoring, lane changing, e.g. blind spot detection

Abstract

The present invention provides a collision avoidance apparatus and method employing stereo vision applications for adaptive vehicular control. The stereo vision applications are comprised of a road detection function and a vehicle detection and tracking function. The road detection function makes use of three-dimensional point data, computed from stereo image data, to locate the road surface ahead of a host vehicle. Information gathered by the road detection function is used to guide the vehicle detection and tracking function, which provides lead motion data to a vehicular control system of the collision avoidance apparatus. Similar to the road detection function, stereo image data is used by the vehicle detection and tracking function to determine the depth of image scene features, thereby providing a robust means for identifying potential lead vehicles in a headway direction of the host vehicle.

Claims

1. A computer-implemented method for adaptive regulation of distance between a host vehicle and a lead vehicle, comprising:

receiving, at an image processing unit, a stereo image of a roadway scene;

locating, using a road detection and awareness module at said image processing unit, said roadway scene in a headway direction of said host vehicle,

computing, using said road detection and awareness module, a plurality of vehicle detection gates, wherein each of said vehicle detection gates specifies a region of interest (ROI) of the roadway scene of the stereo image and a depth interval between said vehicle detection gates of the roadway scene of the stereo image, said each vehicle detection gate overlaps a portion of an adjacent detection gate;

identifying, using said road detection and awareness module, at least two lane boundary markers in said stereo image of said roadway scene;

retrieving, using said road detection and awareness module, ground plane data, said lane boundary markers, and said depth interval/range between said vehicle detection gates;

computing, using said road detection and awareness module, depth for each said vehicle detection gate using the depth interval and computing said region of interest (ROI) for each said vehicle detection gate using the ground plane data and the lane marker positions, wherein said ROI is a projection in the image of the roadway scene of height and depth equal to the depth of the vehicle detection gate lying between two lane boundary makers and the ground;

detecting, using a vehicular detection and tracking module at said image processing unit, potential lead vehicle detections among said vehicle detection gates on said roadway scene in said headway direction of said host vehicle, wherein said detecting comprising searching for edge features/pixels for the potential lead vehicle in said depth range within said ROI;

receiving a plurality of left images and a plurality of right images of the stereo image for each said gate ROI;

sub-sampling said plurality of the left images and the right images along a horizontal axis; and

filtering said sub-sampled left images and said sub-sampled right images,

applying a vertical sobel application to said sub-sampled left images to compute vertical left sobel edge images, wherein said vertical left sobel edge images are candidate image pixels for left edge pixels;

selecting said left candidate image pixels as said left edge pixels based on a threshold;

computing an intensity histogram for said selected left edge pixels to obtain a total number of said selected left sobel edge pixels and;

thresholding the vertical left sobel image using the intensity histogram for said selected right edge pixels to produce a left edge pyramid, wherein said left edge pyramid is a binary image comprising said selected left sobel edge pixels.

2. The method of claim 1 further comprising:

receiving, at the image processing unit, a preset cruise speed for said host vehicle;

verifying, using said vehicular detection and tracking module, said potential lead vehicle detections to be in the headway direction of the host vehicle;

tracking, using said vehicular detection and tracking module, said verified lead vehicle detection and estimating forward and lateral motion of said lead vehicle with respect to said host vehicle; and

adjusting, using an ACC controller, said preset cruise speed of said host vehicle based on said estimated forward and lateral motion data associated with said lead vehicle.

3. The method of claim 1 further comprising the steps of:

(d) applying a vertical sobel application to said sub-sampled right images to compute vertical right sobel edge images, wherein said vertical right sobel edge images are candidate image pixels for right edge pixels;

(e) selecting said right candidate image pixels as said right edge pixels based on said threshold;

(f) computing an intensity histogram for said selected right edge pixels to obtain a total number of said selected right sobel edge pixels;

(g) thresholding the vertical right sobel image using the intensity histogram for said selected right edge pixels to produce a right edge pyramid, wherein said right edge pyramid is a binary image comprising said selected right sobel edge pixels.

4. The method of claim 3 further comprising:

executing stereo matching between the left and the right edge pyramids to yield an estimated disparity/depth data for every edge pixel in one of the left edge image and the right edge image;

computing a 2D disparity histogram using said disparity data and 3D location of each of the vertical image pixels; and

locating and identifying peaks in said 2D disparity histogram;

wherein said peaks comprise vertical image pixels likely to be outer edges of the left edge images and the right edge images of the potential lead vehicle detection.

5. The method of claim 4 further comprising computing coarse disparity of the edge pixels for the potential lead vehicle detections, wherein said coarse disparity is an integral disparity value that best aligns a with majority of the edge pixels from the computed left and right edge pyramids for each said gate ROI.

6. The method of claim 5 further comprising:

selecting a left peak in the left outer edge image and a right peak in the right outer edge image;

computing an average disparity of all the pixels between said left and right peaks to compute a refined disparity/depth data of the edge pixels for the potential lead vehicle detections;

defining a region to be a detection ROI;

said detection ROI defining a region including the pixels at the selected left and the right peaks and the pixels in between the selected left and the right peaks.

7. The method of claim 6 wherein said verifying further comprising:

applying a false positive removal scheme to each of the detection ROI of the potential lead vehicle detections;

removing multiple detections of the detection ROIs of the lead vehicle to yield a set of unique detections, wherein each said unique detection is associated to a separate one of said potential lead vehicle detections; and

applying a temporal consistency based false positive removal scheme to the set of unique detections to remove false detections from the set of unique detections.

8. A computer-implemented method for adaptive regulation of distance between a host vehicle and a lead vehicle, comprising:

receiving, at an image processing unit, a stereo image of a roadway scene;

locating, using a road detection and awareness module at said image processing unit, said roadway scene in a headway direction of said host vehicle,

computing, using said road detection and awareness module, a plurality of vehicle detection gates, wherein each of said vehicle detection gates specifies a region of interest (ROI) of the roadway scene of the stereo image and a depth interval between said vehicle detection gates of the roadway scene of the stereo image, said each vehicle detection gate overlaps a portion of an adjacent detection gate;

identifying, using said road detection and awareness module, at least two lane boundary markers in said stereo image of said roadway scene;

retrieving, using said road detection and awareness module, ground plane data, said lane boundary markers, and said depth interval/range between said vehicle detection gates;

computing, using said road detection and awareness module, depth for each said vehicle detection gate using the depth interval and computing said region of interest (ROI) for each said vehicle detection gate using the ground plane data and the lane marker positions, wherein said ROI is a projection in the image of the roadway scene of height and depth equal to the depth of the vehicle detection gate lying between two lane boundary makers and the ground;

detecting, using a vehicular detection and tracking module at said image processing unit, potential lead vehicle detections among said vehicle detection gates on said roadway scene in said headway direction of said host vehicle, wherein said detecting comprising searching for blobs for the potential lead vehicle in said depth range within said ROI;

receiving a plurality of left images and a plurality of right images of the stereo image for each said gate ROI;

sub-sampling said plurality of the left images and the right images along a horizontal axis;

filtering said sub-sampled left images and said sub-sampled right images;

selecting said bobs located inside said ROI;

determining a maximum width of the ROI and a minimum width of the ROI;

retrieving pairs of blobs from the selected blobs, wherein said retrieved pairs of blobs are at least two of the selected blobs having a vertical position to be same within a defined threshold and width of a bounding box of the selected blobs is between the maximum width and the minimum width of the ROI; and

computing bounding boxes for all said retrieved blob pairs to define detection ROIs.

9. The method of claim 8 further comprising the steps of:

computing disparity of each said detection ROI, sum of absolute difference (SAD) score of each said computed disparity of each said detection ROI and ratio of said SAD using said left sub-sampled image, said right sub-sampled image and said detection ROIs.

10. The method of claim 9 wherein said verifying further comprising the steps of:

determining whether the detection ROI satisfies predefined width and height constraints of the potential lead vehicle;

determining whether said computed disparity of said detection ROI that satisfies said predefined constraints fulfill the criteria of lying between maximum and minimum disparity values of the vehicle detection gate;

identifying a list of said detection ROIs that satisfy said criteria.

11. The method of claim 10 further comprising the steps of:

determining whether the SAD score of said identified detection ROI is below a pre-determined threshold;

determining whether the SAD score for the left half image of the identified detection ROI and the right half image of the identified detection ROI are same;

selecting said identified detection ROI as a potential lead vehicle if said SAD score of said identified detection ROI is below said pre-determined threshold and said the SAD score for left half image of the identified detection ROI and the right half image of the identified detection ROI are the same.