System and method for warning drivers based on road curvature

• US 7,184,073 B2
• Filed: 04/11/2003
• Issued: 02/27/2007
• Est. Priority Date: 04/11/2003
• Status: Expired due to Fees

• Current Assignee: Tech Mahindra Limited
• Original Assignee: Satyam Computer Services Limited of Mayfair Centre
• Inventors: Varadarajan, Sridhar, Sudhindra, Veena, Rao, Korrapati Kalyana
• Primary Examiner(s): LEE, RICHARD J
• Application Number:
  US10/411,088
• Publication Number:
  US 20040201672A1
• Time in Patent Office: 1,418 Days
• Field of Search: 348/143, 348/148, 348/159, 340/425.5, 340/435, 340/439, 340/517, 340/521, 340/576, 340/903, 382/104, 701 23- 28, 362/466
• US Class Current: 348/148
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

A system for warning a driver based on the road curvature consisting of multiple integrated cameras to acquire the images of different scales simultaneously, a camera controller to configure the cameras based on the vehicle speed and visibility criteria, a processor to analyze the multi-scale images to construct the mosaic, and warning generation subsystem to warn the driver about the safety speed based on the comparison of the mosaic with the curvature templates.

Claims

1. A road curvature warning system based on the analysis of multiple multi-scale images for efficiently identifying road curvature while driving at high speed on a highway, said road curvature warning system comprising:

(a) an RCWS controller subsystem for controlling cameras and interfacing with a vehicle, said RCWS controller subsystem comprising;

a system configuration element for dynamically managing system configuration related to camera, system thresholds, and standard road dimensions;

a camera controller element for dynamically controlling the cameras;

a vehicle interface element for obtaining vehicle related parameters comprising ground speed and slope; and

a multi-camera element for acquiring multiple multi-scale images; and

(b) an RCWS preprocessor subsystem for preprocessing an image, said RCWS preprocessor subsystem comprising;

an error correction and binarization element for image preprocessing, correcting said image for heading and offset errors, and binarization of said image;

a road parameter computation element for computing the lane width and lane marker width;

a look-ahead distance computation element for determining an appropriate look-ahead distance for configuring the cameras; and

a visibility checking element for determining the road visibility conditions; and

(c) an RCWS processor subsystem for generating curvature related over-speed warnings based on a plurality of multi-scale images, said RCWS processor subsystem comprising;

a lane identification element for identifying left and right road lane markers in said plurality of multi-scale images;

a multi-scale image processing element for analyzing said plurality of multi-scale images to correlate and correct a plurality left and right lane markers across said plurality of multi-scale images;

a mosaic construction element for constructing a road mosaic using said plurality of left and right lane markers;

a road curvature identification element for identifying a road curvature from said road mosaic; and

a warning generation element for generating an appropriate warning to the driver based on said road curvature.

2. The system of claim 1, wherein said camera controller element of said RCWS controller subsystem comprises means for dynamically configuring and controlling of cameras based on a look-ahead distance, wherein said look-ahead is computed based on vehicle speed and road visibility.

3. The system of claim 1, wherein said vehicle interface element of said RCWS controller subsystem comprises means for interfacing with vehicle control system to obtain vehicle related parameters consisting of vehicle speed and road vertical gradient, wherein said obtaining of said speed and said gradient are performed periodically.

4. The system of claim 1, wherein said multi-camera element of said RCWS controller subsystem comprises means for capturing the multiple multi-scale images of a road from multiple cameras, wherein said cameras are with the same focal point and cover different lengths of said road.

5. The system of claim 1, wherein said error correction and binarization element of said RCWS preprocessor subsystem comprises means for image preprocessing of a road image, said image preprocessing comprises:

performing inverse perspective mapping operation on said road image to determine a perspective free road image;

performing error correction operation by identifying and correcting heading and offset errors in said perspective free road image to determine a corrected road image;

performing resealing operation on said corrected road image to determine a resealed road image; and

performing binarization operation on said resealed road image based on edge detection operation.

6. The system of claim 1, wherein said road parameter computation element of said RCWS preprocessor subsystem comprises means for computing lane width and lane marker width periodically, wherein said means for computing comprises:

derivation of a count matrix from a binarized image of a road image, wherein rows and columns of said count matrix contains counts of consecutive 0'"'"'s and 1'"'"'s in rows and columns of said binarized image;

determination of a plurality of clusters of each column of said count matrix;

selection of a plurality of clusters from said plurality of clusters based on standard lane widths and lane marker widths; and

adjusting said binarized image and said road image based on said selected plurality of clusters.

7. The system of claim 6, wherein said road parameter computation element further comprises means for characterizing lane and lane marker segments of a road image in terms of pixel values, wherein said characterization of lane segment comprises:

determining pixels of said lane segment of said road image;

determining plurality of clusters of said pixels;

selecting a cluster of maximum density from said plurality of clusters; and

computing centroid of said selected cluster, and

said characterization of lane marker segment comprises;

determining pixels of said lane marker segment of said road image;

determining plurality of clusters of said pixels;

selecting a cluster of maximum density from said plurality of clusters; and

computing centroid of said selected cluster.

8. The system of claim 1, wherein said look-ahead distance computation element of said RCWS preprocessor subsystem comprises means for computing look ahead distance and reconfiguring a plurality of cameras, wherein said means for computing and reconfiguring comprises:

obtaining a vehicle speed;

obtaining a look ahead distance from a lookup table based on said vehicle speed;

checking for road visibility conditions at said look ahead distance; and

adjusting parameters of said plurality of cameras based on said vehicle speed and said visibility conditions.

9. The system of claim 1, wherein said visibility checking element of said RCWS preprocessor subsystem comprises means for checking visibility of lane markers in a road image, wherein said checking comprises:

obtaining a characterization of a plurality of lane markers in said road image;

obtaining a characterization of a lane of said road image based on said plurality of lane markers;

categorizing pixels of said road image based on said characterization of lane markers and said characterization of lane;

computing percentage of lane marker information and non-road information in said road image; and

ensuring sustained goodness or badness of visibility conditions.

10. The system of claim 1, wherein said lane identification element of said RCWS processor subsystem comprises means for identifying and labeling of a plurality of bounded lane chains in a binarized image, wherein said identification of each of said plurality of bounded lane chains is based on horizontal and vertical neighborhood connectivity within tolerance, and said labeling is based on position of each of said bounded chains with respect to the image center of said binarized image.

11. The system of claim 10, wherein said lane identification element further comprises means for determining a plurality of merged paired lane chains from a plurality of bounded left lane chains and right lane chains, wherein said means for determining a plurality of merged paired lane chains comprises:

identifying a subset of said plurality of bounded left lane chains;

identifying a subset of said plurality of right lane chains such that said subset of bounded left lane chains and said subset of bounded right lane chains satisfy a plurality of constraints consisting of lane width constraint and closeness of anchor points constraint;

merging a plurality of left lane chains of said subset of left lane chains, wherein said merging is based on identification of one bounding rectangle the bounded rectangles of said subset of left lane chains; and

merging a plurality of right lane chains of said subset of right lane chains, wherein said merging a plurality of right lane chains is based on identification of one bounding rectangle covering the bounded rectangle of each of said subset of right lane chains.

12. The system of claim 10, wherein said lane identification element further comprises means for identifying and merging a plurality of bounded lane chains, wherein said identification and merging comprises:

identifying a subset of said plurality of bounded lane chains such that a plurality of bounded lane chains in said subset satisfy a plurality of constraints consisting of;

similarity in rate of change and orientation constraint;

lane marker gap constraint; and

contiguity constraint;

selecting a plurality of bounded lane chains from said subset such that each of said selected plurality of bounded lane chains correlate with a bounded lane chain in said selected plurality of bounded lane chains; and

merging a plurality of bounded lane chains of said selected plurality of lane chains, wherein said merging is based on identification of one bounding rectangle covering the bounded rectangle of each of said selected plurality of lane chains.

13. The system of claim 10, wherein said lane identification element further comprises means for forming a left lane marker and a right lane marker of a road image, wherein said left lane marker formation from a plurality of thinned left lane chains comprises:

identifying a left minimal subset of said plurality of thinned left lane chains optimally with least upper gap with respect to the upper edge of said road image, least lower gap with respect to the lower edge of said road image, and exhibiting similarity in rate of change and orientation, and

said right lane marker formation from a plurality of thinned right lane chains comprises;

identifying a right minimal subset of said plurality of thinned right lane chains optimally with least upper gap with respect to the upper edge of said road image, least lower gap with respect to the lower edge of said road image, and exhibiting similarity in rate of change and orientation.

14. The system of claim 13, wherein said means for forming a left lane marker and a right lane marker further comprises:

selecting a left maximal subset of said left minimal subset;

selecting a right maximal subset of said right minimal subset;

ensuring that a plurality of lane chains in said left maximal subset and a plurality of lane chains in said right maximal subset optimally satisfy lane width constraint;

merging lane chains in said left maximal subset without violating rate of change and orientation constraint; and

merging lane chains in said right maximal subset without violating rate of change and orientation constraint.

15. The system of claim 1, wherein said multi-scale image processing element of said RCWS processor subsystem comprises means for multi-scale analysis for correcting a fine scale image of a road, wherein said multi-scale analysis comprises:

analyzing a coarse scale image of said road to determine a coarse left lane marker and a coarse right lane marker;

analyzing a medium scale image of said road to determine a medium left lane marker and a medium right lane marker;

analyzing a smooth scale image of said road to determine a smooth left lane marker and a smooth right lane marker;

analyzing said fine scale image of said road to determine a fine left lane marker and a fine right lane marker;

correlating said medium left lane marker of said medium scale image with said coarse left lane marker of said coarse scale image to determine a corrected medium left lane marker;

correlating said medium right lane marker of said medium scale image with said coarse right lane marker of said coarse scale image to determine a corrected medium right lane marker;

correlating said smooth left lane marker of said smooth scale image with said corrected medium left lane marker to determine a corrected smooth left lane marker;

correlating said smooth right lane marker of said smooth scale image with said corrected medium right lane marker to determine a corrected smooth right lane marker;

correlating said fine left lane marker of said fine scale image with said corrected smooth left lane marker to determine a corrected fine left lane marker; and

correlating said fine right lane marker of said fine scale image with said corrected smooth right lane marker to determine a corrected fine right lane marker.

16. The system of claim 15, wherein said multi-scale analysis further comprises means for correlating a medium scale image with a low scale image and correcting said medium scale image, wherein said correlation and correction comprises:

extracting a portion around image center of said low scale image;

scaling said portion to match the size of said medium scale image;

determining a low left bounded rectangle around a left lane marker of said portion;

determining a low right bounded rectangle around a right lane marker of said portion;

determining a medium left bounded rectangle around a left lane marker of said medium scale image;

determining a medium right bounded rectangle around a right lane marker of said medium scale image;

computing a left similarity measure between said low left bounded rectangle and said medium left bounded rectangle to determine a left set of points of dissimilarity;

computing a right similarity measure between said low right bounded rectangle and said medium right bounded rectangle to determine a right set of points of dissimilarity;

applying an adjustment on said medium scale image based on said left set of points of dissimilarity and said right set of points of dissimilarity satisfying a plurality of constraints consisting of;

rate of change and orientation constraint; and

lane width constraint; and

computing the total of said adjustment on said medium scale image.

17. The system of claim 1, wherein said mosaic construction element of said RCWS processor subsystem comprises means for extending a mosaic with a left lane marker and a right lane marker, wherein said extension comprises:

appending said left lane marker and said right lane marker to said mosaic;

analyzing a plurality of consecutive left lane segments consisting of top, middle and bottom segments to determine a left segment error in said appending of said left lane marker to said mosaic;

analyzing a plurality of consecutive right lane segments consisting of top, middle and bottom segments to determine a right segment error in said appending of said right lane marker to said mosaic;

adjusting the end points of said middle segment of said plurality of consecutive left lane segments of said mosaic and the end points of said middle segment of said plurality of consecutive right lane segments of said mosaic to minimize said left segment error and said right segment error; and

computing the cumulative residual error of said mosaic.

18. The system of claim 1, wherein said road curvature identification element of said RCWS processor subsystem comprises means for template matching a mosaic with a plurality of curvature type templates, wherein said template matching is based on component matching of a plurality of curvature components of each of said plurality of curvature type templates and selecting a curvature type based on unambiguous matching of said plurality of curvature components of said curvature type, wherein said component matching is based on successful satisfaction of a set of rules associated with each of said plurality of curvature components based on said mosaic.

19. The system of claim 1, wherein said warning generation element of said RCWS processor subsystem comprises means for generating a warning to the driver driving a vehicle on a road based on comparing computed speed with actual speed of said vehicle, wherein said computation of speed is based on a curvature type of curvature of said road and a set of rules associated with said curvature type.

20. An apparatus for analysis of multiple multi-scale images for efficiently identifying road curvature while driving a vehicle at high speed on a highway, said apparatus comprising:

(a) a hardware and software RCWS controller subsystem for controlling a plurality of cameras and interfacing with said vehicle comprising;

a processor of appropriate capacity;

memory devices of appropriate capacity;

an interface subsystem for interacting with ground speed sensor and slope detector of said vehicle, and

a system configuration repository;

software subsystem for configuring and controlling said plurality of cameras, and for acquiring multiple multi-scale images from said plurality of cameras; and

(b) a hardware and software RCWS preprocessor subsystem for preprocessing an image of a road comprising;

a processor of appropriate capacity;

memory devices of appropriate capacity;

a software subsystem for image preprocessing, heading error and offset error correction, and binarization of said image of said road;

a software subsystem for computing a lane width and a lane marker width in said image;

a software subsystem for determining an appropriate look-ahead distance for configuring said plurality of cameras; and

a software subsystem for determining visibility conditions of said road; and

(c) a hardware and software RCWS processor subsystem to analyze a plurality of multi-scale images of a road comprising;

a processor of appropriate capacity;

memory devices of appropriate capacity;

a repository of lane templates;

a software subsystem for identifying a left lane marker and a right lane marker in each of said plurality of multi-scale images;

a software subsystem for analyzing said plurality of multi-scale images to correlate said left lane marker and said right lane marker in each of said plurality of multi-scale images across said plurality of multi-scale images, and to determine a road left lane marker and a road right lane marker;

a software subsystem for constructing a road mosaic using said road left lane marker and said road right lane marker;

a software subsystem for identifying a road curvature from said road mosaic; and

a software subsystem for generating an appropriate warning to the driver of said vehicle based on said road curvature.