FULL SPEED LANE SENSING WITH A SURROUNDING VIEW SYSTEM
First Claim
1. A method for sensing a travel lane in a roadway that a vehicle is traveling along, said vehicle including a front-view camera mounted to a front of the vehicle, a left-side view camera mounted to a left side of the vehicle, a right-side view camera mounted to a right side of the vehicle and a rear-view camera mounted to a rear of the vehicle, said roadway including a left-side roadway boundary line at a left side of the lane and a right-side roadway boundary line at a right side of the lane, said method comprising:
- providing image data from the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera as consecutive image frames of the front of the vehicle, the left side of the vehicle, the right side of the vehicle and the rear of the vehicle;
constructing a top-down view image of the vehicle for each of the consecutive image frames that combines all of the image data from the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera where the vehicle is shown at the center of the top-down view image;
detecting a representation of the left-side boundary line and the right-side boundary line in the top-down view image;
calibrating an orientation of each of the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera using the representation of the left-side roadway boundary line and the right-side roadway boundary line in the top-down view image;
performing a camera diagnosis process to provide temporal constraint analysis and spatial constraint analysis, where the temporal constraint analysis determines whether consecutive top-down view image frames are co-linear, are at the same angle and are at the same offset within a temporal constraint threshold and where the spatial constraint analysis determines that the representation of the lane lines in the top-down view image that extend across boundaries from one image generated by one camera to another image generated by another camera are straight and have the same angle within a spatial constraint threshold;
re-calibrating the orientation of each of the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera if the temporal constraint threshold and the spatial constraint threshold are not met; and
using a model fitting process to identify the boundary lines in the top-down view image if the temporal constraint threshold and the spatial constraint threshold are met.
3 Assignments
0 Petitions
Accused Products
Abstract
A system and method for providing lane sensing on a vehicle by detecting roadway lane-markers, where the system employs a surround view camera system providing a top-down view image around the vehicle. The method includes detecting left-side and right-side lane boundary lines in the top-down view image, and then determining whether the lane boundary lines in the image are aligned from one image frame to a next image frame and are aligned from image to image in the top-down view image. If the boundary lines are not aligned, then calibration of one or more of the cameras is performed, and if the lines are aligned, then a model fitting process is used to specifically identify the location of the boundary lines on the roadway.
-
Citations
19 Claims
-
1. A method for sensing a travel lane in a roadway that a vehicle is traveling along, said vehicle including a front-view camera mounted to a front of the vehicle, a left-side view camera mounted to a left side of the vehicle, a right-side view camera mounted to a right side of the vehicle and a rear-view camera mounted to a rear of the vehicle, said roadway including a left-side roadway boundary line at a left side of the lane and a right-side roadway boundary line at a right side of the lane, said method comprising:
-
providing image data from the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera as consecutive image frames of the front of the vehicle, the left side of the vehicle, the right side of the vehicle and the rear of the vehicle; constructing a top-down view image of the vehicle for each of the consecutive image frames that combines all of the image data from the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera where the vehicle is shown at the center of the top-down view image; detecting a representation of the left-side boundary line and the right-side boundary line in the top-down view image; calibrating an orientation of each of the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera using the representation of the left-side roadway boundary line and the right-side roadway boundary line in the top-down view image; performing a camera diagnosis process to provide temporal constraint analysis and spatial constraint analysis, where the temporal constraint analysis determines whether consecutive top-down view image frames are co-linear, are at the same angle and are at the same offset within a temporal constraint threshold and where the spatial constraint analysis determines that the representation of the lane lines in the top-down view image that extend across boundaries from one image generated by one camera to another image generated by another camera are straight and have the same angle within a spatial constraint threshold; re-calibrating the orientation of each of the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera if the temporal constraint threshold and the spatial constraint threshold are not met; and using a model fitting process to identify the boundary lines in the top-down view image if the temporal constraint threshold and the spatial constraint threshold are met. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9)
-
-
10. A method for sensing a travel lane in a roadway that a vehicle is traveling along, said vehicle including a front-view camera mounted to a front of the vehicle, a left-side view camera mounted to a left side of the vehicle, a right-side view camera mounted to a right side of the vehicle and a rear-view camera mounted to a rear of the vehicle, said roadway including a left-side roadway boundary line at a left side of the lane and a right-side roadway boundary line at a right side of the lane, said method comprising:
-
providing image data from the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera as consecutive image frames of the front of the vehicle, the left side of the vehicle, the right side of the vehicle and the rear of the vehicle; providing a combined image that combines each of the images from the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera that includes a representation of the left-side boundary line and the right-side boundary line; providing camera calibration to separately calibrate an orientation of each of the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera using either the representation of the left-side road boundary line or the right-side road boundary line in the images from the camera; performing a camera diagnosis process to provide temporal constraint analysis and spatial constraint analysis of the combined image, where the temporal constraint analysis determines whether consecutive combined image frames are co-linear, are at the same angle and are at the same offset within a temporal constraint threshold and where the spatial constraint analysis determines that the representation of the lane lines in the combined view image that extend across boundaries from one image generated by one camera to another image generated by another camera are straight and have the same angle within a spatial constraint threshold; recalibrating the orientation of one or more of the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera if the temporal constraint threshold and the spatial constraint threshold are not met; and model fitting the representation of the left-side boundary line and the right-side boundary line in the combined image to the left side and right side boundary lines if the temporal constraint threshold and the spatial constraint threshold are met. - View Dependent Claims (11, 12, 13, 14, 15, 16, 17, 18)
-
-
19. A system for sensing a travel lane in a roadway that a vehicle is traveling along, said vehicle including a front-view camera mounted to a front of the vehicle, a left-side view camera mounted to a left side of the vehicle, a right-side view camera mounted to a right side of the vehicle and a rear-view camera mounted to a rear of the vehicle, said roadway including a left-side roadway boundary line at a left side of the lane and a right-side roadway boundary line at a right side of the lane, said system comprising:
-
means for providing image data from the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera as consecutive image frames of the front of the vehicle, the left side of the vehicle, the right side of the vehicle and the rear of the vehicle; means for constructing a top-down view image of the vehicle for each of the consecutive image frames that combines all of the image data from the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera where the vehicle is shown at the center of the top-down view image; means for detecting a representation of the left-side boundary line and the right-side boundary line in the top-down view image; means for calibrating an orientation of each of the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera using the representation of the left-side roadway boundary line and the right-side roadway boundary line in the top-down view image; means for performing a camera diagnosis process to provide temporal constraint analysis and spatial constraint analysis, where the temporal constraint analysis determines whether consecutive top-down view image frames are co-linear, are at the same angle and are at the same offset within a temporal constraint threshold and where the spatial constraint analysis determines that the representation of the lane lines in the top-down view image that extend across boundaries from one image generated by one camera to another image generated by another camera are straight and have the same angle within a spatial constraint threshold; means for re-calibrating the orientation of each of the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera if the temporal constraint threshold and the spatial constraint threshold are not met; and means for using a model fitting process to identify the boundary lines in the top-down view image if the temporal constraint threshold and the spatial constraint threshold are met.
-
Specification