FULL SPEED LANE SENSING USING MULTIPLE CAMERAS

• US 20130293714A1
• Filed: 04/09/2013
• Published: 11/07/2013
• Est. Priority Date: 05/02/2012
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Operations LLC
• Inventors: Zhang, Wende, Litkouhi, Bakhtiar Brian, Lybecker, Kent S.
• Granted Patent: US 9,538,144 B2
• Time in Patent Office: Days
• US Class Current: 348/148
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A system and method for providing lane sensing on a vehicle by detecting roadway lane-markers, where the system employs multiple cameras providing images around the vehicle. The method includes detecting left-side and right-side lane boundary lines in the camera images, and then determining whether the lane boundary lines in the images are aligned from one image frame to a next image frame and are aligned from image to image. If the boundary lines are not aligned, then calibration of one or more of the cameras is performed, and if the lines are aligned, then a model fitting process is used to specifically identify the location of the boundary lines on the roadway.

Claims

1. A method for sensing a travel lane in a roadway that a vehicle is traveling along, said vehicle including a plurality of cameras, said roadway including a left-side roadway boundary line at a left side of the lane and a right-side roadway boundary line at a right side of the lane, said method comprising:

providing images for each of the plurality of cameras from consecutive image frames;

detecting a representation of one or both of the left-side boundary line and the right-side boundary line in each of the images;

calibrating an orientation of each of the plurality of cameras using the representation of the left-side roadway boundary line and the right side roadway boundary line in the image;

performing a camera diagnosis process for each of the plurality of cameras to provide temporal constraint analysis and spatial constraint analysis, where the temporal constraint analysis determines whether consecutive image frames are co-linear, are at the same angle and are at the same offset within a temporal constraint threshold and where the spatial constraint analysis determines that the representation of the boundary lines that extend across boundaries for one image from one camera to another image from another camera are straight and have the same angle within a spatial constraint threshold;

re-calibrating the orientation of each of the plurality of cameras if the temporal constraint threshold and the spatial constraint threshold are not met; and

using a model fitting process to identify the boundary lines in each image if the temporal constraint threshold and the spatial constraint threshold are met.

2. The method according to claim 1 wherein the representation of the left-side and right-side boundary lines are a series of image points.

3. The method according to claim 2 wherein using a model fitting process includes using a linear model based on weights given to each of the separate image points.

4. The method according to claim 3 wherein the weights are assigned based on a plurality of factors including camera view points, vehicle dynamic errors, calibration errors, visibility and reliability in the image and a lane sensing application.

5. The method according to claim 2 wherein detecting a representation of the boundary lines includes converting the image points to vehicle coordinates.

6. The method according to claim 1 further comprising providing a lighting normalization process for each image from each camera prior to the image being used to detect the representation of the left-side and right-side boundary lines.

7. The method according to claim 1 wherein performing a camera diagnosis process includes determining whether the representation of the left-side and right-side boundary lines are long solid lines or short dashed lines.

8. The method according to claim 7 wherein the plurality of cameras include a front-view camera and a rear-view camera, and wherein performing the camera diagnosis includes using the front-view camera if the vehicle is traveling in a forward direction and using the rear-view camera if the vehicle is traveling in a rearward direction to determine whether the representation of the boundary lines are the long solid lines or the short dashed lines.

9. The method according to claim 1 further comprising using vehicle motion compensation to enhance the representation of the left-side roadway boundary line and the right-side roadway boundary line in each image from the plurality of cameras prior to performing the camera diagnosis process.

10. The method according to claim 1 wherein performing a camera diagnosis process includes transforming the representation of the lines to top-down view coordinates.

11. A method for sensing a travel lane in a roadway that a vehicle is traveling along, said vehicle including a front-view camera mounted to a front of the vehicle, a left-side view camera mounted to a left side of the vehicle, a right-side view camera mounted to a right side of the vehicle and a rear-view camera mounted to a rear of the vehicle, said roadway including a left-side roadway boundary line at a left side of the lane and a right-side roadway boundary line at a right side of the lane, said method comprising:

providing separate images from the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera from consecutive image frames of the front of the vehicle, the left side of the vehicle, the right side of the vehicle and the rear of the vehicle;

detecting a representation of the left-side boundary line and the right-side boundary line for each separate image from at least the front-view camera and the rear-view camera;

calibrating an orientation of each of the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera using either the representation of the left-side roadway boundary line or the right-side roadway boundary line in the images from the cameras;

performing a camera diagnosis process for each of the cameras to provide temporal constraint analysis and spatial constraint analysis of each image, where the temporal constraint analysis determines whether consecutive image frames are co-linear, are at the same angle and are at the same offset within a temporal constraint threshold and where the spatial constraint analysis determines that the representation of the lane lines in the images that extend across boundaries for one image from one camera to another image from another camera are straight and have the same angle within a spatial constraint threshold;

recalibrating the orientation of one or more of the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera if the temporal constraint threshold and the spatial constraint threshold are not met; and

model fitting the representation of the left-side boundary line and the right-side boundary line in each image to the left-side and right-side boundary lines if the temporal constraint threshold and the spatial constraint threshold are met.

12. The method according to claim 11 wherein the representation of the left-side and right-side boundary lines are a series of image points.

13. The method according to claim 11 wherein model fitting includes using a polynomial model based on weights given to each of the separate image points.

14. The method according to claim 13 wherein the weights are assigned based on a plurality of factors including camera view points, vehicle dynamic errors, calibration errors, visibility and reliability in the image and a lane sensing application.

15. The method according to claim 11 wherein performing a camera diagnosis process includes transforming the representation of the lines to top-down view coordinates.

16. The method according to claim 11 further comprising providing a lighting normalization process for each separate image from the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera prior to each separate image being used to detect the representation of the left and right lane boundary lines.

17. The method according to claim 11 wherein performing a camera diagnosis process includes determining whether the representation of the left and right boundary lines are solid long lines or short dashed lines.

18. The method according to claim 17 wherein performing the camera diagnosis includes using the front-view camera if the vehicle is traveling in a forward direction and using the rear-view camera if the vehicle is traveling a rearward direction to determine whether the representation of the boundary lines are the long solid lines or the short dashed lines.

19. The method according to claim 11 further comprising using vehicle motion compensation to enhance the representation of the left-side roadway boundary line and the right-side roadway boundary line in the combined image prior to performing the camera diagnosis process.

20. A method for sensing a travel lane in a roadway that a vehicle is traveling along, said vehicle including at least one camera, said roadway including one or both of a left-side roadway boundary line at a left side of the lane and a right-side roadway boundary line at a right side of the lane, said method comprising:

providing image data from the at least one camera;

detecting a representation of the left-side boundary line or the right-side boundary line in the image data;

calibrating an orientation of the at least one camera using the representation of the left-side roadway boundary line or the right side roadway boundary line in the image data;

performing a camera diagnosis process for the at least one camera to provide temporal constraint analysis and spatial constraint analysis, where the temporal constraint analysis determines whether consecutive image frames are co-linear, are at the same angle and are at the same offset within a temporal constraint threshold and where the spatial constraint analysis determines that the representation of the lane lines that extend across boundaries are straight and have the same angle within a spatial constraint threshold;

re-calibrating the orientation of the at least one camera if the temporal constraint threshold and the spatial constraint threshold are not met; and

using a model fitting process to identify the boundary lines in the image data if the temporal constraint threshold and the spatial constraint threshold are met.