Full speed lane sensing with a surrounding view system

• US 9,516,277 B2
• Filed: 04/09/2013
• Issued: 12/06/2016
• Est. Priority Date: 05/02/2012
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: Zhang, Wende, Litkouhi, Bakhtiar Brian, Lybecker, Kent S.
• Primary Examiner(s): Bailey, Frederick
• Application Number:
  US13/859,628
• Publication Number:
  US 20130293717A1
• Time in Patent Office: 1,337 Days
• Field of Search: 348/148, 348/149, 382/104
• US Class Current: 1/1
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A system and method for providing lane sensing on a vehicle by detecting roadway lane-markers, where the system employs a surround view camera system providing a top-down view image around the vehicle. The method includes detecting left-side and right-side lane boundary lines in the top-down view image, and then determining whether the lane boundary lines in the image are aligned from one image frame to a next image frame and are aligned from image to image in the top-down view image. If the boundary lines are not aligned, then calibration of one or more of the cameras is performed, and if the lines are aligned, then a model fitting process is used to specifically identify the location of the boundary lines on the roadway.

Claims

1. A method for sensing a travel lane in a roadway that a vehicle is traveling along, said vehicle including a front-view camera mounted to a front of the vehicle, a left-side view camera mounted to a left side of the vehicle, a right-side view camera mounted to a right side of the vehicle and a rear-view camera mounted to a rear of the vehicle, said roadway including a left-side roadway boundary line at a left side of the lane and a right-side roadway boundary line at a right side of the lane, said method comprising:

providing image data from the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera as consecutive image frames of the front of the vehicle, the left side of the vehicle, the right side of the vehicle and the rear of the vehicle;

constructing a top-down view image of the vehicle for each of the consecutive image frames that combines all of the image data from the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera where the vehicle is shown at the center of the top-down view image;

detecting a representation of the left-side boundary line and the right-side boundary line in the top-down view image;

calibrating an orientation of each of the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera using the representation of the left-side roadway boundary line and the right-side roadway boundary line in the top-down view image;

performing a camera diagnosis process to provide temporal constraint analysis and spatial constraint analysis, where the temporal constraint analysis determines whether consecutive top-down view image frames are co-linear, are at the same angle and are at the same offset within a temporal constraint threshold and where the spatial constraint analysis determines that the representation of the lane lines in the top-down view image that extend across boundaries from one image generated by one camera to another image generated by another camera are straight and have the same angle within a spatial constraint threshold;

re-calibrating the orientation of each of the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera if the temporal constraint threshold and the spatial constraint threshold are not met; and

using a model fitting process to identify the boundary lines in the top-down view image if the temporal constraint threshold and the spatial constraint threshold are met.

2. The method according to claim 1 wherein the representation of the left-side and right-side boundary lines are a series of image points.

3. The method according to claim 2 wherein using a model fitting process includes using a linear model based on weights given to each of the separate image points.

4. The method according to claim 3 wherein the weights are assigned based on a plurality of factors including camera view points, vehicle dynamic errors, calibration errors, visibility and reliability in the image and a lane sensing application.

5. The method according to claim 2 wherein detecting a representation of the boundary lines includes converting the image points to vehicle coordinates.

6. The method according to claim 1 further comprising providing a lighting normalization process to the top-down view image prior to the top-down view image being used to detect the representation of the left-side and right-side boundary lines.

7. The method according to claim 1 wherein performing a camera diagnosis process includes determining whether the representation of the left-side and right-side boundary lines are solid long lines or short dashed lines.

8. The method according to claim 7 wherein performing the camera diagnosis includes using the front-view camera if the vehicle is traveling in a forward direction and using the rear-view camera if the vehicle is traveling in a rearward direction to determine whether the representation of the boundary lines are the long solid lines or the short dashed lines.

9. The method according to claim 1 further comprising using vehicle motion compensation to enhance the representation of the left-side roadway boundary line and the right-side roadway boundary line in the top-down view image prior to performing the camera diagnosis process.

10. A method for sensing a travel lane in a roadway that a vehicle is traveling along, said vehicle including a front-view camera mounted to a front of the vehicle, a left-side view camera mounted to a left side of the vehicle, a right-side view camera mounted to a right side of the vehicle and a rear-view camera mounted to a rear of the vehicle, said roadway including a left-side roadway boundary line at a left side of the lane and a right-side roadway boundary line at a right side of the lane, said method comprising:

providing image data from the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera as consecutive image frames of the front of the vehicle, the left side of the vehicle, the right side of the vehicle and the rear of the vehicle;

providing a combined image that combines each of the images from the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera that includes a representation of the left-side boundary line and the right-side boundary line;

providing camera calibration to separately calibrate an orientation of each of the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera using either the representation of the left-side road boundary line or the right-side road boundary line in the images from the camera;

performing a camera diagnosis process to provide temporal constraint analysis and spatial constraint analysis of the combined image, where the temporal constraint analysis determines whether consecutive combined image frames are co-linear, are at the same angle and are at the same offset within a temporal constraint threshold and where the spatial constraint analysis determines that the representation of the lane lines in the combined view image that extend across boundaries from one image generated by one camera to another image generated by another camera are straight and have the same angle within a spatial constraint threshold;

recalibrating the orientation of one or more of the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera if the temporal constraint threshold and the spatial constraint threshold are not met; and

model fitting the representation of the left-side boundary line and the right-side boundary line in the combined image to the left side and right side boundary lines if the temporal constraint threshold and the spatial constraint threshold are met.

11. The method according to claim 10 wherein the representation of the left-side and right-side boundary lines are a series of image points.

12. The method according to claim 10 wherein model fitting includes using a polynomial model based on weights given to each of the separate image points.

13. The method according to claim 12 wherein the weights are assigned based on a plurality of factors including camera view points, vehicle dynamic errors, calibration errors, visibility and reliability in the image and a lane sensing application.

14. The method according to claim 10 wherein performing a camera diagnosis process includes transforming the representation of the lines to top-down view coordinates.

15. The method according to claim 10 further comprising providing a lighting normalization process for the combined image.

16. The method according to claim 10 wherein performing a camera diagnosis process includes determining whether the representation of the left-side and right-side boundary lines are solid long lines or short dashed lines.

17. The method according to claim 16 wherein performing the camera diagnosis includes using the front-view camera if the vehicle is traveling in a forward direction and using the rear-view camera if the vehicle is traveling a rearward direction to determine whether the representation of the boundary lines are the long solid lines or the short dashed lines.

18. The method according to claim 10 further comprising using vehicle motion compensation to enhance the representation of the left-side roadway boundary line and the right-side roadway boundary line in the combined image prior to performing the camera diagnosis process.

19. A system for sensing a travel lane in a roadway that a vehicle is traveling along, said vehicle including a front-view camera mounted to a front of the vehicle, a left-side view camera mounted to a left side of the vehicle, a right-side view camera mounted to a right side of the vehicle and a rear-view camera mounted to a rear of the vehicle, said roadway including a left-side roadway boundary line at a left side of the lane and a right-side roadway boundary line at a right side of the lane, said system comprising:

means for providing image data from the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera as consecutive image frames of the front of the vehicle, the left side of the vehicle, the right side of the vehicle and the rear of the vehicle;

means for constructing a top-down view image of the vehicle for each of the consecutive image frames that combines all of the image data from the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera where the vehicle is shown at the center of the top-down view image;

means for detecting a representation of the left-side boundary line and the right-side boundary line in the top-down view image;

means for calibrating an orientation of each of the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera using the representation of the left-side roadway boundary line and the right-side roadway boundary line in the top-down view image;

means for performing a camera diagnosis process to provide temporal constraint analysis and spatial constraint analysis, where the temporal constraint analysis determines whether consecutive top-down view image frames are co-linear, are at the same angle and are at the same offset within a temporal constraint threshold and where the spatial constraint analysis determines that the representation of the lane lines in the top-down view image that extend across boundaries from one image generated by one camera to another image generated by another camera are straight and have the same angle within a spatial constraint threshold;

means for re-calibrating the orientation of each of the front-view camera, the left-side view camera, the right-side view camera and the rear-view camera if the temporal constraint threshold and the spatial constraint threshold are not met; and

means for using a model fitting process to identify the boundary lines in the top-down view image if the temporal constraint threshold and the spatial constraint threshold are met.