Vision system for vehicle

• US 9,643,605 B2
• Filed: 10/26/2015
• Issued: 05/09/2017
• Est. Priority Date: 05/03/2002
• Status: Expired due to Term

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Pawlicki, John A., McMahon, Martha A., Chinn, Steven G., Gibson, Joel S.
• Primary Examiner(s): PHAM, TOAN NGOC
• Application Number:
  US14/922,640
• Publication Number:
  US 20160046288A1
• Time in Patent Office: 561 Days
• Field of Search: 340435, 3404255, 340901, 340461, 340438, 340937, 340933, 340935, 340938, 340436, 359601, 359604, 359603, 359265, 348148, 348151
• CPC Class Codes: B60K 31/00 Vehicle fittings, acting on a single sub-unit only, for automatically controlling vehicle speed, i.e. preventing speed from exceeding an arbitrarily established velocity or maintaining speed at a particular velocity, as selected by the vehicle operator fittings acting on two or more sub-units B60W30/14; speedometers G01P; systems or devices for controlling speed in general G05D13/00 ; in traffic anti-collision system for road vehicles G08G1/16
• CPC Class Codes: B60K 31/0008 including means for detecting potential obstacles in vehicle path
• CPC Class Codes: B60Q 1/525 automatically indicating risk of collision between vehicles in traffic or with pedestrians, e.g. after risk assessment using the vehicle sensor data
• CPC Class Codes: B60Q 9/008 for anti-collision purposes
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: B60R 2011/004 outside the vehicle
• CPC Class Codes: B60T 2201/08 Lane monitoring; Lane Keeping Systems
• CPC Class Codes: B60T 2201/082 using alarm actuation
• CPC Class Codes: B60T 2201/089 using optical detection
• CPC Class Codes: B60T 2210/34 Blind spots
• CPC Class Codes: B60T 7/22 initiated by contact of vehicle, e.g. bumper, with an external object, e.g. another vehicle , or by means of contactless obstacle detectors mounted on the vehicle
• CPC Class Codes: B60W 10/04 including control of propulsion units
• CPC Class Codes: B60W 10/20 including control of steering systems
• CPC Class Codes: B60W 2050/143 Alarm means
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2554/4041 Position
• CPC Class Codes: B60W 2554/80 Spatial relation or speed relative to objects
• CPC Class Codes: B60W 2554/801 Lateral distance
• CPC Class Codes: B60W 2554/802 Longitudinal distance
• CPC Class Codes: B60W 2555/20 : Ambient conditions, e.g. wind or rain
• CPC Class Codes: B60W 2556/50 : of positioning data, e.g. GPS [Global Positioning System] data
• CPC Class Codes: B60W 2710/20 : Steering systems
• CPC Class Codes: B60W 2720/10 : Longitudinal speed
• CPC Class Codes: B60W 30/09 : Taking automatic action to avoid collision, e.g. braking and steering
• CPC Class Codes: B60W 30/12 : Lane keeping
• CPC Class Codes: B60W 30/143 : Speed control B60W30/16 takes precedence
• CPC Class Codes: B60W 30/16 : Control of distance between vehicles, e.g. keeping a distance to preceding vehicle
• CPC Class Codes: B60W 30/18 : Propelling the vehicle
• CPC Class Codes: B60W 50/14 : Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: B62D 15/025 : Active steering aids, e.g. helping the driver by actively influencing the steering system after environment evaluation B62D1/28 takes precedence
• CPC Class Codes: G01S 11/12 : using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 13/867 : Combination of radar systems with cameras
• CPC Class Codes: G01S 13/931 : of land vehicles
• CPC Class Codes: G06T 2207/30244 : Camera pose
• CPC Class Codes: G06T 2207/30256 : Lane; Road marking
• CPC Class Codes: G06T 2207/30261 : Obstacle
• CPC Class Codes: G06T 7/13 : Edge detection
• CPC Class Codes: G06T 7/174 : involving the use of two or more images
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06T 7/73 : using feature-based methods
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 : Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/582 : of traffic signs
• CPC Class Codes: G06V 20/584 : of vehicle lights or traffic lights
• CPC Class Codes: G06V 20/588 : Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G06V 2201/10 : Recognition assisted with metadata
• CPC Class Codes: G08G 1/16 : Anti-collision systems road vehicle drive control systems for predicting or avoiding probable or impending collision otherwise than by control of a particular sub-unit B60W30/08
• CPC Class Codes: G08G 1/167 : Driving aids for lane monitoring, lane changing, e.g. blind spot detection
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A forward-facing vision system for a vehicle includes a forward-facing camera disposed in a windshield electronics module attached at a windshield of the vehicle and viewing through the windshield. A control includes a processor that, responsive to processing of captured image data, detects taillights of leading vehicles during nighttime conditions and, responsive to processing of captured image data, detects lane markers on a road being traveled by the vehicle. The control, responsive to lane marker detection and a determination that the vehicle is drifting out of a traffic lane, may control a steering system of the vehicle to mitigate such drifting, with the steering system manually controllable by a driver of the vehicle irrespective of control by the control. The processor, based at least in part on detection of lane markers via processing of captured image data, determines curvature of the road being traveled by the vehicle.

Claims

1. A forward-facing vision system for a vehicle, said forward-facing vision system comprising:

a forward-facing camera comprising an imaging sensor and a lens;

said imaging sensor comprising a plurality of photo-sensing pixels arranged in a matrix of multiple horizontal rows of photo-sensing pixels and multiple vertical columns of photo-sensing pixels;

wherein said forward-facing camera is disposed in a windshield electronics module attached at a windshield of a vehicle equipped with said forward-facing vision system, said forward-facing camera, when disposed in said windshield electronics module attached at the windshield, viewing through the windshield exteriorly of the equipped vehicle;

wherein said imaging sensor is operable to capture image data representative of a scene exterior of the equipped vehicle and in the field of view of said forward-facing camera;

a control comprising a processor;

wherein said processor, responsive to processing of captured image data, detects taillights of leading vehicles within the exterior field of view of said forward-facing camera when the equipped vehicle is operated under nighttime conditions;

wherein said processor, responsive to processing of captured image data, detects lane markers within the exterior field of view of said forward-facing camera on a road being traveled by the equipped vehicle;

wherein said control, responsive to said lane markers detection and to a determination based at least in part on processing of captured image data by said processor that the equipped vehicle is drifting out of a traffic lane that the equipped vehicle is currently travelling in and when a turn signal of the equipped vehicle is not being activated, controls a steering system of the equipped vehicle to adjust the steering direction of the equipped vehicle to mitigate such drift out of the traffic lane the equipped vehicle is travelling in, and wherein said steering system is manually controllable by a driver of the equipped vehicle irrespective of control by said control;

wherein said processor processes captured image data via an edge detection algorithm to detect edges of vehicles present exteriorly of the equipped vehicle and within the exterior field of view of said forward-facing camera;

wherein said processor processes captured image data to estimate distance from the equipped vehicle to a detected vehicle that is present exteriorly of the equipped vehicle and within the exterior field of view of said forward-facing camera;

wherein said processor, based at least in part on detection of lane markers via processing of captured image data, determines curvature of the road being traveled by the equipped vehicle; and

wherein said processor processes captured image data for at least one driver assistance system of the equipped vehicle selected from the group consisting of (i) a traffic sign recognition system of the equipped vehicle, (ii) a stop light recognition system of the equipped vehicle, (iii) a stop sign recognition system of the equipped vehicle and (iv) a vehicle speed control system of the equipped vehicle.

2. The forward-facing vision system of claim 1, wherein, responsive to processing of captured image data by said processor, said forward-facing vision system approximates distance to a vehicle detected within the exterior field of view of said forward-facing camera.

3. The forward-facing vision system of claim 1, wherein said processor, responsive to processing of captured image data, detects headlights of oncoming vehicles within the exterior field of view of said forward-facing camera when the equipped vehicle is operated under nighttime conditions.

4. The forward-facing vision system of claim 1, wherein said control receives image data from a camera mounted at a driver-side exterior rearview mirror of the equipped vehicle and wherein said control receives image data from a camera mounted at a passenger-side exterior rearview mirror of the equipped vehicle.

5. The forward-facing vision system of claim 1, wherein said processor processes less than approximately 75 percent of captured image data.

6. The forward-facing vision system of claim 1, wherein said control, responsive to taillight detection via processing of captured image data, provides an output for a headlamp control system of the equipped vehicle that controls at least one headlamp of the equipped vehicle.

7. The forward-facing vision system of claim 6, wherein image data captured by said forward-facing camera is processed by said processor one way for said headlamp control system and is processed by said processor another way for said at least one driver assistance system of the equipped vehicle.

8. The forward-facing vision system of claim 1, wherein said at least one driver assistance system of the equipped vehicle comprises a vehicle speed control system of the equipped vehicle, and wherein, responsive at least in part to processing of captured image data by said processor, speed of the equipped vehicle is adjusted in accordance with at least one of (i) a road condition detected by said forward-facing vision system and (ii) a traffic condition detected by said forward-facing vision system.

9. The forward-facing vision system of claim 8, wherein said vehicle speed control system, responsive at least in part to processing of captured image data by said processor, adjusts a cruise control setting of an adaptive cruise control system of the equipped vehicle.

10. The forward-facing vision system of claim 9, wherein, responsive at least in part to processing by said processor of captured image data detecting a curve in the road ahead of the equipped vehicle, speed of the equipped vehicle is reduced to an appropriate speed for traveling around the detected curve without a driver of the equipped vehicle manually deactivating cruise control of the equipped vehicle, and wherein the equipped vehicle resumes the speed set by the adaptive cruise control system after travelling through the detected curve to again travel along a generally straight section of road.

11. The forward-facing vision system of claim 1, wherein, when the detected leading vehicle is determined to be within a threshold distance from the equipped vehicle, speed of the equipped vehicle is reduced.

12. The forward-facing vision system of claim 1, wherein said forward-facing vision system is operable to provide image data to a remote device that is remote from the equipped vehicle, and wherein said forward-facing vision system provides said image data to indicate at least one of (i) a location of the equipped vehicle and (ii) a condition of the equipped vehicle.

13. The forward-facing vision system of claim 1, wherein said processor is operable to determine whether detected edges constitute a portion of a vehicle located exterior of and ahead of the equipped vehicle, and wherein said processor is operable to track said detected edges over multiple frames of image data captured by said imaging sensor.

14. The forward-facing vision system of claim 1, wherein said at least one driver assistance system of the equipped vehicle comprises a vehicle speed control system of the equipped vehicle, and wherein said vehicle speed control system comprises an adaptive cruise control system, and wherein said adaptive cruise control system comprises said forward-facing camera and a radar.

15. The forward-facing vision system of claim 14, wherein, during operation of said adaptive cruise control system, said forward-facing camera is primary to said radar.

16. The forward-facing vision system of claim 14, wherein, during operation of said adaptive cruise control system, said forward-facing camera is secondary to said radar.

17. A forward-facing vision system for a vehicle, said forward-facing vision system comprising:

a forward-facing camera comprising an imaging sensor and a lens;

said imaging sensor comprising a plurality of photo-sensing pixels arranged in a matrix of multiple horizontal rows of photo-sensing pixels and multiple vertical columns of photo-sensing pixels;

wherein said forward-facing camera is disposed in a windshield electronics module attached at a windshield of a vehicle equipped with said forward-facing vision system, said forward-facing camera, when disposed in said windshield electronics module attached at the windshield, viewing through the windshield exteriorly of the equipped vehicle;

wherein said imaging sensor is operable to capture image data representative of a scene exterior of the equipped vehicle and in the field of view of said forward-facing camera;

a control comprising a processor;

wherein said processor, responsive to processing of captured image data, detects taillights of leading vehicles within the exterior field of view of said forward-facing camera when the equipped vehicle is operated under nighttime conditions;

wherein said processor, responsive to processing of captured image data, detects lane markers within the exterior field of view of said forward-facing camera on a road being traveled by the equipped vehicle;

wherein said processor processes captured image data via an edge detection algorithm to detect edges of vehicles present exteriorly of the equipped vehicle and within the exterior field of view of said forward-facing camera;

wherein said control, responsive to taillight detection via processing of captured image data, provides an output for a headlamp control system of the equipped vehicle that controls at least one headlamp of the equipped vehicle;

wherein said processor processes captured image data to estimate distance from the equipped vehicle to a detected vehicle that is present exteriorly of the equipped vehicle and within the exterior field of view of said forward-facing camera; and

wherein said processor, based at least in part on detection of lane markers via processing of captured image data, determines curvature of the road being traveled by the equipped vehicle.

18. The forward-facing vision system of claim 17, wherein, when the detected leading vehicle is determined to be within a threshold distance from the equipped vehicle, speed of the equipped vehicle is reduced.

19. The forward-facing vision system of claim 17, wherein, responsive at least in part to processing of captured image data by said processor, speed of the equipped vehicle is adjusted in accordance with at least one of (i) a road condition detected by said forward-facing vision system and (ii) a traffic condition detected by said forward-facing vision system.

20. The forward-facing vision system of claim 19, wherein, responsive at least in part to processing of captured image data by said processor, a cruise control setting of an adaptive cruise control system of the equipped vehicle is adjusted.

21. The forward-facing vision system of claim 20, wherein said adaptive cruise control system comprises said forward-facing camera and a radar and wherein said forward-facing camera is primary to said radar.

22. The forward-facing vision system of claim 20, wherein said forward-facing camera comprises a CMOS forward-facing camera, and wherein said control, responsive to said lane markers detection and to a determination based at least in part on processing of captured image data by said processor that the equipped vehicle is drifting out of a traffic lane that the equipped vehicle is currently travelling in and when a turn signal of the equipped vehicle is not being activated, controls a steering system of the equipped vehicle to adjust the steering direction of the equipped vehicle to mitigate such drift out of the traffic lane the equipped vehicle is travelling in, and wherein said steering system is manually controllable by a driver of the equipped vehicle irrespective of control by said control.

23. A forward-facing vision system for a vehicle, said forward-facing vision system comprising:

a forward-facing camera comprising an imaging sensor and a lens;

said imaging sensor comprising a plurality of photo-sensing pixels arranged in a matrix of multiple horizontal rows of photo-sensing pixels and multiple vertical columns of photo-sensing pixels;

wherein said forward-facing camera is disposed in a windshield electronics module attached at a windshield of a vehicle equipped with said forward-facing vision system, said forward-facing camera, when disposed in said windshield electronics module attached at the windshield, viewing through the windshield exteriorly of the equipped vehicle;

wherein said imaging sensor is operable to capture image data representative of a scene exterior of the equipped vehicle and in the field of view of said forward-facing camera;

a control comprising a processor;

wherein said processor, responsive to processing of captured image data, detects taillights of leading vehicles within the exterior field of view of said forward-facing camera when the equipped vehicle is operated under nighttime conditions;

wherein said processor, responsive to processing of captured image data, detects lane markers within the exterior field of view of said forward-facing camera on a road being traveled by the equipped vehicle;

wherein said control, responsive to lane markers detection via processing of captured image data and to a determination based at least in part on processing of captured image data by said processor that the equipped vehicle is drifting out of a traffic lane that the equipped vehicle is currently travelling in and when a turn signal of the equipped vehicle is not being activated, controls a steering system of the equipped vehicle to adjust the steering direction of the equipped vehicle to mitigate such drift out of the traffic lane the equipped vehicle is travelling in, and wherein said steering system is manually controllable by a driver of the equipped vehicle irrespective of control by said control;

wherein said processor processes captured image data via an edge detection algorithm to detect edges of vehicles present exteriorly of the equipped vehicle and within the exterior field of view of said forward-facing camera;

wherein said processor processes captured image data to estimate distance from the equipped vehicle to a detected vehicle that is present exteriorly of the equipped vehicle and within the exterior field of view of said forward-facing camera;

wherein said control receives image data from a camera mounted at a driver-side exterior rearview mirror of the equipped vehicle and wherein said control receives image data from a camera mounted at a passenger-side exterior rearview mirror of the equipped vehicle;

wherein said processor, based at least in part on detection of lane markers via processing of captured image data, determines curvature of the road being traveled by the equipped vehicle;

wherein said processor processes captured image data for an adaptive cruise control system of the equipped vehicle; and

wherein said adaptive cruise control system comprises said forward-facing camera and a radar.

24. The forward-facing vision system of claim 23, wherein, responsive at least in part to processing of captured image data by said processor, speed of the equipped vehicle is adjusted in accordance with a traffic condition detected by said forward-facing vision system.

25. The forward-facing vision system of claim 23, wherein said control, responsive to taillight detection via processing of captured image data, provides an output for a headlamp control system of the equipped vehicle that controls at least one headlamp of the equipped vehicle.

26. The forward-facing vision system of claim 25, wherein said processor processes captured image data to estimate distance from the equipped vehicle to a detected leading vehicle that is present exteriorly of the equipped vehicle and within the exterior field of view of said forward-facing camera, and wherein, when the detected leading vehicle is determined to be within a threshold distance from the equipped vehicle, speed of the equipped vehicle is reduced.

27. The forward-facing vision system of claim 26, wherein said forward-facing camera is primary to said radar.

28. The forward-facing vision system of claim 26, wherein, responsive at least in part to processing of captured image data by said processor, speed of the equipped vehicle is adjusted in accordance with a road condition detected by said forward-facing vision system.