Driver active safety control system for vehicle

• US 9,911,050 B2
• Filed: 09/04/2015
• Issued: 03/06/2018
• Est. Priority Date: 02/27/2009
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Lynam, Niall R., Nix, Axel
• Primary Examiner(s): Huang, Frank
• Application Number:
  US14/845,830
• Publication Number:
  US 20160042238A1
• Time in Patent Office: 914 Days
• Field of Search: 348148
• CPC Class Codes: B60Q 9/005 using a video camera
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/12 Mirror assemblies combined with other articles, e.g. clocks
• CPC Class Codes: B60R 1/26 to the rear of the vehicle
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 1/28 with an adjustable field of view
• CPC Class Codes: B60R 2001/1215 with information displays
• CPC Class Codes: B60R 2001/1253 with cameras, video cameras or video screens
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/302 combining image information with GPS information or vehicle data, e.g. vehicle speed, gyro, steering angle data
• CPC Class Codes: B60R 2300/303 using joined images, e.g. multiple camera images
• CPC Class Codes: B60R 2300/605 the adjustment being automatic
• CPC Class Codes: B60R 2300/607 from a bird's eye viewpoint
• CPC Class Codes: B60R 2300/806 for aiding parking
• CPC Class Codes: B60W 10/04 including control of propulsion units
• CPC Class Codes: B60W 10/18 including control of braking systems
• CPC Class Codes: B60W 10/20 including control of steering systems
• CPC Class Codes: B60W 40/02 related to ambient conditions
• CPC Class Codes: B60W 50/0098 Details of control systems ensuring comfort, safety or stability not otherwise provided for
• CPC Class Codes: G05D 1/0061 for transition from automatic pilot to manual pilot and vice versa
• CPC Class Codes: G06V 20/58 : Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: H04N 23/63 : by using electronic viewfinders
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A driver active safety control system for a vehicle includes a plurality of image capture sensors, a video display screen, a central driver active safety control module and at least one non-imaging sensor. The control module at least includes an image processor, a vision core and a fusion core. The image processor processes image data captured by and received from at least a forward viewing image capture sensor for at least one of (i) automatic headlamp control, (ii) lane departure warning and (iii) traffic sign recognition. The vision core is operable to manipulate image data captured by and received at least from rearward and sideward viewing image capture sensors to form video images for displaying on the video display screen. The fusion core is operable to process inputs received to enhance control by the control module of a driver assistance system of the equipped vehicle.

Claims

1. A driver active safety control system for a vehicle, said driver active safety control system comprising:

a plurality of image capture sensors configured to be disposed at a vehicle equipped with said driver active safety control system so as to have respective fields of view exterior of the equipped vehicle;

wherein said plurality of image capture sensors comprises at least (i) a forward viewing image capture sensor configured to be disposed at the equipped vehicle so as to have a forward field of view in a direction of forward travel of the equipped vehicle, (ii) a rearward viewing image capture sensor configured to be disposed at the equipped vehicle so as to have a rearward field of view in a direction of rearward travel of the equipped vehicle, (iii) a driver-side sideward viewing image capture sensor configured to be disposed at a driver side of the body of the equipped vehicle so as to have a driver-side sideward field of view sideward of the equipped vehicle and (iv) a passenger-side sideward viewing image capture sensor configured to be disposed at a passenger side of the body of the equipped vehicle so as to have a passenger-side sideward field of view sideward of the equipped vehicle;

a video display screen for displaying video images derived from image data captured at least by said rearward viewing image capture sensor, said driver-side sideward viewing image capture sensor and said passenger-side sideward viewing image capture sensor of said plurality of image capture sensors, wherein said video display screen is viewable by a driver of a vehicle equipped with said driver active safety control system;

a central driver active safety control module;

wherein said central driver active safety control module at least comprises an image processor, a vision core and a fusion core;

wherein image data captured by said plurality of image capture sensors is provided to said central driver active safety control module;

wherein said image processor processes, via machine vision analysis, image data captured by and received from at least said forward viewing image capture sensor for at least one of (i) automatic headlamp control, (ii) lane departure warning and (iii) traffic sign recognition;

wherein said vision core is operable to manipulate image data captured by and received at least from said rearward viewing image capture sensor, said driver-side sideward viewing image capture sensor and said passenger-side sideward viewing image capture sensor of said plurality of image capture sensors to form said video images for displaying on said video display screen for viewing by the driver of the equipped vehicle;

at least one non-imaging sensor configured to be disposed at the equipped vehicle and sensing exterior the equipped vehicle;

wherein sensor data captured by said at least one non-imaging sensor is provided to said central driver active safety control module;

wherein said fusion core at least receives (a) input from said image processor, (b) input from said vision core and (c) input from said at least one non-imaging sensor;

wherein said fusion core receives input from at least one of (i) a global positioning system, (ii) a telematics system, (iii) a Car2Car communication system and (iv) a Car2X communication system;

wherein said fusion core receives vehicle data relating to operation of the equipped vehicle, said vehicle data comprising at least one of (i) vehicle speed data, (ii) vehicle steering wheel angle data, (iii) vehicle yaw rate data, (iv) vehicle type data and (v) vehicle acceleration data;

wherein said fusion core is operable to process vehicle data and inputs received (a) from said image processor, (b) from said vision core, (c) from said at least one non-imaging sensor and (d) from at least one of (i) a global positioning system, (ii) a telematics system, (iii) a Car2Car communication system and (iv) a Car2X communication system to enhance control by said central driver active safety control module of a driver assistance system of the equipped vehicle; and

wherein, responsive at least in part to processing by said central driver active safety control module of at least one selected from the group consisting of (i) captured image data and (ii) captured sensor data, presence of an object rearward of the equipped vehicle is determined.

2. The driver active safety control system of claim 1, wherein at least two of said image processor, said vision core and said fusion core are co-established on a common substrate.

3. The driver active safety control system of claim 1, wherein said image processor processes image data captured by said rearward viewing image capture sensor for object detection rearward of the equipped vehicle, and wherein, during a reversing maneuver of the equipped vehicle and responsive to processing by said central driver active safety control module of captured image data and captured sensor data, presence of an object rearward of the equipped vehicle is determined.

4. The driver active safety control system of claim 1, wherein said non-imaging sensor comprises part of at least one of (a) a radar sensing system having at least one radar sensor at the rear of the equipped vehicle, (b) an ultrasonic sensing system having at least one ultrasonic sensor at the rear of the equipped vehicle, and (c) a lidar sensing system having at least one lidar sensor at the rear of the equipped vehicle.

5. The driver active safety control system of claim 1, wherein said non-imaging sensor comprises a sensor selected from the group consisting of (i) a radar sensor, (ii) an infrared sensor, (iii) an IROD sensor, (iv) a lidar sensor and (v) an ultrasonic sensor.

6. The driver active safety control system of claim 1, wherein said non-imaging sensor comprises a three dimensional sensing sensor.

7. The driver active safety control system of claim 1, wherein said non-imaging sensor comprises an array of infrared emitting light emitting diodes.

8. The driver active safety control system of claim 1, wherein said non-imaging sensor comprises a scanning sensor.

9. The driver active safety control system of claim 1, wherein said image processor processes image data to determine a driving situation and wherein, in determining said driving situation, said fusion core processes sensor data to augment processing of image data by said image processor.

10. The driver active safety control system of claim 1, wherein at least two of said image processor, said vision core and said fusion core are co-established on a common semiconductor substrate.

11. The driver active safety control system of claim 1, wherein captured image data is communicated to said central driver active safety control module via at least one Ethernet link.

12. The driver active safety control system of claim 1, wherein said central driver active safety control module, responsive to a geographical location of the equipped vehicle as determined by a global positioning system, at least in part controls said driver assistance system.

13. The driver active safety control system of claim 1, wherein said vision core processes image data to generate a bird'"'"'s eye view of a region exterior the equipped vehicle for displaying on said video display screen.

14. The driver active safety control system of claim 13, wherein said vision core is operable, responsive to detection of an object external of the equipped vehicle by said central driver active safety control module, to generate a graphic overlay on the displayed bird'"'"'s eye view image.

15. The driver active safety control system of claim 1, wherein said central driver active safety control module generates an output for at least one vehicle control.

16. The driver active safety control system of claim 15, wherein said at least one vehicle control comprises at least one of (i) a vehicle body control, (ii) a vehicle chassis control, (iii) a vehicle supervisory control, (iv) a vehicle brake control, (v) a vehicle steering control and (vi) a vehicle engine control.

17. The driver active safety control system of claim 15, wherein said central driver active safety control module controls said at least one vehicle control responsive at least in part to image processing of image data captured by said plurality of image capture sensors and responsive at least in part to processing of sensor data captured by said at least one non-imaging sensor.

18. The driver active safety control system of claim 1, wherein said image processor, said vision core and said fusion core are co-established on a common substrate.

19. The driver active safety control system of claim 1, wherein said driver assistance system comprises automatic emergency braking.

20. The driver active safety control system of claim 19, wherein said forward viewing image capture sensor is disposed in the equipped vehicle and behind a windshield of the equipped vehicle.

21. A driver active safety control system for a vehicle, said driver active safety control system comprising:

a plurality of image capture sensors configured to be disposed at a vehicle equipped with said driver active safety control system so as to have respective fields of view exterior of the equipped vehicle;

wherein said plurality of image capture sensors comprises at least (i) a forward viewing image capture sensor configured to be disposed at the equipped vehicle so as to have a forward field of view in a direction of forward travel of the equipped vehicle, (ii) a rearward viewing image capture sensor configured to be disposed at the equipped vehicle so as to have a rearward field of view in a direction of rearward travel of the equipped vehicle, (iii) a driver-side sideward viewing image capture sensor configured to be disposed at a driver side of the body of the equipped vehicle so as to have a driver-side sideward field of view sideward of the equipped vehicle and (iv) a passenger-side sideward viewing image capture sensor configured to be disposed at a passenger side of the body of the equipped vehicle so as to have a passenger-side sideward field of view sideward of the equipped vehicle;

a video display screen for displaying video images derived from image data captured at least by said rearward viewing image capture sensor, said driver-side sideward viewing image capture sensor and said passenger-side sideward viewing image capture sensor of said plurality of image capture sensors, wherein said video display screen is viewable by a driver of a vehicle equipped with said driver active safety control system;

a central driver active safety control module;

wherein said central driver active safety control module at least comprises an image processor, a vision core and a fusion core;

wherein image data captured by said plurality of image capture sensors is provided to said central driver active safety control module;

wherein said image processor processes, via machine vision analysis, image data captured by and received from at least said forward viewing image capture sensor for at least one of (i) automatic headlamp control, (ii) lane departure warning and (iii) traffic sign recognition;

wherein said vision core is operable to manipulate image data captured by and received from at least said rearward viewing image capture sensor, said driver-side sideward viewing image capture sensor and said passenger-side sideward viewing image capture sensor of said plurality of image capture sensors to form said video images for displaying on said video display screen for viewing by the driver of the equipped vehicle;

at least one non-imaging sensor configured to be disposed at the equipped vehicle and sensing exterior the equipped vehicle;

wherein said non-imaging sensor comprises a three dimensional sensing sensor;

wherein sensor data captured by said at least one non-imaging sensor is provided to said central driver active safety control module;

wherein said fusion core at least receives (a) input from said image processor, (b) input from said vision core and (c) input from said at least one non-imaging sensor;

wherein said fusion core receives input from at least one of (i) a global positioning system, (ii) a telematics system, (iii) a Car2Car communication system and (iv) a Car2X communication system;

wherein said fusion core receives vehicle data relating to operation of the equipped vehicle, said vehicle data comprising at least one of (i) vehicle speed data, (ii) vehicle steering wheel angle data, (iii) vehicle yaw rate data, (iv) vehicle type data and (v) vehicle acceleration data;

wherein said fusion core is operable to process vehicle data and inputs received (a) from said image processor, (b) from said vision core, (c) from said at least one non-imaging sensor and (d) from at least one of (i) a global positioning system, (ii) a telematics system, (iii) a Car2Car communication system and (iv) a Car2X communication system to enhance control by said central driver active safety control module of a driver assistance system of the equipped vehicle;

wherein said central driver active safety control module generates an output for at least one vehicle control;

wherein, responsive at least in part to processing by said central driver active safety control module of at least one selected from the group consisting of (i) captured image data and (ii) captured sensor data, presence of an object rearward of the equipped vehicle is determined.

22. The driver active safety control system of claim 21, wherein said non-imaging sensor comprises a lidar sensor.

23. The driver active safety control system of claim 22, wherein said non-imaging sensor comprises a scanning sensor.

24. The driver active safety control system of claim 21, wherein said central driver active safety control module, responsive to a geographical location of the equipped vehicle as determined by a global positioning system, at least in part controls said driver assistance system.

25. The driver active safety control system of claim 21, wherein said vision core processes image data to generate a bird'"'"'s eye view of a region exterior the equipped vehicle for displaying on said video display screen.

26. The driver active safety control system of claim 25, wherein said vision core is operable, responsive to detection of an object external of the equipped vehicle by said central driver active safety control module, to generate a graphic overlay on the displayed bird'"'"'s eye view image.

27. The driver active safety control system of claim 21, wherein said at least one vehicle control comprises at least one of (i) a vehicle body control, (ii) a vehicle chassis control, (iii) a vehicle supervisory control, (iv) a vehicle brake control, (v) a vehicle steering control and (vi) a vehicle engine control.

28. A driver active safety control system for a vehicle, said driver active safety control system comprising:

a plurality of image capture sensors configured to be disposed at a vehicle equipped with said driver active safety control system so as to have respective fields of view exterior of the equipped vehicle;

wherein said plurality of image capture sensors comprises at least (i) a forward viewing image capture sensor configured to be disposed at the equipped vehicle so as to have a forward field of view in a direction of forward travel of the equipped vehicle, (ii) a rearward viewing image capture sensor configured to be disposed at the equipped vehicle so as to have a rearward field of view in a direction of rearward travel of the equipped vehicle, (iii) a driver-side sideward viewing image capture sensor configured to be disposed at a driver side of the body of the equipped vehicle so as to have a driver-side sideward field of view sideward of the equipped vehicle and (iv) a passenger-side sideward viewing image capture sensor configured to be disposed at a passenger side of the body of the equipped vehicle so as to have a passenger-side sideward field of view sideward of the equipped vehicle;

a video display screen for displaying video images derived from image data captured at least by said rearward viewing image capture sensor, said driver-side sideward viewing image capture sensor and said passenger-side sideward viewing image capture sensor of said plurality of image capture sensors, wherein said video display screen is viewable by a driver of a vehicle equipped with said driver active safety control system;

wherein said vision core processes image data to generate a bird'"'"'s eye view of a region exterior the equipped vehicle for displaying on said video display screen;

a central driver active safety control module;

wherein said central driver active safety control module at least comprises an image processor, a vision core and a fusion core;

wherein image data captured by said plurality of image capture sensors is provided to said central driver active safety control module;

wherein said image processor processes, via machine vision analysis, image data captured by and received from at least said forward viewing image capture sensor for at least one of (i) automatic headlamp control, (ii) lane departure warning and (iii) traffic sign recognition;

wherein said vision core is operable to manipulate image data captured by and received from at least said rearward viewing image capture sensor, said driver-side sideward viewing image capture sensor and said passenger-side sideward viewing image capture sensor of said plurality of image capture sensors to form said video images for displaying on said video display screen for viewing by the driver of the equipped vehicle;

at least one non-imaging sensor configured to be disposed at the equipped vehicle and sensing exterior the equipped vehicle;

wherein said non-imaging sensor comprises a three dimensional sensing sensor and wherein said three dimensional sensing sensor comprises a lidar sensor;

wherein sensor data captured by said at least one non-imaging sensor is provided to said central driver active safety control module;

wherein said fusion core at least receives (a) input from said image processor, (b) input from said vision core and (c) input from said at least one non-imaging sensor;

wherein said fusion core receives input from at least one of (i) a global positioning system, (ii) a telematics system, (iii) a Car2Car communication system and (iv) a Car2X communication system;

wherein said fusion core receives vehicle data relating to operation of the equipped vehicle, said vehicle data comprising at least one of (i) vehicle speed data, (ii) vehicle steering wheel angle data, (iii) vehicle yaw rate data, (iv) vehicle type data and (v) vehicle acceleration data;

wherein said fusion core is operable to process vehicle data and inputs received (a) from said image processor, (b) from said vision core, (c) from said at least one non-imaging sensor and (d) from at least one of (i) a global positioning system, (ii) a telematics system, (iii) a Car2Car communication system and (iv) a Car2X communication system to enhance control by said central driver active safety control module of a driver assistance system of the equipped vehicle;

wherein said central driver active safety control module generates an output for at least one vehicle control; and

wherein, responsive at least in part to processing by said central driver active safety control module of at least one selected from the group consisting of (i) captured image data and (ii) captured sensor data, presence of an object rearward of the equipped vehicle is determined.

29. The driver active safety control system of claim 28, wherein said non-imaging sensor comprises a scanning sensor.

30. The driver active safety control system of claim 28, wherein said central driver active safety control module, responsive to a geographical location of the equipped vehicle as determined by a global positioning system, at least in part controls said driver assistance system.

31. The driver active safety control system of claim 28, wherein said vision core is operable, responsive to detection of an object external of the equipped vehicle by said central driver active safety control module, to generate a graphic overlay on the displayed bird'"'"'s eye view image.

32. The driver active safety control system of claim 28, wherein said at least one vehicle control comprises at least one of (i) a vehicle body control, (ii) a vehicle chassis control, (iii) a vehicle supervisory control, (iv) a vehicle brake control, (v) a vehicle steering control and (vi) a vehicle engine control.