Multi-camera vision system for a vehicle

• US 10,144,353 B2
• Filed: 10/23/2017
• Issued: 12/04/2018
• Est. Priority Date: 08/21/2002
• Status: Expired due to Term

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Karner, Lee, Schofield, Kenneth, Lynam, Niall R.
• Primary Examiner(s): Perungavoor, Sathyanaraya V
• Assistant Examiner(s): Brown, Jr., Howard D
• Application Number:
  US15/790,173
• Publication Number:
  US 20180056871A1
• Time in Patent Office: 407 Days
• CPC Class Codes: B29C 45/0017 moulding interconnected elements which are movable with respect to one another, e.g. chains or hinges
• CPC Class Codes: B29C 45/14639 for obtaining an insulating effect, e.g. for electrical components
• CPC Class Codes: B29C 45/14754 being in movable or releasable engagement with the coating, e.g. bearing assemblies
• CPC Class Codes: B29C 45/1671 with an insert
• CPC Class Codes: B29L 2011/0058 Mirrors
• CPC Class Codes: B29L 2031/30 Vehicles, e.g. ships or aircraft, or body parts thereof vanes or blades B29L2031/08, air bags B29L2022/027
• CPC Class Codes: B60C 9/00 Reinforcements or ply arrangement of pneumatic tyres inserts having reinforcing means B60C5/08; bead structure, e.g. turnup or overlap construction, B60C15/00
• CPC Class Codes: B60Q 9/008 for anti-collision purposes
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/04 mounted inside vehicle B60R1/025, B60R1/08 take precedence
• CPC Class Codes: B60R 1/06 mounted on vehicle exterior B60R1/025, B60R1/08 take precedence
• CPC Class Codes: B60R 1/08 involving special optical features, e.g. avoiding blind spots , e.g. convex mirrors; Side-by-side associations of rear-view and other mirrors B60R1/025, B60R1/10 take precedence
• CPC Class Codes: B60R 1/12 Mirror assemblies combined with other articles, e.g. clocks
• CPC Class Codes: B60R 1/23 with a predetermined field of view
• CPC Class Codes: B60R 1/30 providing vision in the non-visible spectrum, e.g. night or infrared vision
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: B60R 2001/1215 with information displays
• CPC Class Codes: B60R 2001/1223 with sensors or transducers
• CPC Class Codes: B60R 2001/1253 with cameras, video cameras or video screens
• CPC Class Codes: B60R 2001/1269 with sun visors
• CPC Class Codes: B60R 2011/0033 : Rear-view mirrors
• CPC Class Codes: B60R 2300/105 : using multiple cameras
• CPC Class Codes: B60R 2300/308 : by overlaying the real scene, e.g. through a head-up display on the windscreen
• CPC Class Codes: B60R 2300/8026 : in addition to a rear-view mirror system
• CPC Class Codes: B60R 2300/8066 : for monitoring rearward traffic
• CPC Class Codes: B60R 2300/808 : for facilitating docking to a trailer
• CPC Class Codes: B60R 2300/8093 : for obstacle warning
• CPC Class Codes: G06T 2207/10016 : Video; Image sequence
• CPC Class Codes: G06T 2207/30232 : Surveillance
• CPC Class Codes: G06T 2207/30261 : Obstacle
• CPC Class Codes: G06T 7/13 : Edge detection
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/58 : Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G08G 1/165 : for passive traffic, e.g. including static obstacles, trees
• CPC Class Codes: G08G 1/166 : for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: G08G 1/167 : Driving aids for lane monitoring, lane changing, e.g. blind spot detection
• CPC Class Codes: G09G 2300/023 : Display panel composed of stacked panels
• CPC Class Codes: G09G 2300/0456 : with a reflective area and a transmissive area combined in one pixel, such as in transflectance pixels
• CPC Class Codes: G09G 2340/12 : Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 2380/10 : Automotive applications
• CPC Class Codes: G09G 3/001 : using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background slide projectors per se G03B23/00 = 42 HP
• CPC Class Codes: G09G 3/346 : based on modulation of the reflection angle, e.g. micromirrors micromirrors devices per se G02B26/0833
• CPC Class Codes: G09G 5/026 : Control of mixing and/or overlay of colours in general G09G5/022 and G09G5/024 take precedence
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source

Abstract

A multi-camera vision system for a vehicle includes a plurality of cameras viewing exterior the equipped vehicle and at least one non-visual sensor sensing exterior of the equipped vehicle. Captured image data and sensor data are provided to a central data processor. At the central data processor, image data is fused with sensor data and processed in order to detect vehicles that are present external the equipped vehicle. The central data processor includes a digital signal processor operable to process data at a data processioning speed of at least 30 MIPS. Via processing at the central data processor of image data and sensor data, a potential hazard present exterior the equipped vehicle is determined to exist. The potential hazard arises from presence of rear-approaching traffic in at least one of (i) a side lane to the equipped vehicle and (ii) a rear lane to the equipped vehicle.

Claims

1. A multi-camera vision system for a vehicle, said multi-camera vision system comprising:

a rear-mounted video camera at a rear portion of a vehicle equipped with said multi-camera vision system;

said rear-mounted video camera having at least a rearward field of view external of the equipped vehicle;

said rear-mounted video camera operable to capture image data at least rearward of the equipped vehicle during driving of the equipped vehicle;

a front-mounted video camera at a front portion of the equipped vehicle;

said front-mounted video camera having at least a forward field of view external of the equipped vehicle;

said front-mounted video camera operable to capture image data at least forward of the equipped vehicle during driving of the equipped vehicle;

a first side-mounted video camera at a driver-side side portion of the equipped vehicle;

said first side-mounted video camera having at least a sideward field of view external of the equipped vehicle;

said first side-mounted video camera operable to capture image data at least sideward of the equipped vehicle during driving of the equipped vehicle;

a second side-mounted video camera at a passenger-side side portion of the equipped vehicle;

said second side-mounted video camera having at least a sideward field of view external of the equipped vehicle;

said second side-mounted video camera operable to capture image data at least sideward of the equipped vehicle during driving of the equipped vehicle;

wherein the equipped vehicle is further equipped with at least one non-visual sensor having a field of sensing exterior of the equipped vehicle;

wherein said at least one non-visual sensor senses sensor data;

a central data processor disposed in the equipped vehicle;

wherein image data is provided to said central data processor;

wherein sensor data is provided to said central data processor;

wherein, at said central data processor, image data is fused with sensor data and processed in order to detect vehicles that are present external the equipped vehicle;

wherein processing at said central data processor of captured image data comprises use of an edge detection algorithm;

wherein said central data processor comprises a digital signal processor operable to process data at a data processioning speed of at least 30 MIPS;

wherein, via processing at said central data processor of image data and sensor data, a potential hazard present exterior the equipped vehicle is determined to exist; and

wherein said potential hazard arises from presence of rear-approaching traffic in at least one of (i) a side lane to the equipped vehicle and (ii) a rear lane to the equipped vehicle.

2. The multi-camera vision system of claim 1, wherein data is provided to said central data processor via a vehicle network of the equipped vehicle.

3. The multi-camera vision system of claim 1, wherein image data and sensor data are processed at said central data processor to generate an alert to warn a driver of the equipped vehicle who is operating the equipped vehicle of the potential hazard determined to exist exterior the equipped vehicle.

4. The multi-camera vision system of claim 1, wherein said first side-mounted video camera at the driver-side side portion of the equipped vehicle is included in a driver-side side view mirror assembly of the equipped vehicle, and wherein said second side-mounted video camera at the passenger-side side portion of the equipped vehicle is included in a passenger-side side view mirror assembly of the equipped vehicle, and wherein the field of view of said rear-mounted video camera encompasses at least a portion of a vehicle bumper of the equipped vehicle.

5. The multi-camera vision system of claim 4, wherein said front-mounted video camera, said first side-mounted video camera, said second side-mounted video camera, said rear-mounted video camera and said at least one non-visual sensor are part of a park aid system of the equipped vehicle.

6. The multi-camera vision system of claim 4, wherein video images derived from image data captured by at least some of said front-mounted video camera, said first side-mounted video camera, said second side-mounted video camera and said rear-mounted video camera are output by said central data processor for display by a display device of the equipped vehicle, and wherein said display device comprises a video screen operable to display the video images for viewing by a driver of the equipped vehicle who is operating the equipped vehicle.

7. The multi-camera vision system of claim 6, wherein the video images displayed by said video screen inform the driver of the environment in which the equipped vehicle is being operated.

8. The multi-camera vision system of claim 7, wherein said display device is disposed at an interior rearview mirror assembly of the equipped vehicle, and wherein at least one of (a) said display device displays video images for viewing through a mirror reflective element of said interior rearview mirror assembly by the driver of the equipped vehicle and (b) said display device automatically extends from said interior rearview mirror assembly responsive to the vehicle transmission being shifted into reverse gear, and wherein said interior rearview mirror assembly comprises a molded plastic mounting arm having at least one ball member at least one end thereof and having electrical conducting elements extending therethrough, and wherein a mirror mounting portion is molded over said ball member, and wherein said mounting arm comprises a mounting arm polymeric material having a first linear mold shrinkage factor and a first flexural modulus, and wherein said mirror mounting portion comprises a mirror mounting polymeric material having a second linear mold shrinkage factor and a second flexural modulus, and wherein said second linear mold shrinkage factor is greater than said first linear mold shrinkage factor, and wherein said second flexural modulus is lower than said first flexural modulus.

9. The multi-camera vision system of claim 7, wherein the video images comprise an overlay that guides the driver of the equipped vehicle when hitching a trailer to a vehicle hitch of the equipped vehicle.

10. The multi-camera vision system of claim 9, wherein said overlay aids in guiding connection of the vehicle hitch of the equipped vehicle to a trailer tongue of the trailer.

11. The multi-camera vision system of claim 10, wherein said video screen of said display device comprises a backlit thin film transistor LCD video screen backlit by a plurality of white light emitting light emitting diodes.

12. The multi-camera vision system of claim 11, wherein said plurality of white light emitting light emitting diodes comprises an array of at least four white light emitting light emitting diodes.

13. The multi-camera vision system of claim 12, wherein said display device is operable to display video images with a display intensity, as viewed by the driver of the equipped vehicle, greater than 200 candelas/sq. meter.

14. The multi-camera vision system of claim 4, wherein a vehicle hitch is disposed at a rear portion of the equipped vehicle, and wherein the rearward field of view of said rear-mounted video camera encompasses the vehicle hitch, and wherein the vehicle hitch is configured to connect to a trailer tongue of a trailer to connect the trailer to the equipped vehicle.

15. The multi-camera vision system of claim 14, wherein said rear-mounted video camera comprises a rear backup camera, and wherein video images derived from image data captured by at least some of said front-mounted video camera, said first side-mounted video camera, said second side-mounted video camera and said rear-mounted video camera are output by said central data processor for display by a display device of the equipped vehicle, and wherein said display device comprises a video screen operable to display the video images for viewing by a driver of the equipped vehicle who is operating the equipped vehicle, and wherein the video images displayed by said video screen inform the driver of the environment in which the equipped vehicle is being operated.

16. The multi-camera vision system of claim 15, wherein a vehicle hitch is disposed at a rear portion of the equipped vehicle, and wherein the rearward field of view of said rear-mounted video camera encompasses the vehicle hitch, and wherein the video images output by said central data processor for display by a display device of the equipped vehicle comprise an overlay that guides the driver of the equipped vehicle when hitching a trailer to the vehicle hitch of the equipped vehicle, and wherein said overlay aids in guiding connection of the vehicle hitch of the equipped vehicle to a trailer tongue of the trailer to connect the trailer to the equipped vehicle.

17. The multi-camera vision system of claim 4, wherein said at least one non-visual sensor comprises an ultrasonic sensor, and wherein at least one of (i) image data captured by at said rear-mounted video camera and (ii) sensor data sensed by said ultrasonic sensor is processed at said central data processor in order to detect an obstacle that is within the field of view of said rear-mounted video camera.

18. The multi-camera vision system of claim 17, wherein image data captured by said rear-mounted video camera is processed at said central data processor in order to detect a person that is within the field of view of said rear-mounted video camera.

19. The multi-camera vision system of claim 1, wherein said at least one non-visual sensor comprises a radar sensor.

20. The multi-camera vision system of claim 1, wherein said at least one non-visual sensor comprises an ultrasonic sensor.

21. The multi-camera vision system of claim 20, wherein processing at said central data processor of image data and sensor data detects the presence of an obstacle exterior the equipped vehicle.

22. The multi-camera vision system of claim 1, wherein said rear-mounted video camera comprises a CMOS imaging array.

23. The multi-camera vision system of claim 1, wherein image data captured by at least some of said front-mounted video camera, said first side-mounted video camera, said second side-mounted video camera and said rear-mounted video camera is processed at said central data processor in order to detect an obstacle that is within the combined field of view of said front-mounted video camera, said first side-mounted video camera, said second side-mounted video camera and said rear-mounted video camera.

24. The multi-camera vision system of claim 1, wherein processing at said central data processor of image data and sensor data detects a person that is within the combined field of view of said front-mounted video camera, said first side-mounted video camera, said second side-mounted video camera and said rear-mounted video camera.

25. A multi-camera vision system for a vehicle, said multi-camera vision system comprising:

a rear-mounted video camera at a rear portion of a vehicle equipped with said multi-camera vision system;

said rear-mounted video camera having at least a rearward field of view external of the equipped vehicle;

said rear-mounted video camera operable to capture image data at least rearward of the equipped vehicle during driving of the equipped vehicle;

a front-mounted video camera at a front portion of the equipped vehicle;

said front-mounted video camera having at least a forward field of view external of the equipped vehicle;

said front-mounted video camera operable to capture image data at least forward of the equipped vehicle during driving of the equipped vehicle;

a first side-mounted video camera at a driver-side side portion of the equipped vehicle;

said first side-mounted video camera having at least a sideward field of view external of the equipped vehicle;

said first side-mounted video camera operable to capture image data at least sideward of the equipped vehicle during driving of the equipped vehicle;

a second side-mounted video camera at a passenger-side side portion of the equipped vehicle;

said second side-mounted video camera having at least a sideward field of view external of the equipped vehicle;

said second side-mounted video camera operable to capture image data at least sideward of the equipped vehicle during driving of the equipped vehicle;

wherein the equipped vehicle is further equipped with at least one non-visual sensor having a field of sensing exterior of the equipped vehicle;

wherein said at least one non-visual sensor senses sensor data;

a central data processor disposed in the equipped vehicle;

wherein image data is provided to said central data processor;

wherein sensor data is provided to said central data processor;

wherein image data and sensor data are processed at said central data processor;

wherein processing at said central data processor of captured image data comprises use of an edge detection algorithm;

wherein said central data processor comprises a digital signal processor operable to process data at a data processioning speed of at least 30 MIPS;

wherein said rear-mounted video camera comprises a rear backup camera;

wherein video images derived from image data captured by said rear-mounted video camera and at least one of said first side-mounted video camera and said second side-mounted video camera are output by said central data processor for display by a display device of the equipped vehicle;

wherein said display device comprises a video screen operable to display the video images for viewing by a driver of the equipped vehicle who is operating the equipped vehicle;

wherein the video images displayed by said video screen inform the driver of the environment in which the equipped vehicle is being operated;

wherein said at least one non-visual sensor comprises an ultrasonic sensor;

wherein at least one of (i) image data captured by said rear-mounted video camera and (ii) sensor data sensed by said ultrasonic sensor is processed at said central data processor in order to detect an obstacle that is within the field of view of said rear-mounted video camera; and

wherein said front-mounted video camera, said first side-mounted video camera, said second side-mounted video camera, said rear-mounted video camera and said ultrasonic sensor are part of a park aid system of the equipped vehicle.

26. The multi-camera vision system of claim 25, wherein said first side-mounted video camera at the driver-side side portion of the equipped vehicle is included in a driver-side side view mirror assembly of the equipped vehicle, and wherein said second side-mounted video camera at the passenger-side side portion of the equipped vehicle is included in a passenger-side side view mirror assembly of the equipped vehicle, and wherein the field of view of said rear-mounted video camera encompasses at least a portion of a vehicle bumper of the equipped vehicle.

27. The multi-camera vision system of claim 26, wherein a vehicle hitch is disposed at a rear portion of the equipped vehicle, and wherein the rearward field of view of said rear-mounted video camera encompasses the vehicle hitch, and wherein the video images comprise an overlay that guides the driver of the equipped vehicle when hitching a trailer to the vehicle hitch of the equipped vehicle, and wherein said overlay aids in guiding connection of the vehicle hitch of the equipped vehicle to a trailer tongue of the trailer to connect the trailer to the equipped vehicle.

28. The multi-camera vision system of claim 27, wherein image data captured by said rear-mounted video camera is processed at said central data processor in order to detect an obstacle that is within the field of view of said rear-mounted video camera.

29. The multi-camera vision system of claim 27, wherein sensor data sensed by said ultrasonic sensor is processed at said central data processor in order to detect an obstacle that is within the field of sensing of said ultrasonic sensor.

30. The multi-camera vision system of claim 26, wherein image data and sensor data are processed at said central data processor to generate an alert to warn a driver of the equipped vehicle who is operating the equipped vehicle of the potential hazard determined to exist exterior the equipped vehicle.

31. A multi-camera vision system for a vehicle, said multi-camera vision system comprising:

a rear-mounted video camera at a rear portion of a vehicle equipped with said multi-camera vision system;

said rear-mounted video camera having at least a rearward field of view external of the equipped vehicle;

said rear-mounted video camera operable to capture image data at least rearward of the equipped vehicle during driving of the equipped vehicle;

a front-mounted video camera at a front portion of the equipped vehicle;

said front-mounted video camera having at least a forward field of view external of the equipped vehicle;

said front-mounted video camera operable to capture image data at least forward of the equipped vehicle during driving of the equipped vehicle;

a first side-mounted video camera at a driver-side side portion of the equipped vehicle;

said first side-mounted video camera having at least a sideward field of view external of the equipped vehicle;

said first side-mounted video camera operable to capture image data at least sideward of the equipped vehicle during driving of the equipped vehicle;

a second side-mounted video camera at a passenger-side side portion of the equipped vehicle;

said second side-mounted video camera having at least a sideward field of view external of the equipped vehicle;

said second side-mounted video camera operable to capture image data at least sideward of the equipped vehicle during driving of the equipped vehicle;

wherein the equipped vehicle is further equipped with at least one radar sensor having a field of sensing exterior of the equipped vehicle;

wherein said at least one radar sensor senses radar data;

a central data processor disposed in the equipped vehicle;

wherein image data is provided to said central data processor;

wherein radar data is provided to said central data processor;

wherein said central data processor processes image data and radar data in order to detect vehicles that are present external the equipped vehicle;

wherein processing at said central data processor of captured image data comprises use of an edge detection algorithm;

wherein said central data processor comprises a digital signal processor operable to process data at a data processioning speed of at least 30 MIPS;

wherein, via processing at said central data processor of image data and radar data, a potential hazard present exterior the equipped vehicle is determined to exist; and

wherein said potential hazard arises from presence of rear-approaching traffic in a side lane to the equipped vehicle.

32. The multi-camera vision system of claim 31, wherein said first side-mounted video camera at the driver-side side portion of the equipped vehicle is included in a driver-side side view mirror assembly of the equipped vehicle, and wherein said second side-mounted video camera at the passenger-side side portion of the equipped vehicle is included in a passenger-side side view mirror assembly of the equipped vehicle.

33. The multi-camera vision system of claim 32, wherein said rear-mounted video camera comprises a rear backup camera, and wherein video images derived from image data captured by at least said rear-mounted video camera are output by said central data processor for display by a display device of the equipped vehicle, and wherein said display device comprises a video screen operable to display the video images for viewing by a driver of the equipped vehicle who is operating the equipped vehicle, and wherein the field of view of said rear-mounted video camera encompasses at least a portion of a vehicle bumper of the equipped vehicle.

34. The multi-camera vision system of claim 33, wherein a vehicle hitch is disposed at a rear portion of the equipped vehicle, and wherein the rearward field of view of said rear-mounted video camera encompasses the vehicle hitch, and wherein the vehicle hitch is configured to connect to a trailer tongue of a trailer to connect the trailer to the equipped vehicle.

35. The multi-camera vision system of claim 34, wherein the video images output by said central data processor for display by the display device of the equipped vehicle comprise an overlay that guides the driver of the equipped vehicle when hitching a trailer to the vehicle hitch of the equipped vehicle, and wherein said overlay aids in guiding connection of the vehicle hitch of the equipped vehicle to a trailer tongue of the trailer to connect the trailer to the equipped vehicle.

36. The multi-camera vision system of claim 33, wherein said at least one non-visual sensor comprises an ultrasonic sensor, and wherein at least one of (i) image data captured by at said rear-mounted video camera and (ii) sensor data sensed by said ultrasonic sensor is processed at said central data processor in order to detect an obstacle that is within the field of view of said rear-mounted video camera.

37. The multi-camera vision system of claim 36, wherein said front-mounted video camera, said first side-mounted video camera, said second side-mounted video camera, said rear-mounted video camera and said ultrasonic sensor are part of a park aid system of the equipped vehicle.

38. The multi-camera vision system of claim 36, wherein said at least one radar sensor comprises a Doppler radar sensor.