Vehicle imaging system with accessory control

• US 20020135468A1
• Filed: 01/14/2002
• Published: 09/26/2002
• Est. Priority Date: 09/22/1997
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Donnelly Corporation (Magna International Incorporated)
• Inventors: Schofield, Kenneth, Bos, Brent J.
• Granted Patent: US 6,822,563 B2
• Time in Patent Office: Days
• US Class Current: 340/436
• CPC Class Codes: B60N 2/002 Seats provided with an occupancy detection means mounted therein or thereon detection means responsive to presence or absence of children B60N2/266
• CPC Class Codes: B60Q 1/143 combined with another condition, e.g. using vehicle recognition from camera images or activation of wipers
• CPC Class Codes: B60Q 2300/41 preceding vehicle
• CPC Class Codes: B60Q 2300/42 oncoming vehicle
• CPC Class Codes: B60Q 9/008 for anti-collision purposes
• CPC Class Codes: B60R 1/12 Mirror assemblies combined with other articles, e.g. clocks
• CPC Class Codes: B60R 1/26 to the rear of the vehicle
• CPC Class Codes: B60R 1/31 providing stereoscopic vision
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: B60R 2011/0003 inside the vehicle
• CPC Class Codes: B60R 2011/0026 Windows, e.g. windscreen
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/107 using stereoscopic cameras
• CPC Class Codes: B60R 2300/30 characterised by the type of image processing
• CPC Class Codes: B60R 2300/305 merging camera image with lines or icons
• CPC Class Codes: B60R 2300/804 for lane monitoring
• CPC Class Codes: B60R 2300/8066 for monitoring rearward traffic
• CPC Class Codes: G01B 11/026 by measuring distance between sensor and object G01B11/0608 takes precedence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30252 : Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/593 : from stereo images
• CPC Class Codes: G06T 7/97 : Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 20/584 : of vehicle lights or traffic lights
• CPC Class Codes: H04N 13/239 : using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 2013/0081 : Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 25/00 : Circuitry of solid-state image sensors [SSIS]; Control thereof
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source

Abstract

A vehicular stereoscopic imaging system provides a calculation of a distance between one or more sensors on a vehicle and a light source or object in a target scene remote from the vehicle. The imaging system may be useful in a headlamp dimmer control system, such that the headlamps are modulated between their low and high beams in response to the calculated distance and intensities and/or colors of the light sources sensed by the sensors. The stereoscopic imaging system determines the distance to the object by comparing similarly classified signals received by each of two imaging array sensors in order to determine a relative position of an image representing the light source on each sensor. The distance to the object or light source may then be calculated as a function of the respective positions on the sensors, the focal lengths of focusing optics associated with each sensor and a predetermined separation distance between the axes of the two sensors. An associated accessory, such as a display, headlamps, windshield wipers, a warning indicator or signaling device and/or the brake system of the vehicle may be adjusted or activated in response to an output of the imaging system.

Claims

1. A vehicular imaging system, comprising:

at least one imaging array sensor mounted at a vehicle and directed outwardly from the vehicle, said at least one imaging array sensor having stereoscopic distance-sensing capability; and

a control that is responsive to an output of said at least one imaging array sensor in order to capture an image of at least one object external to the vehicle and determine a distance between said at least one imaging array sensor and the at least one object external to the vehicle.

2. The vehicular imaging system of claim 1, wherein said at least one imaging array sensor receives a stereoscopic image of the at least one object, said at least one imaging array sensor including a first portion for receiving a first image of the at least one object and a second portion for receiving a second image of the at least one object.

3. The vehicular imaging system of claim 2, wherein said control compares a first registration of the first image on said first portion to a second registration of the second image on said second portion, said control determining a deviation between said first and second registrations relative to at least one reference point associated with said at least one imaging array sensor and determining said distance as a function of said deviation and a separation distance associated with said first and second portions.

4. The vehicular imaging system of claim 3, wherein said at least one imaging array sensor is at least one pixelated array sensor defining said first and second portions, said first registration being at least one pixel on said first portion and said second registration being at least one pixel on said second portion.

5. The vehicular imaging system of claim 3, wherein said at least one imaging array sensor is at least one pixelated array sensor defining said first and second portions, said first registration being an average pixel location of said first image and said second registration being an average pixel location of said second image.

6. The vehicular imaging system of claim 3, wherein said at least one imaging array sensor is at least one pixelated array sensor, said control providing sub-pixel resolution to said deviation.

7. The vehicular imaging system of claim 3 further including a first optic element positioned along a first optic path between said first portion and the at least one object and a second optic element positioned along a second optic path between said second portion and the at least one object, said first optic having a first focal length and said second optic having a second focal length, said control determining said distance as a function of said deviation, said separation distance and said first and second focal lengths.

8. The vehicular imaging system of claim 2, wherein said at least one imaging array sensor is a divided imaging array sensor, said first portion being a first region of said divided imaging array sensor and said second portion being a second region of said divided imaging array sensor.

9. The vehicular imaging system of claim 2, wherein said at least one imaging array sensor comprises first and second imaging arrays, said first portion being on said first imaging array and said second portion being on said second imaging array.

10. The vehicular imaging system of claim 2 further including first and second optic elements, said first optic element having a first focal length and being positioned along a first optic path between said first portion and the object and said second optic element having a second focal length and being positioned along a second optic path between said second portion and the object.

11. The vehicular imaging system of claim 10, wherein said control is operable to determine the distance as a function of said first and second focal lengths.

12. The vehicular imaging system of claim 10, wherein said at least one imaging array sensor comprises first and second imaging array sensors, said first optic path being between said first imaging array sensor and the at least one object and said second optic path being between said second imaging array sensor and the at least one object.

13. The vehicular imaging system of claim 12 further including a first reflective surface positioned along and defining said first optic path and a second reflective surface positioned along and defining said second optic path, said first and second reflective surface redirecting said first and second images toward said first and second imaging array sensors, respectively.

14. The vehicular imaging system of claim 13, wherein said first and second imaging array sensors face generally opposite to one another, said first and second reflective surfaces redirecting said first and second images toward said first and second imaging array sensors, respectively.

15. The vehicular imaging system of claim 10, wherein said first and second portions are generally coplanar.

16. The vehicular imaging system of claim 15, wherein said at least one imaging array sensor is a divided imaging array sensor, said first portion being a first region of said divided imaging array sensor and said second portion being a second region of said divided imaging array sensor.

17. The vehicular imaging system of claim 15 further including at least one first reflective surface positioned along and defining said first optic path and at least one second reflective surface positioned along and defining said second optic path, said first and second reflective surfaces redirecting said first and second images toward said first and second portions of said imaging array sensor, respectively.

18. The vehicular imaging system of claim 2, wherein a first reference point of said first portion and a second reference point of said second portion are separated by a separation distance, said control calculating said distance as a function of said separation distance.

19. The vehicular imaging system of claim 18, wherein said control determines a first position of said first image relative to said first reference point and a second position of said second image relative to said second reference point, said control calculating said distance as a function of said first and second positions and said separation distance.

20. The vehicular imaging system of claim 19 further including a first optic element positioned along a first optic path between said first portion and the at least one object and a second optic element positioned along a second optic path between said second portion and the at least one object, said first optic having a first focal length and said second optic having a second focal length, said control determining said distance as a function of said first and second positions, said separation distance and said first and second focal lengths.

21. The vehicular imaging system of claim 20, wherein said distance is calculated according to the following equation:

22. The vehicular imaging system of claim 21, wherein said first reference point corresponds to a center axis of said first portion and said second reference point corresponds to a center axis of said second portion.

23. The vehicular imaging system of claim 21, wherein said at least one imaging array sensor is a pixelated array sensor, said first and second positions being an average of pixel locations of first and second segments.

24. The vehicular imaging system of claim 1, wherein said control compares a first registration of a first image of the at least one object relative to a first reference point on said at least one imaging array sensor with a second registration of a second image of the at least one object relative to a second reference point on said at least one imaging array sensor in order to determine the distance to the at least one object.

25. The vehicular imaging system of claim 1, wherein said control classifies each of a plurality of images received by said at least one imaging array sensor, said each image being classified according to at least one of a location relative to said at least one imaging array sensor, a size of said image and an intensity of said image.

26. The vehicular imaging system of claim 25, wherein said control is operable to vary a shutter speed of said at least one imaging array sensor in order to classify said each image according to intensity.

27. The vehicular imaging system of claim 25 further including at least one color filter associated with said at least one imaging array sensor, said control determining a color of said each image in order to classify said each image.

28. The vehicular imaging system of claim 25, wherein said imaging system is associated with a headlamp control, said images being classified as one of headlamps or taillights.

29. The vehicular imaging system of claim 25, wherein said at least one imaging array sensor is at least one pixelated imaging array comprising a plurality of photo-sensing pixels.

30. The vehicular imaging system of claim 29, wherein said at least one imaging array sensor receives a stereoscopic image of the object, said at least one imaging array sensor including a first portion for receiving a first image of the object and a second portion for receiving a second image of the object.

31. The vehicular imaging system of claim 30, wherein said control classifies and labels a plurality of segments representing a plurality of objects in the target scene in response to signals received by said pixels of said at least one pixelated imaging array, said plurality of segments being labeled according to a position on said at least one pixelated imaging array and an intensity of said signals, said control comparing each of said plurality of segments on said first portion with corresponding segments on said second portion in order to determine a deviation between a first relative position of said segments on said first portion and a second relative position of said corresponding segments on said second portion, said distance to each of the plurality of objects being calculated as a function of said deviation and a separation distance associated with said first and second portions.

32. The vehicular imaging system of claim 1, wherein said imaging system is associated with a headlamp control for controlling headlamps on a vehicle, said headlamp control adjusting an output of said headlamps in response to said distance and said output of said at least one imaging array sensor.

33. The vehicular imaging system of claim 1, wherein said imaging system is associated with a rearview imaging system including a display within a vehicle for providing an operator of the vehicle with a display image of the scene rearward of the vehicle, said control communicating a signal associated with said distance to said display.

34. The vehicular imaging system of claim 1, wherein said imaging system is associated with a rain sensor for determining whether precipitation is present on a window of a vehicle, said rain sensor being operable in response to said output of said at least one imaging array sensor and said distance.

35. The vehicular imaging system of claim 1, wherein said imaging system is operable to control a vehicle accessory, said vehicle accessory being at least one of a display for displaying an image of the scene, a distance indicator, a warning indicator, a windshield wiper of the vehicle, headlamps of the vehicle and a brake system of the vehicle.

37. The vehicular imaging system of claim 36, wherein said control further calculates an angular position of the object with respect to said imaging system.

38. The vehicular imaging system of claim 36, wherein said first and second imaging array sensors receive a plurality of signals associated with a plurality of objects in the scene, said control classifying each of said plurality of signals as segments with respect to at least one of intensity, color, size and position on said first and second imaging array sensors and determining a distance between said imaging system and objects associated with similarly classified signals.

39. The vehicular imaging system of claim 38, wherein said first and second imaging array sensors are pixelated array sensors, each comprising a plurality of photo sensing pixels, said segments being determined by assigning a value to each of said pixels sensing an intensity greater than a predetermined intensity threshold and determining groups of adjacent pixels having the value assigned thereto.

40. The vehicular imaging system of claim 39, wherein said control determines a maximum intensity and an average pixel location for each said segment on each of said first and second imaging array sensors.

41. The vehicular imaging system of claim 40, wherein said control compares first segments on said first imaging array sensor with second segments on said second imaging array sensor, where said first and second segments have maximum intensity and pixel locations within a predetermined threshold.

42. The vehicular imaging system of claim 41, wherein said control calculates said distance to at least one of the plurality of objects according to the following equation:

43. The vehicular imaging system of claim 42, wherein said first and second reference points correspond to first and second center axes of said first and second imaging array sensors, respectively.

44. The vehicular imaging system of claim 42, wherein said control further calculates a lateral angular position of at least one of the plurality of objects with respect to said imaging system.

45. The vehicular imaging system of claim 44, wherein said control calculates said lateral angular position according to the following equation:

46. The vehicular imaging system of claim 45, wherein said imaging system is associated with a headlamp control, said headlamp control adjusting a headlamp setting in response to at least one of said distance, said angular position, said intensity and a color of the plurality of objects.

47. The vehicular imaging system of claim 46, wherein said headlamp control adjusts said headlamp setting in response to at least one of a rate of approach and a vertical position of the plurality of objects relative to the vehicle.

48. The vehicular imaging system of claim 42, wherein said imaging system is associated with at least one of a vehicular vision system and a rain sensor.

49. The vehicular imaging system of claim 36, wherein said first and second imaging array sensors are pixelated imaging array sensors, said control comparing individual pixels on said first imaging array sensor with similarly illuminated individual pixels on said second imaging array sensor.

50. The vehicular imaging system of claim 49, wherein said control calculates said distance to at least one of the plurality of objects according to the following equation:

51. The vehicular imaging system of claim 50, wherein said first and second reference points correspond to first and second center axes of said first and second imaging array sensors, respectively.

36. A vehicular imaging system comprising:

a first imaging array sensor for receiving a first image of a scene remote from said imaging system;

a second imaging array sensor for receiving a second image of the scene, said first and second imaging array sensors being positionable relative to one another and defining a separation distance therebetween;

a first optic element positionable along a first optic path between said first imaging array sensor and the scene, said first optic element having a first focal length;

a second optic element positionable along a second optic path between said second imaging array sensor and the scene, said first optic element having a second focal length; and

a control operable to determine a distance between an object in the scene and said imaging system, said control determining a first registration of said first image relative to a first reference point on said first imaging array sensor and a second registration of said second image relative to a second reference point on said second imaging array sensor, said control calculating said distance as a function of said first and second registrations, said separation distance and said first and second focal lengths.

52. A vehicular headlamp control for controlling a headlamp of a vehicle, said headlamp control comprising:

at least one imaging array sensor adaptable to receive stereoscopic images forward of the vehicle; and

a control responsive to said at least one imaging array sensor to identify light sources of interest and provide a control output to the vehicle, said control calculating a distance between at least one of the light sources of interest and said at least one imaging array sensor and providing said control output in response to said distance, said headlamp control controlling the headlamps of the vehicle in response to said control output.

53. The vehicular headlamp control of claim 52, wherein said control determines if a light source is one of a headlamp and a taillight, said control further determining if an intensity of the light source is within a threshold intensity level and if said distance is within a threshold distance range from the vehicle, said control controlling the headlamps of the vehicle in response to at least one of said intensity and said distance of the light source.

54. The vehicular headlamp control of claim 53, wherein said threshold intensity level and threshold distance level are different between a headlamp and a taillight determination.

55. The vehicular headlamp control of claim 53 further including at least one color filter associated with said at least one imaging array sensor, said control determining a color associated with the light source, said control controlling the headlamps of the vehicle in response to at least one of said color, said intensity level and said distance.

56. The vehicular headlamp control of claim 53, wherein said control determines a lateral angular position of the light source relative to the vehicle, said control controlling the headlamps of the vehicle in response to at least one of said intensity level, said distance and said angular position.

57. The vehicular headlamp control of claim 56 further including at least one color filter associated with said at least one imaging array sensor, said control determining a color associated with the light source, said control controlling the headlamps of the vehicle in response to at least one of said color, said lateral angular position, said intensity level and said distance.

58. The vehicular headlamp control of claim 57, wherein said control controls the headlamps of the vehicle in response to at least one of a rate of approach and a vertical position of the light source relative to the vehicle.

59. The vehicular headlamp control of claim 52, wherein said at least one imaging array sensor comprises a first portion and a second portion, said first portion receiving a first image of the light sources and said second portion receiving a second image of the light sources.

60. The vehicular headlamp control of claim 59 further including first and second optic elements, said first optic element having a first focal length and being positioned along a first optic path between the light sources and said first portion, said second optic element having a second focal length and being positioned along a second optic path between the light sources and said second portion.

61. The vehicular headlamp control of claim 60, wherein said first portion is associated with a first reference point and said second portion is associated with a second reference point, a separation distance being defined by said first and second reference points, said control calculating said distance as a function of said separation distance and said first and second focal lengths.

62. The vehicular headlamp control of claim 61, wherein said distance is calculated according to the following equation:

63. The vehicular headlamp control of claim 62, wherein said first reference point corresponds to a center axis of said first portion and said second reference point corresponds to a center axis of said second portion.

64. The vehicular headlamp control of claim 62, wherein said control further calculates an angular position of at least one of the light sources with respect to said headlamp control, said headlamp control controlling the headlamps of the vehicle in response to at least one of said angular position and said distance of the at least one light source relative to the vehicle.

65. The vehicular headlamp control of claim 64, wherein said headlamp control adjusts a setting of the headlamps in response to at least one of an intensity of the at least one light source, a color of the at least one light source, said distance between said headlamp control and the at least one light source, said angular position of the at least one light source relative to the vehicle, and a rate of approach of the at least one light source relative to the vehicle.

66. The vehicular headlamp control of claim 62, wherein said first image is an average position of a first portion of said first image associated with the light sources and said second image is an average position of a second portion of said second image associated with the light sources.

67. The vehicular headlamp control of claim 62, wherein said at least one imaging array sensor is a pixelated imaging array sensor, said first image being an individual pixel location associated with one of the light sources and said second image being an individual pixel location associated with the one light source.

68. The vehicular headlamp control of claim 60, wherein said headlamp control is mounted within a rear view mirror assembly of the vehicle, said at least one imaging array sensor being directed forwardly of the vehicle to receive the stereoscopic image therefrom.

69. The vehicular headlamp control of claim 68, wherein said at least one imaging array sensor is a divided imaging array sensor, said first portion being a first region of said divided imaging array sensor and said second portion being a second region of said divided imaging array sensor.

70. The vehicular headlamp control of claim 68, wherein said at least one imaging array sensor comprises first and second imaging arrays, said first portion being on said first imaging array and said second portion being on said second imaging array.

71. The vehicular headlamp control of claim 70 further including a first reflective surface positioned along and defining said first optic path and a second reflective surface positioned along and defining said second optic path.

72. The vehicular headlamp control of claim 71, wherein said first and second imaging arrays face generally opposite to one another, said first and second reflective surfaces redirecting said first and second images toward said first and second imaging arrays, respectively.

73. The vehicular headlamp control of claim 68, wherein said first and second portions are generally coplanar.

74. The vehicular headlamp control of claim 73 further including first and second reflective surfaces positioned along said first and second optic paths, respectively, said first and second reflective surfaces directing the first and second images toward said first and second portions of said at least one imaging array sensor, respectively.

75. The vehicular headlamp control of claim 52, wherein said headlamp control is mounted within a rear view mirror assembly of the vehicle, said at least one imaging array sensor being directed forwardly of the vehicle to receive the stereoscopic image therefrom.

76. The vehicular headlamp control of claim 52, wherein said at least one imaging array sensor is a pixelated imaging array sensor, said control providing sub-pixel resolution to the distance calculation.

77. A rearview vision system for a vehicle, comprising:

at least one imaging array sensor positioned on the vehicle and directed rearwardly with respect to the direction of travel of the vehicle, said at least one imaging array sensor having stereoscopic distance-sensing capability; and

a control operable to determine a distance to at least one object rearward of the vehicle in response to an output of said at least one imaging array sensor.

78. The rearview vision system of claim 77, wherein a first portion of said at least one imaging array sensor receives a first image of the at least one object and a second portion of said at least one imaging array sensor receives a second image of the at least one object.

79. The rearview vision system of claim 78 further including first and second optic elements, said first optic element having a first focal length and being positioned along a first optic path between the at least one object and said first portion, said second optic element having a second focal length and being positioned along a second optic path between the at least one object and said second portion.

80. The rearview vision system of claim 79, wherein said first portion is associated with a first reference point and said second portion is associated with a second reference point, a separation distance being defined by said first and second reference points, said control calculating said distance as a function of said separation distance and said first and second focal lengths.

81. The rearview vision system of claim 80, wherein said distance is calculated according to the following equation:

82. The rearview vision system of claim 81, wherein said first reference point corresponds to a center axis of said first portion and said second reference point corresponds to a center axis of said second portion.

83. The rearview vision system of claim 81, wherein said control further calculates an angular position of said at least one object with respect to said rearview vision system.

84. The rearview vision system of claim 81, wherein said first image is an average position of a portion of said first image associated with the at least one object and said second image is an average position of a portion of said second image associated with the at least one object.

85. The rearview vision system of claim 81, wherein said at least one imaging array sensor is a pixelated imaging array sensor, said first image being a first individual pixel location associated with the at least one object and said second image being a second individual pixel location associated with the at least one object.

86. The rearview vision system of claim 77, wherein said at least one imaging array sensor is a pixelated imaging array sensor, said control providing sub-pixel resolution to the distance calculation.

87. The rearview vision system of claim 77, wherein said at least one imaging array sensor is positioned within a rear view mirror assembly on the vehicle and directed rearwardly therefrom.

88. The rearview vision system of claim 77, wherein said at least one imaging array sensor is positioned at a rearward portion of the vehicle and directed rearwardly therefrom.

89. The rearview vision system of claim 77, wherein said at least one imaging array sensor comprises first and second imaging array sensors, said first imaging array sensor being positioned at a first side of the vehicle and said second imaging array sensor being positioned at a second side of the vehicle.

90. The rearview vision system of claim 89 further including a display system which displays an image synthesized from an output of said first and second imaging array sensors and which approximates a rearward-facing substantially seamless panoramic view from a single location substantially without duplicate redundant images of objects.

91. The rearview vision system of claim 77 further including a display system which displays an image synthesized from said output of said at least one imaging array sensor.

92. The rearview vision system of claim 91, wherein said rearview vision system is operable to produce a graphic overlay superimposed on said display.

93. The rearview vision system of claim 92, wherein said graphic overlay includes distance indicia indicating distances to the at least one object rearward of the vehicle.

94. The rearview vision system of claim 93, wherein said distance indicia comprises at least one horizontal mark.

95. The rearview vision system of claim 93, wherein said control adjusts said distance indicia in response to at least one of the vehicle'"'"'s steering system, the vehicle'"'"'s differential system and a compass.

96. The rearview vision system of claim 91, wherein said rearview vision system is operable to provide an additional enhancement to the displayed image.

97. The rearview vision system of claim 96, wherein said additional enhancement includes visually distinguishing images of objects within a threshold distance to the vehicle.

98. The rearview vision system of claim 97, wherein said additional enhancement includes at least one of displaying the objects in a particular color and flashing the image of the objects.