Method and system for visualizing the environment of a vehicle with a distance-dependent merging of an infrared and a visual image

• US 20050110621A1
• Filed: 12/17/2004
• Published: 05/26/2005
• Est. Priority Date: 06/18/2002
• Status: Active Grant

• Current Assignee: Bayerische Motoren Werke AG
• Original Assignee: Bayerische Motoren Werke AG
• Inventors: Hahn, Wolfgang, Weidner, Thomas
• Granted Patent: US 7,834,905 B2
• Time in Patent Office: Days
• US Class Current: 340/435
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/24 in front of the vehicle
• CPC Class Codes: B60R 1/30 providing vision in the non-visible spectrum, e.g. night or infrared vision
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/106 using night vision cameras
• CPC Class Codes: B60R 2300/301 combining image information with other obstacle sensor information, e.g. using RADAR/LIDAR/SONAR sensors for estimating risk of collision
• CPC Class Codes: B60R 2300/302 combining image information with GPS information or vehicle data, e.g. vehicle speed, gyro, steering angle data
• CPC Class Codes: B60R 2300/304 using merged images, e.g. merging camera image with stored images
• CPC Class Codes: B60R 2300/402 Image calibration
• CPC Class Codes: B60R 2300/8053 for bad weather conditions or night vision
• CPC Class Codes: G02B 23/12 with means for image conversion or intensification
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

The invention relates to an improved method for visualizing the environment of a vehicle, especially in the dark. The invention also relates to a night vision system, which especially provides a visual image of the environment or the digital data thereof. Preferably, the visual image is a color image which indicates the visually perceptible objects of the environment. The system also provides an infrared image of the environment or the digital data thereof. The infrared image indicates the infrared radiation radiated by the visually perceptible and/or other objections. In a preferred form of embodiment, a merged image of the visual image and the infrared image of largely identical sections of the environment of the vehicle is represented on a display comprising at least one merged region and at least one region which is not merged or not merged to the same extent or not merged with the same weighting.

Claims

1. Method of visualizing an environment of a vehicle in darkness, the method comprising the acts of:

providing a visual image or its digital data of the environment, preferably a colored visual image, the visual image showing visible objects; and

providing an infrared image or its digital data of the environment, the infrared image showing the infrared radiation emanating from the visible and/or other objects.

2. Method according to claim 1, wherein the visual image or its digital data are provided by a visual camera sensitive in a visual spectral range, preferably a color-sensitive visual camera, or such a first sensor, and the infrared image or its digital data are provided by an infrared camera sensitive in the infrared spectral range or such a second sensor.

3. Method according to claim 2, wherein the visual camera or the first sensor or its lens system has a first optical axis, and the infrared camera or the second optical sensor or its lens system has a second optical axis, which are offset parallel to one another so that the cameras or sensors provide at least partially different cutouts of the environment of the vehicle in the form of a first and a second cutout.

4. Method according to claim 3, wherein the provided first cutout and the provided second cutout are completely or partially superposed or merged by a superposing or merging device with respect to pixels and/or areas, and wherein during the merging at least one distance-dependent adaptation parameter obtained during calibration for different distances, such as particularly at least one registration or transformation parameter, is taken into account, and the adaptation parameter or parameters are preferably stored during the calibration in a data memory in the vehicle.

5. Method according to claim 4, wherein the at least one distance-dependent adaptation parameter is obtained by a first calibration for a first distance or a first distance range and at least one additional calibration for at least one other distance or one other distance range.

6. Method according to claim 1, further comprising the act of:

merging or superposing the visual image and the infrared image, during which a merged image is created which has at least a first area which shows a portion of the visual image and a portion of the infrared image, and has at least a second area which is formed by the merging of another portion of the visual image and of the corresponding portion of the infrared image.

7. Method according to claim 1, further comprising the act of:

merging or superposing the visual image and the infrared image during which a merged image is created which has at least a first merged image area and a second merged image area, the first merged image area being formed by using at least a first distance-dependent adaptation parameter, and the second merged image area is formed by using at least a second distance-dependent adaptation parameter.

8. Method according to claim 6, wherein the distance between the vehicle and a vehicle driving in front of it is determined and it is checked whether the actually used distance-dependent adaptation parameter is suitable for the determined distance for providing a ghost-image-free merged image and, in the event of a lacking suitability, only the corresponding portion of the visual image or of the infrared image is partially illustrated in the merged image.

9. Method according to claim 7, wherein the distance between the vehicle and a vehicle driving in front of it is determined and it is checked whether the actually used distance-dependent adaptation parameter is suitable for the determined distance for providing a ghost-image-free merged image and, in the event of a lacking suitability, at least one additional suitable distance-dependent adaptation parameter is determined and the latter is used for providing a partially merged image which shows the vehicle driving ahead.

10. Method according to claim 5, wherein the first distance or the first distance range corresponds to a driving situation typical of city driving, such as particularly a distance range of from approximately 15 to 75 m.

11. Method according to claim 5, wherein a second distance range corresponds to a driving situation typical of highway driving, such as particularly a distance range of from approximately 30 to 150 m.

12. Method according to claim 10, wherein a second distance range corresponds to a driving situation typical of highway driving, such as particularly a distance range of from approximately 30 to 150 m.

13. Method according to claim 5, wherein a third distance or a third distance range corresponds to a driving situation typical of expressway driving, such as particularly a distance range of from approximately 50 to 250 m.

14. Method according to claim 10, wherein a third distance or a third distance range corresponds to a driving situation typical of expressway driving, such as particularly a distance range of from approximately 50 to 250 m.

15. Method according to claim 11, wherein a third distance or a third distance range corresponds to a driving situation typical of expressway driving, such as particularly a distance range of from approximately 50 to 250 m.

16. Method according to claim 12, wherein a third distance or a third distance range corresponds to a driving situation typical of expressway driving, such as particularly a distance range of from approximately 50 to 250 m.

17. Method according to claim 1, wherein the actual driving situation of the vehicle is determined automatically and/or the driver selects an adaptation parameter, which seems suitable to him, from a plurality of adaptation parameters, and the adaptation parameter, which corresponds to the actual driving situation or is selected, is obtained by calibration and stored in the vehicle, is taken into account by the digital merged image processing during the pixel-type and/or area-type superposing or merging of the visual image and of the infrared image, whereby a largely distortion-free and/or ghost-image-free merged image of the environment is obtained.

18. Method according to claim 17, wherein the actual position of the vehicle is automatically determined by a vehicle navigation system, particularly a satellite navigation system, and the street/road category or driving situation corresponding to the position, particularly the city street, highway or expressway, is determined.

19. Method according to claim 17, wherein the driving situation is automatically determined by using at least one quantity of driving dynamics, such as particularly the vehicle speed and/or the operation of the low beam or high beam, and/or the vehicle acceleration.

20. Method according to claim 18, wherein the driving situation is automatically determined by using at least one quantity of driving dynamics, such as particularly the vehicle speed and/or the operation of the low beam or high beam, and/or the vehicle acceleration.

21. Method according to claim 1, wherein infrared radiation emanating from the visible objects and/or the additional objects, and detected, has a wavelength in the range of from approximately 7 to 14 mm, preferably approximately 7.5-10.5 mm.

22. Method according to claim 1, wherein infrared radiation emanating from the visible objects and/or the additional objects, and detected, has a wavelength in the range of from approximately 3 mm to approximately 5 mm.

23. Method according to claim 1, wherein infrared radiation emanating from the visible objects and/or the additional objects and detected has a wavelength in the range of from approximately 800 nm to approximately 2.5 mm.

24. Method according to claim 1, wherein the visual image of the environment of the vehicle present in the form of digital data is normalized by using a calibrating device.

25. Method according to claim 1, wherein infrared image of the cutout of the environment present in the form of digital data is normalized by using the calibrating device.

26. Method according to claim 24, wherein the calibrating device emits visible radiation and infrared radiation.

27. Method according to claim 25, wherein the calibrating device emits visible radiation and infrared radiation.

28. Method according to claim 24, wherein the calibrating device has several incandescent lamps arranged in a checkerboard-type manner.

29. Method according to claim 25, wherein the calibrating device has several incandescent lamps arranged in a checkerboard-type manner.

30. Method according to claim 26, wherein the calibrating device has several incandescent lamps arranged in a checkerboard-type manner.

31. Method according to claim 1, wherein the visual image or the normalized visual image is aligned with respect to the infrared image or the normalized visual image or vice-versa by the processing of digital data of the images, so that image pairs of both spectral ranges are provided which are identical with respect to time and location.

32. Method according to claim 1, wherein the digital data of the visual image and/or of the infrared image are temporarily stored in an image data memory.

33. Method according to claim 1, wherein an image repetition rate of the visual camera or of the first sensor and of the infrared camera or of the second sensor are at least largely identical.

34. Method according to claim 31, wherein same-location pixels or pixel areas of the images of the different spectral ranges which are largely identical with respect to time and location are superposed by the processing of their digital data, or that an averaging takes place.

35. Method according to claim 34, wherein brightness values and/or color values of the pixels or pixel areas are superposed or are used for an averaging.

36. Method according to claim 31, wherein a weighted superposition or averaging is carried out for one or more pixels largely having the same location from the visual image and the infrared image.

37. Method according to claim 36, wherein the weighting takes place taking into account the brightness and/or the visual condition in the environment of the vehicle.

38. Method according to claim 24, wherein infrared pixels and visual pixels or such pixel areas are weighted differently.

39. Method according to claim 25, wherein infrared pixels and visual pixels or such pixel areas are weighted differently.

40. Method according to claim 34, wherein regions with large amounts of information in comparison to regions with small amounts of information of the visual image and/or of the infrared image are weighted higher during the superposition or averaging.

41. Method according to claim 35, wherein regions with large amounts of information in comparison to regions with small amounts of information of the visual image and/or of the infrared image are weighted higher during the superposition or averaging.

42. Method according to claim 36, wherein regions with large amounts of information in comparison to regions with small amounts of information of the visual image and/or of the infrared image are weighted higher during the superposition or averaging.

43. Method according to claim 37, wherein regions with large amounts of information in comparison to regions with small amounts of information of the visual image and/or of the infrared image are weighted higher during the superposition or averaging.

44. Method according to claim 38, wherein regions with large amounts of information in comparison to regions with small amounts of information of the visual image and/or of the infrared image are weighted higher during the superposition or averaging.

45. A system for visualizing an environment of a vehicle in darkness, wherein the system implements the method according to claim 1.

46. The system according to claim 45, wherein a color visual camera, an infrared camera, a first normalization device for normalizing a colored visual image of a cutout of the environment of the vehicle provided by the color visual camera, a second normalization device for normalizing the infrared image of the cutout of the environment of the vehicle provided by the infrared camera, an aligning device for generating image pairs identical with respect to time and location from visual images and infrared images, as well as a merging or superposing device which superposes the image pairs largely identical with respect to time and location in a pixel-type or area-type manner and/or forms average values.

47. A calibrating device for calibrating a system according to claim 45, the calibration device having at least one radiation source, which source emits visible radiation as well as infrared radiation.

48. A calibrating device for calibrating a system according to claim 46, the calibration device having at least one radiation source, which source emits visible radiation as well as infrared radiation.

49. The calibration device according to claim 47, wherein the radiation source is an incandescent lamp.

50. The calibration device according to claim 48, wherein the radiation source is an incandescent lamp.