Method and device for visualizing a motor vehicle environment with environment-dependent fusion of an infrared image and a visual image

• US 7,199,366 B2
• Filed: 08/05/2005
• Issued: 04/03/2007
• Est. Priority Date: 02/06/2003
• Status: Active Grant

• Current Assignee: Bayerische Motoren Werke AG
• Original Assignee: BMW AG (Bayerische Motoren Werke AG)
• Inventors: Hahn, Wolfgang, Weidner, Thomas
• Primary Examiner(s): Porta; David
• Assistant Examiner(s): Sung; Christine
• Application Number:
  US11/197,264
• Publication Number:
  US 20050270784A1
• Time in Patent Office: 606 Days
• Field of Search: 250/330, 250/338.1
• US Class Current: 250/330
• CPC Class Codes: G02B 23/12 with means for image conversion or intensification
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A method is provided for visualizing the environment of a vehicle, particularly in darkness. For improving known methods, a weighted superimposition of isochronous and same-location image pairs of a visual image and an infrared image are performed for forming a fusion image. For a further improvement, it is provided to make the weighting dependent on the environment; thus, in darkness, a higher weighting and therefore a stressing of the infrared information is provided with respect to the visual information in the fusion image. In fog, it is provided to weight the visual information clearly higher in comparison to the infrared information, whereby the infrared information, which is often not very helpful in fog, is largely not taken into account in the fusion image. For implementing the method, a system is provided having a color-sensitive visual camera, an infrared camera and a fusion or superposing device, which superposes isochronous and same-location image pairs by pixels or areas and/or forms average values.

Claims

1. A method of visualizing an environment of a vehicle, the method comprising the acts of:

supplying a visual image of the environment, the visual image showing visible objects located in the environment;

supplying an infrared image of the environment, the infrared image showing infrared radiation emanating from the visible and other object; and

providing a fused image based on the visual and infrared images and at least one image adaptation parameter affecting image presentation and associated with at least one calibration, the fused image having a visual fraction, including the visible objects, and an infrared fraction, including the infrared radiation, that are superosed and weighted based on a condition of the environment.

2. The method according to claim 1, wherein providing a fused image comprises:

aligning at least one of the visual image and a normalized visual image with at least one of the infrared image and a normalized infrared image, so that substantially isochronous and same-location image pairs of the infrared and visual spectral ranges are provided, the aligning based on the at least one image adaptation parameter.

3. The method according to claim 2, wherein providing a fused image further comprises:

superposing same-location pixels of the substantially isochronous and same-location image pairs by at least one of processing associated digital data and performing an averaging of pixel information.

4. The method according to claim 3, wherein the superposing is based on at least one of brightness values and color values of the pixels or pixel areas.

5. The method according to claim 3, wherein the visual and infrared fractions are combined and weighted such that a weighted superposition takes place for one or more substantially same-location pixels from the visual image and the infrared image.

6. The method according to claim 5, wherein the weighted superposition takes place while taking into account at least one of the brightness and the visual conditions of the environment of the vehicle.

7. The method according to claim 3, wherein infrared pixels and visual pixels are weighted differently when superposing mutually assigned pixels.

8. The method according to claim 3, wherein at least one of an actual driving situation of the vehicle is determined automatically, and a driver selects an adaptation parameter appearing suitable to him from a plurality of adaptation parameters, and further wherein the adaptation parameter corresponding to the actual driving situation or the one selected, obtained by calibration and stored in the vehicle, is taken into account during the pixel and/or pixel area superposition or fusion of the visual image and of the infrared image by digital fusion image processing, whereby a largely distortion-free and/or ghost-image-free fusion image of the environment is created.

9. The method according to claim 1, wherein high-information regions are weighted higher than low-information regions of at least one of the visual fraction and the infrared fraction.

10. The method according to claim 1, wherein at least one of the weighting and the superposition of spectral fractions is manually adjusted by a user.

11. The method according to claim 1, wherein at least one of the weighting and the superposition of spectral fractions is adjusted as a function of the condition of the environment, which takes into account at least one of ecology-dependent parameters, driving-situation-dependent parameters, and output signals of sensors.

12. The method according to claim 11, wherein the adjustment to the spectral fractions takes place for an entire fused image or for a partial area of the entire fused image.

13. The method according to claim 11, wherein the adjustment depends on an actual brightness of the surroundings of the vehicle.

14. The method according to claim 11, wherein the driving-situation-dependent adjustment depends on the actual driving-dynamic quantities of the vehicle.

15. The method according to claim 14, wherein the actual driving-dynamic quantities of the vehicle include at least one of vehicle speed, vehicle acceleration, and vehicle steering angle.

16. The method according to claim 11, wherein an actual position of the vehicle is determined automatically by a vehicle satellite navigation system, and at least one of a road category and a driving situation corresponding to the position is determined.

17. The method according to claim 16, wherein the driving situation is determined automatically via at least one driving-dynamic quantity from among:

a vehicle speed, an operation of a low beam or a high beam, and a vehicle acceleration.

18. The method according to claim 1, wherein the condition of the environment is determined via a navigation system.

19. The method according to claim 1, wherein the condition of the environment is associated with at least one of:

fog, rain, a reflective road, a distance to a vehicle driving ahead, city driving, country driving, or turnpike driving.

20. The method according to claim 1, wherein the visual image is provided by a visual camera sensitive in a visual spectral range, and the infrared image is provided by an infrared camera sensitive in the infrared spectral range.

21. The method according to claim 20, wherein the visual camera has a first optical axis, and the infrared camera has a second optical axis, which axes are offset parallel to one another, so that the cameras at least partially provide first and second cutouts of the environment of the vehicle which are different from one another.

22. The method according to claim 21, wherein the first cutout and the second cutout are at least partially fused with respect to pixels and/or pixel areas by a fusion device, and further wherein during the fusion at least one distance-dependent adaptation parameter obtained during a calibration for different distances is taken into account, and the adaption parameter(s) are stored during the calibration in a data memory in the vehicle.

23. The method according to claim 22, wherein at least one distance-dependent adaptation parameter is obtained by a first calibration for a first distance, or a first distance range, and at least one other calibration for at least one other distance, or other distance range.

24. The method according to claim 23, wherein the first distance or the first distance range corresponds to a driving situation typical of city driving in a distance range of approximately 15 to 75 m.

25. The method according to claim 24, wherein a second distance or a second distance range corresponds to a driving situation typical of country road driving in a distance range of approximately 30 to 150 m.

26. The method according to claim 23, wherein a second distance or a second distance range corresponds to a driving situation typical of country road driving in a distance range of approximately 30 to 150 m.

27. The method according to claim 26, wherein a third distance or a third distance range corresponds to a driving situation typical of turnpike driving in a distance range of approximately 50 to 250 m.

28. The method according to claim 20, wherein an image repetition rate of the visual camera and of the infrared camera are at least largely identical.

29. The method according to claim 28, wherein providing at least one fused image comprises:

superposing same-location pixels or pixel areas of the largely isochronous and same-location images of the different spectral ranges by processing associated digital data or by carrying out an averaging.

30. The method according to claim 29, wherein the superposing is based on at least one of brightness values and color values of the pixels or pixel areas.

31. The method according to claim 30, wherein a weighted superposition takes place for one or more largely same-location pixels from the visual image and the infrared image.

32. The method according to claim 1, wherein the infrared radiation emanating from the at least one of the visible objects and other objects, and detected, has a wavelength in the range of approximately 7 to 14 μ

m.

33. The method according to claim 32, wherein the range is approximately 7.5–

10.5 μ

m.

34. The method according to claim 1, wherein the infrared radiation emanating from the at least one of the visible objects and other objects, and detected, has a wavelength in the range of approximately 3 μ

m to approximately 5 μ

m.

35. The method according to claim 1, wherein the infrared radiation emanating from the at least one of the visible objects and other objects, and detected, has a wavelength in the range of approximately 800 nm to approximately 2.5 μ

m.

36. The method according to claim 1, wherein the visual image of the environment of the vehicle present in the form of digital data is normalized by using a calibration device.

37. The method according to claim 32, wherein the infrared image of a cutout of the environment present in the form of digital data is normalized by using a calibration device.

38. The method according to claim 37, wherein visible radiation and infrared radiation is emitted by the calibration device.

39. The method according to claim 38, wherein the calibration device has several incandescent lamps arranged in a checkerboard-type manner.

40. The method according to claim 1, wherein the digital data of the visual image and/or the infrared image are temporarily stored in an image data memory.

41. The method according to claim 1, wherein the visual image is provided by one of a color-sensitive visual camera or a first sensor, and the infrared image is provided by one of an infrared camera or a second sensor.

42. The method according to claim 1, wherein the condition of the environment includes at least one of an ecological condition, a vehicle condition, a driving situation, and a user preference.

43. The method according to claim 1, wherein the at least one adaptation parameter is based on at least one of distance, range, ecological conditions, and driving conditions.

44. The method according to claim 1, wherein the aligning comprises a stage selected from the following:

altering the at least one of the visual image and normalized visual image to align with the at least one of the infrared image or normalized infrared image, and

altering the at least one of the infrared image and normalized infrared image to align with the at least one of the visual image or normalized visual image.

45. The method according to claim 1, wherein the visual fraction includes a plurality of individually weighted pixel areas and wherein the infrared fraction includes a plurality of individually weighted pixel areas.

46. The method according to claim 1, wherein the visual fraction is weighted higher than the infrared fraction in a foreground of the fused image and wherein the visual fraction is weighted less than the infrared fraction in a background of the fused image.

47. The method according to claim 1, wherein the visual and infrared fractions are weighted such that pixel information in the visual image is weighted differently than isochronous and same-location pixel information in the infrared fraction based on the condition of the environment.

48. A system for visualizing an environment of a vehicle, the system comprising:

a visual camera;

an infrared camera;

a first normalizing device for normalizing a visual image of a cutout of the environment of the vehicle supplied by the visual camera;

a second normalizing device for normalizing an infrared image of a cutout of the environment of the vehicle supplied by the infrared camera;

an alignment device which generates substantially isochronous and same-location image pairs from the visual and infrared images based on at least one adaptation parameter that is associated with a calibration and affects image presentation; and

a fusion device configured to generate a fused image by combining the largely isochronous and same-location image pairs, the fused image having a visual fraction and an infrared fraction that are respectively weighted based on a condition of the environment.

49. The system according to claim 48, wherein the visual camera is a color-sensitive visual camera.

50. The system according to claim 48, further comprising a calibration device, the calibration device including at least one radiation source emitting visual and infrared radiation.

51. The system according to claim 50, wherein the at least one radiation source is an incandescent lamp.

52. The system according to claim 50, wherein the calibration device includes a plurality of radiation sources arranged in a checkerboard manner.

53. The system according to claim 48, wherein the condition of the environment includes at least one of an ecological condition, a vehicle condition, a driving situation, and a user preference.