Method and device for visualizing a motor vehicle environment with environment-dependent fusion of an infrared image and a visual image

• US 20050270784A1
• Filed: 08/05/2005
• Published: 12/08/2005
• Est. Priority Date: 02/06/2003
• Status: Active Grant

• Current Assignee: Bayerische Motoren Werke AG
• Original Assignee: BMW AG (Bayerische Motoren Werke AG)
• Inventors: Hahn, Wolfgang, Weidner, Thomas
• Granted Patent: US 7,199,366 B2
• Time in Patent Office: Days
• US Class Current: 362/459
• CPC Class Codes: G02B 23/12 with means for image conversion or intensification
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A method is provided for visualizing the environment of a vehicle, particularly in darkness. For improving known methods, a weighted superimposition of isochronous and same-location image pairs of a visual image and an infrared image are performed for forming a fusion image. For a further improvement, it is provided to make the weighting dependent on the environment; thus, in darkness, a higher weighting and therefore a stressing of the infrared information is provided with respect to the visual information in the fusion image. In fog, it is provided to weight the visual information clearly higher in comparison to the infrared information, whereby the infrared information, which is often not very helpful in fog, is largely not taken into account in the fusion image. For implementing the method, a system is provided having a color-sensitive visual camera, an infrared camera and a fusion or superposing device, which superposes isochronous and same-location image pairs by pixels or areas and/or forms average values.

Claims

1. A method of visualizing an environment of a vehicle, the method comprising the acts of:

supplying a visual image of the environment, the visual image being capable of showing visible objects located in the environment; and

supplying an infrared image of the environment, the infrared image being capable of showing infrared radiation emanating from the visible and other objects.

2. The method according to claim 1, wherein the visual image or a normalized visual image is aligned with respect to the infrared image or a normalized infrared image, or vice-versa, by processing digital data of the images, so that largely isochronous and same-location image pairs of both spectral ranges are provided.

3. The method according to claim 2, wherein same-location pixels, or pixel areas, of the largely isochronous and same-location image pairs, of the different spectral ranges are mutually superposed by processing their digital data, or that an averaging takes place, and are illustrated in a fused image.

4. The method according to claim 3, wherein at least one of brightness values and color values of the pixels or pixel areas are superposed or are used for the averaging.

5. The method according to claim 3, wherein a weighted superposition or averaging takes place for one or more largely same-location pixels from the visual image and the infrared image.

6. The method according to claim 5, wherein the weighting takes place while taking into account at least one of the brightness and the visual conditions of the environment of the vehicle.

7. The method according to claim 3, wherein infrared pixels and visual pixels, or such pixel areas, are weighted differently when superposing or fusing mutually assigned pixels or pixel areas.

8. The method according to claim 3, wherein high-information regions are weighted higher than low-information regions of at least one of the visual image and the infrared image during the superposition or averaging.

9. The method according to claim 3, wherein the weighting and/or superposition of spectral fractions in the fused image is manually adjusted by a driver as a function of the environment or the driving situation.

10. The method according to claim 3, wherein the weighting and/or superposition of spectral fractions is adjusted by a superposition or fusion device as a function of the environment or driving situation, which takes into account environment-dependent or driving-situation-dependent parameters or output signals of sensors arranged on the vehicle.

11. The method according to claim 10, wherein the environment-dependent or driving-situation dependent adjustment takes place for an entire fused image or for a partial area of the entire fused image.

12. The method according to claim 10, wherein the environment-dependent or driving-situation dependent adjustment depends on an actual driving environment determined via a navigation system provided in the vehicle with processing of cartographic data.

13. The method according to claim 12, wherein the actual driving environment includes at least one of the following factors:

fog, rain, a reflective road, a distance to a vehicle driving ahead, city driving, country driving, or turnpike driving.

14. The method according to claim 10, wherein the environment-dependent or driving-situation-dependent adjustment depends on an actual brightness of the surroundings of the vehicle.

15. The method according to claim 10, wherein the driving-situation-dependent adjustment depends on the actual driving-dynamic quantities of the vehicle.

16. The method according to claim 15, wherein the actual driving-dynamic quantities of the vehicle include at least one of vehicle speed, vehicle acceleration, and vehicle steering angle.

17. The method according to claim 1, wherein the visual image is provided by a visual camera sensitive in a visual spectral range, and the infrared image is provided by an infrared camera sensitive in the infrared spectral range.

18. The method according to claim 17, wherein the visual camera has a first optical axis, and the infrared camera has a second optical axis, which axes are offset parallel to one another, so that the cameras at least partially provide first and second cutouts of the environment of the vehicle which are different from one another.

19. The method according to claim 18, wherein the first cutout and the second cutout are completely or partially superposed or fused with respect to pixels and/or pixel areas by a superposition or fusion device, and further wherein during the fusion at least one distance-dependent adaptation parameter obtained during a calibration for different distances is taken into account, and the adaptation parameter(s) are stored during the calibration in a data memory in the vehicle.

20. The method according to claim 19, wherein at least one distance-dependent adaptation parameter is obtained by a first calibration for a first distance, or a first distance range, and at least one other calibration for at least one other distance, or other distance range.

21. The method according to claim 20, wherein the first distance or the first distance range corresponds to a driving situation typical of city driving in a distance range of approximately 15 to 75 m.

22. The method according to claim 20, wherein a second distance or a second distance range corresponds to a driving situation typical of country road driving in a distance range of approximately 30 to 150 m.

23. The method according to claim 21, wherein a second distance or a second distance range corresponds to a driving situation typical of country road driving in a distance range of approximately 30 to 150 m.

24. The method according to claim 22, wherein a third distance or a third distance range corresponds to a driving situation typical of turnpike driving in a distance range of approximately 50 to 250 m.

25. The method according to claim 3, wherein at least one of an actual driving situation of the vehicle is determined automatically, and a driver selects an adaptation parameter appearing suitable to him from a plurality of adaptation parameters, and further wherein the adaptation parameter corresponding to the actual driving situation or the one selected, obtained by calibration and stored in the vehicle, is taken into account during the pixel and/or pixel area superposition or fusion of the visual image and of the infrared image by digital fusion image processing, whereby a largely distortion-free and/or ghost-image-free fusion image of the environment is created.

26. The method according to claim 10, wherein an actual position of the vehicle is determined automatically by a vehicle satellite navigation system, and a road category or driving situation corresponding to the position is determined.

27. The method according to claim 26, wherein the driving situation is determined automatically via at least one driving-dynamic quantity from among:

a vehicle speed, an operation of a low beam or a high beam, and a vehicle acceleration.

28. The method according to claim 1, wherein the infrared radiation emanating from the at least one of the visible objects and other objects, and detected, has a wavelength in the range of approximately 7 to 14 μ

m.

29. The method according to claim 28, wherein the range is approximately 7.5-10.5 μ

m.

30. The method according to claim 1, wherein the infrared radiation emanating from the at least one of the visible objects and other objects, and detected, has a wavelength in the range of approximately 3 μ

m to approximately 5 μ

m.

31. The method according to claim 1, wherein the infrared radiation emanating from the at least one of the visible objects and other objects, and detected, has a wavelength in the range of approximately 800 nm to approximately 2.5 μ

m.

32. The method according to claim 1, wherein the visual image of the environment of the vehicle present in the form of digital data is normalized by using a calibration device.

33. The method according to claim 32, wherein the infrared image of a cutout of the environment present in the form of digital data is normalized by using a calibration device.

34. The method according to claim 33, wherein visible radiation and infrared radiation is emitted by the calibration device.

35. The method according to claim 34, wherein the calibration device has several incandescent lamps arranged in a checkerboard-type manner.

36. The method according to claim 1, wherein the digital data of the visual image and/or the infrared image are temporarily stored in an image data memory.

37. The method according to claim 17, wherein an image repetition rate of the visual camera and of the infrared camera are at least largely identical.

38. The method according to claim 37, wherein same-location pixels or pixel areas of the largely isochronous and same-location images of the different spectral ranges are mutually superposed by processing their digital data or by carrying out an averaging.

39. The method according to claim 38, wherein at least one of brightness values and color values of the pixels or pixel areas are superposed or used for an averaging.

40. The method according to claim 39, wherein a weighted superposition or averaging takes place for one or more largely same-location pixels from the visual image and the infrared image.

41. The method according to claim 1, wherein the visual image is provided by one of a color-sensitive visual camera or a first sensor, and the infrared image is provided by one of an infrared camera or a second sensor.

42. A system for visualizing an environment of a vehicle, the system comprising:

a visual camera;

an infrared camera;

a first normalizing device for normalizing a visual image of a cutout of the environment of the vehicle supplied by the visual camera;

a second normalizing device for normalizing an infrared image of a cutout of the environment of the vehicle supplied by the infrared camera;

an alignment device which generates largely isochronous and same-location image pairs from the visual and infrared images; and

a fusion device, which at least one of superposes the largely isochronous and same-location image pairs by pixels or pixel areas and forms average values of the largely isochronous and same-location image pairs.

43. The system according to claim 42, wherein the visual camera is a color-sensitive visual camera.

44. The system according to claim 42, further comprising a calibration device, the calibration device including at least one radiation source emitting visual and infrared radiation.

45. The system according to claim 44, wherein the at least one radiation source is an incandescent lamp.

46. The system according to claim 44, wherein the calibration device includes a plurality of radiation sources arranged in a checkerboard manner.

47. A calibration device for a system of visualizing an environment of a vehicle, which system superposes largely isochronous and same-location image pairs from a visual camera and an infrared camera mountable on a vehicle, the calibration device comprising:

a housing; and

a plurality of incandescent lamps arranged in the housing in an arrayed manner, said lamps emitting visual and infrared radiation that is detectable by the system; and

wherein said calibration device is moveably located at different distances from the system.