Image fusion system and method

• US 20040047518A1
• Filed: 08/28/2002
• Published: 03/11/2004
• Est. Priority Date: 08/28/2002
• Status: Active Grant

• Current Assignee: BAE Systems Controls Incorporated (BAE Systems Plc)
• Original Assignee: BAE Systems Aircraft Controls Inc.
• Inventors: Tiana, Carlo
• Granted Patent: US 6,898,331 B2
• Time in Patent Office: Days
• US Class Current: 382/284
• CPC Class Codes: G06T 15/503 Blending, e.g. for anti-aliasing
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction

Abstract

A contrast-based image fusion system and method of processing multiple images to form a processed or fused image including regions selected from one or more images. Images are divided into image regions. Portions of the images are filtered if necessary. A contrast map is generated for each image via a convolution kernel resulting in a contrast map with contrast values for each image region. Contrast values are compared and image regions are selected based on a selection criteria or process such as greater or maximum contrast. The selected image regions form the fused image. If necessary, the luminance of one or more portions of the fused image is adjusted. One sensor is selected as a reference sensor, and an average intensity of each region of the reference sensor image is determined across the reference sensor image. The intensity of one or more regions in the final image is adjusted by combining the determined average luminance values and intensity values of the final image.

Claims

1. A method of forming a processed image using a plurality of images, each image generated by a respective sensor, comprising:

dividing each image into a plurality of image regions;

generating a contrast map for each image, each contrast map including a contrast value for each image region;

applying a selection process to said contrast value for selecting an image region for use in said processed image; and

forming said processed image with the selected image regions.

2. The method of claim 1, wherein dividing the images into the plurality of image regions further comprises dividing each image on a pixel-by-pixel basis, into blocks of pixels, or into arbitrary shaped regions.

3. The method of claim 1, wherein generating the contrast map further comprises performing a convolution to determine the contrast value of the contrast map.

4. The method of claim 3, wherein performing the convolution further comprises performing the convolution with a Kernel K_c, wherein

[{K_c*S1(x,y), K_c*S2(x,y)}] represents the convolution;

$K_c=<br><br>\begin{array}{ccc}-\frac{1}{2<br><br>\sqrt{2}}& -\frac{1}{2}& -\frac{1}{2<br><br>\sqrt{2}}\\ -\frac{1}{2}& 2<br><br>\frac{1+\sqrt{2}}{\sqrt{2}}& -\frac{1}{2}\\ -\frac{1}{2<br><br>\sqrt{2}}& -\frac{1}{2}& -\frac{1}{2<br><br>\sqrt{2}}\end{array}<br><br>\cong <br><br><br><br>\begin{array}{ccc}-0.354& -0.500& -0.354\\ -0.500& 3.414& -0.500\\ -0.354& -0.500& -0.354\end{array}<br><br>$

S1 represents image regions of a first image;

S2 represents image regions of a second image; and

(x,y) represent spatial coordinates of the images.

5. The method of claim 1, wherein each sensor detects a different wavelength.

6. The method of claim 1, wherein the plurality of sensors includes an infrared (IR) sensor and a radar sensor.

7. The method of claim 1, wherein the plurality of sensors includes an infrared (IR) and an ultraviolet (UV) sensor.

8. The method of claim 1, wherein the plurality of sensors includes a radar sensor and an ultraviolet (UV) sensor.

9. The method of claim 1, wherein the plurality of images are generated by two or more infrared (IR) sensors, each IR sensor detecting a different wavelength.

10. The method of claim 1, wherein applying the selection process includes comparing competing contrast values of two corresponding image regions from two respective images.

11. The method of claim 10, wherein said selection process operates to select the greater of the competing contrast values.

12. The method of claim 10, wherein comparing competing contrast values further comprises comparing corresponding contrast values of overlapping image regions.

13. The method of claim 1, wherein contrast values of contrast maps of respective first, second, and third sensors are compared together to form said processed image.

14. The method of claim 13, further comprising

identifying contrast values from first and second sensor images to form an intermediate contrast map;

applying a selection process to the contrast values of the intermediate contrast map and contrast values of a contrast map of a third sensor image; and

forming said processed image with the selected image regions.

15. The method of claim 14, wherein the first and second sensors are infrared (IR) sensors and the third sensor is a radar sensor.

16. The method of claim 1, wherein a sensor image displays a view from a moving vehicle.

17. The method of claim 16, wherein the moving vehicle is an aircraft, a watercraft, an automobile, or a train.

18. The method of claim 1, further comprising adjusting an intensity of one or more regions of said processed image.

19. The method of claim 18, further comprising weighting the degree of intensity adjustment.

20. The method of claim 18, wherein adjusting the intensity further comprises adjusting the intensity across said processed image.

21. The method of claim 18, wherein adjusting the intensity across said processed image further comprises:

selecting one sensor as a reference sensor;

determining an average intensity of each region of the reference sensor image; and

adjusting the intensity of one or more regions in said processed image by combining the determined average luminance values and intensity values of said processed image.

22. The method of claim 21, wherein the sensors include a radar sensor and an infrared (IR) sensor, and wherein the reference sensor comprises the radar sensor.

23. The method of claim 21, wherein the sensors include a radar sensor and an infrared (IR) sensor, and wherein the reference sensor comprises the infrared (IR) sensor.

24. The method of claim 21, wherein adjusting the intensity of one or more regions in said processed image further comprises adjusting the intensity of a line in said processed image corresponding to a line in the reference sensor image for which the average intensity was determined.

25. The method of claim 24, wherein adjusting the intensity of one or more lines in said processed image further comprises adjusting the intensity of a line in said processed image that is adjacent to a line in said processed image corresponding to the same line in the reference sensor image for which the average intensity was determined.

26. The method of claim 21, wherein a scene horizon is repositioned at an angle relative to an image horizon, further comprising

determining an average intensity of the reference sensor image on a pixel-by-pixel basis;

and

adjusting the intensity of said processed image on a pixel-by-pixel basis.

27. The method of claim 26, wherein the scene horizon is repositioned due to roll, bank, yaw or pitch motions.

28. The method of claim 1, before generating the contrast map for each image, further comprising filtering regions of one or more images.

29. The method of claim 28, wherein filtering further comprises spatially filtering regions of each image by weighting selected image regions.

30. The method of claim 29, wherein one sensor comprises a radar sensor, and wherein spatial filtering is performed by filtering image regions above a radar horizon.

31. A method of forming a processed image using a plurality of images, each image generated by a respective sensor to form a processed image, comprising:

filtering portions of one or more images;

comparing contrast values of the images by

dividing each image into a plurality of image regions,

generating a contrast map for each image, each contrast map including contrast values for each image region of each image,

comparing contrast values in each contrast map of the image regions,

identifying maximum contrast values based on the comparison of contrast values, and

selecting image regions corresponding to the maximum contrast values,

forming the processed image with the selected image regions; and

adjusting an intensity of one or more portions of the processed image by

selecting one sensor as a reference sensor,

determining an average intensity of one or more regions of the reference sensor image across the reference sensor image, and

adjusting the intensity of one or more regions in the processed image by combining the determined average luminance values and intensity values of the processed image.

32. A system for combining a plurality of images to form a final image, comprising:

a plurality of sensors that generate respective images;

a processor configured to

divide each image into a plurality of image regions,

generate a contrast map for each image, each contrast map including a contrast value for each image region,

apply a selection criterion to said contrast value for selecting an image region for use in said processed image, and

form said processed image with the selected image regions.

33. The system of claim 32, wherein the processor is configured to divide each image into individual pixels, into blocks of pixels, or into arbitrary shaped regions.

34. The system of claim 32, wherein the processor is configured to generate the contrast map by performing a convolution to determine the contrast value of the contrast map.

35. The system of claim 34, wherein the processor is configured to perform the convolution with a Kernel K_c, wherein [{K_c*S1 (x,y), K_c*S2 (x,y)}] represents the convolution;

36. The system of claim 32, wherein each sensor detects a different wavelength.

37. The system of claim 32, wherein the plurality of sensors includes an infrared (IR) sensor and a radar sensor.

38. The system of claim 32, wherein the plurality of sensors includes an infrared (IR) and an ultraviolet (UV) sensor.

39. The system of claim 32, wherein the plurality of sensors includes a radar sensor and an ultraviolet (UV) sensor.

40. The system of claim 32, wherein the plurality of sensors includes two or more infrared (IR) sensors, each IR sensor detecting a different wavelength.

41. The system of claim 32, wherein the processor is further configured to compare competing contrast values of two corresponding image regions from two respective images.

42. The system of claim 41, wherein the processor is further configured to select the greater of the competing contrast values.

43. The system of claim 41, wherein the processor is configured to compare corresponding contrast values of overlapping image regions.

44. The system of claim 32, wherein contrast values of contrast maps of a respective first, second, and third sensors are compared together to form the final image.

45. The system of claim 44, wherein the processor is further configured to

identify contrast values from first and second sensor images to form an intermediate contrast map,

apply a selection process to the contrast values of the intermediate contrast map and contrast values of a contrast map of a third sensor image, and

forming said processed image with the selected image regions.

46. The system of claim 45, wherein the first and second sensors are infrared (IR) sensors and the third sensor is a radar sensor.

47. The system of claim 32, wherein a sensor image displays a view from a moving vehicle.

48. The system of claim 47, wherein the moving vehicle comprises an aircraft, a watercraft, an automobile, or a train.

49. The system of claim 32, wherein the processor is further configured to adjust an intensity of one or more regions of said processed image.

50. The system of claim 49, wherein the processor is configured to adjust the intensity across said processed image.

51. The system of claim 50, wherein the processor is configured to weight the degree of intensity adjustment.

52. The system of claim 50, wherein the processor is further configured to

select one sensor as a reference sensor,

determine an average intensity of each region of the reference sensor image across the reference sensor image, and

adjust the intensity of one or more regions in said processed image by combining the determined average luminance values and intensity values of said processed image.

53. The system of claim 52, wherein the sensors include a radar sensor and an infrared (IR) sensor, and wherein the reference sensor comprises the radar sensor.

54. The system of claim 52, wherein the sensors include a radar sensor and an infrared (IR) sensor, and wherein the reference sensor comprises the infrared (IR) sensor.

55. The system of claim 52, wherein the processor is configured to adjust the intensity of a line in said processed image corresponding to a line in the reference sensor image for which the average intensity was determined.

56. The system of claim 52, wherein the processor is configured to adjust the intensity of a line in said processed image that is adjacent to a line in said processed image corresponding to the same line in the reference sensor image for which the average intensity was determined.

57. The system of claim 52, wherein a scene horizon is repositioned at an angle relative to an image horizon, the processor being further configured to

determine an average intensity of the reference sensor image on a pixel-by-pixel basis, and

adjust the intensity of said processed image on the pixel-by-pixel basis.

58. The system of claim 57, wherein the scene horizon is repositioned relative to the image horizon due to roll, bank, pitch or yaw motions.

59. The system of claim 32, wherein the processor is configured to filter one or more image regions.

60. The system of claim 59, wherein the processor is configured to filter the one or more image regions by weighting selected image regions.

61. The system of claim 59, wherein one sensor comprises a radar sensor, and wherein the processor is further configured to spatially filter image regions above a radar horizon.

62. A system for forming a processed image using a plurality of images, comprising:

a first sensor that generates a first image;

a second sensor that generates a second image,

wherein the first and second images are divided into a plurality of image regions;

a processor configured to

filter one or more portions of one or more images;

compare contrast values of the images by

dividing each image into a plurality of image regions,

generating a contrast map for each image, each contrast map including contrast values for each image region of each image,

comparing contrast values in each contrast map of the image regions,

identifying maximum contrast values,

selecting image regions corresponding to the maximum contrast values, and

forming the processed image with the selected image regions; and

adjust an intensity of one or more regions of the final image by

selecting one sensor as a reference sensor,

determining an average intensity of each region of the reference sensor image across the reference sensor image, and

adjusting the intensity of one or more regions in said processed image by combining the determined average luminance values and intensity values of the final image.