Video image processing and fusion

• US 8,144,927 B2
• Filed: 01/31/2008
• Issued: 03/27/2012
• Est. Priority Date: 01/31/2008
• Status: Active Grant

• Current Assignee: Peco Limited (Astronics Corporation)
• Original Assignee: Max-Viz, Inc. (Astronics Corporation)
• Inventors: Zuro, Gregory
• Primary Examiner(s): Ahmed, Samir
• Assistant Examiner(s): RASHIDIAN, MOHAMMAD M
• Application Number:
  US12/023,851
• Publication Number:
  US 20090196457A1
• Time in Patent Office: 1,517 Days
• Field of Search: 382/100, 382103-104, 382/113, 382128-132, 382/165, 382/168, 382190-191, 382208-209, 382/240, 382/249, 382/254, 382260-264, 382/268, 382270-275, 382/284, 382296-300, 382/305, 382/38, 348/33, 348/113, 348115-118, 348/164, 348/218.1, 348/234, 348/23.1, 348280-583, 348606-607, 348625-631, 348/742, 358/451, 358/436, 358/525, 345/560, 345605-606, 345/611, 345/629, 250/201.9, 250/330, 250332-334, 250/370.06, 250/370.09, 250/372, 356/151, 356/521, 353/31, 353/85, 353/94, 353121-122, 257/440, 600/306, 707/101, 340/578
• US Class Current: 382/103
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

Processing of image data derived from radiation emanating from a scene and acquired by a multi-channel enhanced vision system renders an image of the scene for display. Detected first and second wavelength bands of radiation produce respective first and second sets of image data that include representations of relatively low contrast, high spatial frequency detail of features of the scene. Nonlinear intensity transformation of data derived from the first and second sets of image data produces, respectively, first and second sets of low dynamic range image data representing, respectively, first and second sets of intensity values. Different pairs of associated intensity values of the first and second sets correspond to different pixels forming an image. The associated intensity values of the different pairs are combined to form fused image data representing brightness levels of the pixels forming a displayed image that exhibits with high brightness and in great detail the features of the scene.

Claims

1. A method of processing image data derived from radiation emanating from a scene and acquired by a multi-channel enhanced vision system to render an image of the scene for display, the image exhibiting features of spatial regions of the scene in great detail, comprising:

detecting first and second wavelength bands of radiation emanating from a scene to produce respective first and second sets of high dynamic range image data that include representations of relatively low contrast, high spatial frequency detail of features of spatial regions of the scene;

performing nonlinear intensity transformation of data derived from the first set and second set of high dynamic range image data to produce, respectively, a first set of low dynamic range image data representing a first low dynamic range detailed image of the scene and a second set of low dynamic range image data representing a second low dynamic range detailed image of the scene, the nonlinear intensity transformation substantially preserving or enhancing in the first and second low dynamic range detailed images the relatively low contrast, high spatial frequency detail of features of the spatial regions of the scene represented in the first and second sets of high dynamic range image data; and

combining the first low dynamic range detailed image and the second low dynamic range detailed image to form a fused image that, when rendered on a display, exhibits in great detail the features of the spatial regions of the scene.

2. The method of claim 1, in which:

the first set of low dynamic range image data represents a first set of intensity values of pixels of the first low dynamic range detailed image,

the second set of low dynamic range image data represents a second set of intensity values of pixels of the second low dynamic range image,

different intensity values of the first set are paired with associated intensity values of the second set to form different pairs of associated intensity values,

the fused image includes pixels having intensity values, and

the combining to form the fused image comprises performing weighted addition of the intensity values of each of the different pairs of associated intensity values to produce the intensity values of the pixels of the fused image.

3. The method of claim 1, in which the first and second wavelength bands are at least partly non-overlapping.

4. The method of claim 3, in which a wavelength range of the first wavelength band is between about 0.7 μ

m and about 3 μ

m and a wavelength range of the second wavelength band is between about 3 μ

m and about 15 μ

m.

5. The method of claim 1, further comprising:

detecting a third wavelength band of radiation emanating from the scene to produce a third set of high dynamic range image data that includes representations of relatively low contrast, high spatial frequency detail of features of spatial regions of the scene;

performing nonlinear intensity transformation of data derived from the third set of high dynamic range image data to produce a third set of low dynamic range image data representing a third low dynamic range detailed image of the scene;

combining the first, second, and third detailed images to form the fused image.

6. The method of claim 5, further comprising:

producing first, second, and third sets of high-pass filtered image data from respectively, the first, second, and third sets of high dynamic range image data by;

blurring the first, second, and third sets of high dynamic range image data,

inverting the blurred first, second, and third sets of high dynamic range image data to form first, second, and third sets of inverted blurred high dynamic range image data, and

combining the first, second, and third sets of high dynamic range image data with, respectively, the first, second, and third sets of inverted blurred high dynamic range image data; and

performing the nonlinear intensity transformation to the first, second, and third sets of high-pass filtered image data to produce, respectively, the first, second, and third sets of low dynamic range image data.

7. The method of claim 5, in which a wavelength range of the first wavelength band is between about 0.7 μ

m and about 3 μ

m, a wavelength range of the second wavelength band is between about 3 μ

m and about 15 μ

m, and a wavelength range of the third wavelength band is between about 0.4 μ

m and about 0.7 μ

m.

8. The method of claim 1, further comprising:

producing first and second sets of high-pass filtered image data from, respectively, the first and second sets of high dynamic range image data; and

performing the nonlinear intensity transformation to the first and second sets of high-pass filtered image data to produce, respectively, the first and second sets of low dynamic range image data.

9. The method of claim 8, in which the first and second sets of high-pass filtered image data are produced by:

blurring the first and second sets of high dynamic range image data;

inverting the blurred first and second sets of high dynamic range image data to form first and second sets of inverted blurred high dynamic range image data; and

combining the first and second sets of high dynamic range image data with, respectively, the first and second sets of inverted blurred high dynamic range image data.

10. The method of claim 1, in which the nonlinear intensity transformation of the data derived from the second set of high dynamic range image data produces a set of high frequency, low dynamic range image data, the method further comprising:

applying to data derived from the second set of high dynamic range image data a gain and level adjustment to produce a set of low frequency, low dynamic range image data; and

combining the set of high frequency, low dynamic range image data with the set of low frequency, low dynamic range image data to produce the second set of low dynamic range image data that represents the second low dynamic range image.

11. The method of claim 1, in which a wavelength range of the first wavelength band is between about 0.4 μ

m and about 0.7 μ

m and a wavelength range of the second wavelength band is between about 3 μ

m and about 15 μ

m.

12. A method of processing image data derived from radiation emanating from a scene and acquired by a multi-channel enhanced vision system to render an image of the scene for display, the image exhibiting features of spatial regions of the scene with high brightness and in great detail, comprising:

detecting first and second wavelength bands of radiation emanating from a scene to produce respective first and second sets of image data that include representations of relatively low contrast, high spatial frequency detail of features of spatial regions of the scene;

performing nonlinear intensity transformation of data derived from the first set and second set of image data to produce, respectively, a first set of low dynamic range image data representing a first set of intensity values of pixels of a first image of the scene and a second set of low dynamic range image data representing a second set of intensity values of pixels of a second image of the scene;

performing distortion correction to the first set of intensity values to form corrected intensity values of the pixels of the first image, the corrected intensity values being paired with intensity values of the second set to form different pairs of associated intensity values, the distortion correction to an intensity value of the first set corresponding to a first pixel of the first image comprising;

selecting a first set of predetermined offset values that identify intensity values of the first set corresponding to a first group of adjacent pixels of the first image;

selecting a first set of predetermined weight values for the adjacent pixels in the first group;

determining a first weighted average intensity value based on the first set of weight values and the intensity values of the first set corresponding to the first group of adjacent pixels; and

assigning the first weighted average intensity value as a corrected intensity value of the first pixel; and

combining the associated intensity values of the different pairs to form fused image data representing brightness levels of pixels forming a fused image that, when rendered on a display, exhibits with high brightness and in great detail the features of the spatial regions of the scene.

13. The method of claim 12, in which the first set of image data represents the intensity values of the first set and the distortion correction is performed before the performance of nonlinear intensity transformation.

14. The method of claim 12, in which the first set of predetermined offset values and the first set of predetermined weight values are stored in and selected from a look-up table.

15. The method of claim 12, further comprising performing the distortion correction to the second set of intensity values to form corrected intensity values of the pixels of the second image, in which the distortion correction to an intensity value of the second set corresponding to a second pixel comprises:

selecting a second set of predetermined offset values that identify intensity values of the second set corresponding to a second group of adjacent pixels;

selecting a second set of predetermined weight values for the adjacent pixels in the second group;

determining a second weighted average intensity value based on the second set of weight values and the intensity values of the second set corresponding to the second group of adjacent pixels; and

assigning the second weighted average intensity value as a corrected intensity value of the second pixel.

16. A method of processing image data derived from radiation emanating from a scene and acquired by a multi-channel enhanced vision system to render an image of the scene for display, the image exhibiting features of spatial regions of the scene with high brightness and in great detail, comprising:

detecting first and second wavelength bands of radiation emanating from a scene to produce respective first and second sets of image data that include representations of relatively low contrast, high spatial frequency detail of features of spatial regions of the scene;

performing nonlinear intensity transformation of data derived from the first set and second set of image data to produce, respectively, a first set of low dynamic range image data representing a first set of intensity values and a second set of low dynamic range image data representing a second set of intensity values;

adjusting intensity values of the first set representing negative-going excursions to reduce perception of visible artifacts of the image of the scene, the adjusting producing a third set of intensity values, and different pairs of associated intensity values of the second and third sets corresponding to different pixels forming the image of the scene; and

combining the associated intensity values of the different pairs to form fused image data representing brightness levels of the pixels forming the image that, when rendered on a display, exhibits with high brightness and in great detail the features of the spatial regions of the scene.

17. A method of processing image data derived from radiation emanating from a scene and acquired by a multi-channel enhanced vision system to render an image of the scene for display, the image exhibiting features of spatial regions of the scene with high brightness and in great detail, comprising:

detecting first, second, and third wavelength bands of radiation emanating from a scene to produce respective first, second, and third sets of image data that include representations of relatively low contrast, high spatial frequency detail of features of spatial regions of the scene;

performing nonlinear intensity transformation of data derived from the first, second, and third sets of image data to produce, respectively, a first set of low dynamic range image data representing a first set of intensity values, a second set of low dynamic range image data representing a second set of intensity values, and a third set of low dynamic range image data representing a third set of intensity values;

determining an average value of the intensity values of the first set;

adjusting the intensity values of the third set according to the determined average value such that the intensity values of the third set are decreased corresponding to an increase of the average value of the first set, the adjusting producing a fourth set of intensity values, and different groups of associated intensity values of the first, second, and fourth sets corresponding to different pixels forming an image of the scene; and

combining the associated intensity values of the different groups to form fused image data representing brightness levels of the pixels forming an image that, when rendered on a display, exhibits with high brightness and in great detail the features of the spatial regions of the scene.

18. The method of claim 17, further comprising adjusting intensity values of the first and third sets representing negative-going excursions to reduce perception of visible artifacts of the image of the scene.

19. The method of claim 17, further comprising:

producing first, second, and third sets of high-pass filtered image data from respectively, the first, second, and third sets of high dynamic range image data, the first, second, and third sets of high-pass filtered image data being produced by;

blurring the first, second, and third sets of high dynamic range image data,

inverting the blurred first, second, and third sets of high dynamic range image data to form first, second, and third sets of inverted blurred high dynamic range image data, and

combining the first, second, and third sets of high dynamic range image data with, respectively, the first, second, and third sets of inverted blurred high dynamic range image data; and

performing the nonlinear intensity transformation to the first, second, and third sets of high-pass filtered image data to produce, respectively, the first, second, and third sets of low dynamic range image data.

20. The method of claim 17, in which a wavelength range of the first wavelength band is between about 0.7 μ

m and about 3 μ

m, a wavelength range of the second wavelength band is between about 3 μ

m and about 15 μ

m, and a wavelength range of the third wavelength band is between about 0.4 μ

m and about 0.7 μ

m.

21. A method of processing image data derived from radiation emanating from a scene and acquired by a multi-channel enhanced vision system to render an image of the scene for display, the image exhibiting features of spatial regions of the scene with high brightness and in great detail, comprising:

detecting first and second wavelength bands of radiation emanating from a scene to produce respective first and second sets of image data that include representations of relatively low contrast, high spatial frequency detail of features of spatial regions of the scene;

identifying a subset of the image data of the first set;

determining a peak intensity level of the first wavelength band of radiation represented in the subset;

producing a control signal representing the peak intensity level;

regulating an amount of radiation detected in the first wavelength band in response to the control signal to reduce saturation caused by image features of the scene represented in the first wavelength band of radiation;

performing nonlinear intensity transformation of data derived from the first set and second set of image data to produce, respectively, a first set of low dynamic range image data representing a first set of intensity values and a second set of low dynamic range image data representing a second set of intensity values, different pairs of associated intensity values of the first and second sets corresponding to different pixels forming an image of the scene; and

combining the associated intensity values of the different pairs to form fused image data representing brightness levels of the pixels forming an image that, when rendered on a display, exhibits with high brightness and in great detail the features of the spatial regions of the scene.

22. The method of claim 21, in which a wavelength range of the first wavelength band is between about 0.7 μ

m and about 3 μ

m.

23. A multi-channel enhanced vision system for processing image data derived from radiation emanating from a scene and acquired by the multi-channel enhanced vision system to render an image of the scene for display, comprising:

a first channel configured to detect a first wavelength band of radiation emanating from a scene and produce a first set of high dynamic range image data including representations of relatively low contrast, high spatial frequency detail of features of spatial regions of the scene, the first channel comprising;

a first conversion unit implemented with a nonlinear intensity transformation for transforming data derived from the first set of high dynamic range image data to produce a first set of low dynamic range image data representing a first low dynamic range detailed image of the scene, the first conversion unit configured to substantially preserve or enhance in the first low dynamic range detailed image the relatively low contrast, high spatial frequency detail of the features of the spatial regions of the scene represented in the first set of high dynamic range image data;

a second channel configured to detect a second wavelength band of radiation emanating from a scene and produce a second set of high dynamic range image data including representations of relatively low contrast, high spatial frequency detail of features of spatial regions of the scene, the second channel comprising;

a second conversion unit implemented with a nonlinear intensity transformation for transforming data derived from the second set of high dynamic range image data to produce a second set of low dynamic range image data representing a second low dynamic range detailed image of the scene, the second conversion unit configured to substantially preserve or enhance in the second low dynamic range detailed image the relatively low contrast, high spatial frequency detail of the features of the spatial regions of the scene represented in the second set of high dynamic range image data; and

a fusion unit configured to combine the first low dynamic range detailed image and the second low dynamic range detailed image to form a fused image that, when rendered on a display, exhibits in great detail the features of the spatial regions of the scene.

24. The system of claim 23, in which the first set of low dynamic range image data includes representations of a first set of intensity values of pixels of the first low dynamic range detailed image, and in which the second set of low dynamic range image data includes representations of a second set of intensity values of pixels of the second low dynamic range image, the system further comprising:

a first distortion correction unit configured to perform distortion correction to an intensity value of the first set, in which the first distortion correction unit is configured to identify the first set of intensity values corresponding to a first group of adjacent pixels of the first low dynamic range detailed image and determine a first weighted average of the first set of intensity values corresponding to the first group of adjacent pixels; and

a second distortion correction unit configured to perform distortion correction to an intensity value of the second set, in which the second distortion correction unit is configured to identify the second set of intensity values corresponding to a second group of adjacent pixels of the second low dynamic range detailed image and determine a second weighted average of the second set of intensity values corresponding to the second group of adjacent pixels.

25. The system of claim 23, further comprising:

a third channel configured to detect a third wavelength band of radiation emanating from a scene to produce a third set of high dynamic range image data including representations of relatively low contrast, high spatial frequency detail of features of spatial regions of the scene, the third wavelength channel comprising a third conversion unit implemented with a nonlinear intensity transformation for transforming data derived from the third set of high dynamic range image data to produce a third set of low dynamic range image data representing a third low dynamic range detailed image of the scene, the fusion unit configured to combine the first, second, and third detailed images to form the fused image.