MULTI-SCALE MULTI-CAMERA ADAPTIVE FUSION WITH CONTRAST NORMALIZATION

• US 20090169102A1
• Filed: 11/21/2008
• Published: 07/02/2009
• Est. Priority Date: 11/29/2007
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: van der Wal, Gooitzen Sieman, Burt, Peter Jeffrey, Zhang, Chao
• Granted Patent: US 8,411,938 B2
• Time in Patent Office: Days
• US Class Current: 382/167
• CPC Class Codes: G06T 2200/28 involving image processing hardware
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 3/4061 by injecting details from different spectral ranges
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/90 Dynamic range modification of images or parts thereof
• CPC Class Codes: H04N 25/672 between adjacent sensors or output registers for reading a single image

Abstract

A computer implemented method for fusing images taken by a plurality of cameras is disclosed, comprising the steps of: receiving a plurality of images of the same scene taken by the plurality of cameras; generating Laplacian pyramid images for each source image of the plurality of images; applying contrast normalization to the Laplacian pyramids images; performing pixel-level fusion on the Laplacian pyramid images based on a local salience measure that reduces aliasing artifacts to produce one salience-selected Laplacian pyramid image for each pyramid level; and combining the salience-selected Laplacian pyramid images into a fused image. Applying contrast normalization further comprises, for each Laplacian image at a given level: obtaining an energy image from the Laplacian image; determining a gain factor that is based on at least the energy image and a target contrast; and multiplying the Laplacian image by a gain factor to produce a normalized Laplacian image.

Claims

1. A computer implemented method for fusing images taken by a plurality of cameras, comprising the steps of:

receiving a plurality of images of the same scene taken by the plurality of cameras;

generating Laplacian pyramid images for each source image of the plurality of images;

applying contrast normalization to the Laplacian pyramids images;

performing pixel-level fusion on the Laplacian pyramid images based on a local salience measure that reduces aliasing artifacts to produce one salience-selected Laplacian pyramid image for each pyramid level; and

combining the salience-selected Laplacian pyramid images into a fused image.

2. The method of claim 1, wherein the step of performing pixel-level fusion further comprises the steps of, for each level of the Laplacian pyramid images corresponding to each source image of the plurality of images:

passing a Laplacian pyramid image through a filter which average each pixel with surrounding pixels;

convolving the filtered Laplacian pyramid image with the absolute value of the Laplacian pyramid image to produce an energy image;

generating a selection mask wherein a pixel at a given location is selected based on comparing the energy of the pixels originating from energy images corresponding each of the source images; and

multiplying the selection mask by a filter to reduce aliasing artifacts by smoothing the contributions to the selection mask from each of the energy images to produce a contribution image.

3. The method of claim 2, further comprising the step of summing contribution images corresponding to each of the source images to produce the salience-selected Laplacian pyramid image at one level of a Laplacian pyramid.

4. The method of claim 2, wherein the pixel that is selected at a given location is the pixel among the energy images corresponding to each source image which has the highest energy when only one pixel has the highest energy;

otherwise, partial contributions from each of the pixels from each of the sources which have about the same energy are selected.

5. The method of claim 2, wherein the pixel that is selected at a given location is the pixel among the energy images corresponding to each source image which has the highest energy.

6. The method of claim 2, further comprising the step of multiplying the energy image with a region of interest (ROI) mask to remove a static overlay in a source image or remove the border induced by warping or by a different source size.

7. The method of claim 4, further comprising the step of incrementing a selected pixel by a hysteresis factor to reduce flickering artifacts.

8. The method of claim 1, wherein the step of generating Laplacian pyramid images further comprises the steps of:

generating a Gaussian pyramid for at least one of the plurality of images; and

applying noise coring to at least the Level 0 Gaussian image of the Gaussian pyramid to produce a noise-cored Laplacian image.

9. The method of claim 8, wherein said step of applying noise coring further comprising the steps of:

processing at least the Level 0 Gaussian image substantially simultaneously through a plurality of derivative (gradient) filters to produce a first set of gradient images;

applying a noise coring function to each of the gradient images;

processing the noise cored gradient images substantially simultaneously through a second set of derivative (gradient) filters; and

negatively summing the resulting filtered/cored images to produce a the noise-cored Laplacian image.

10. The method of claim 1, wherein the step of applying contrast normalization further comprises the steps of, for each Laplacian image at a given level:

obtaining an energy image from the Laplacian image;

determining a gain factor that is based on at least the energy image and a target contrast; and

multiplying the Laplacian image by a gain factor to produce a normalized Laplacian image.

11. The method of claim 10, wherein the target contrast is halved with increasing pyramid level.

12. The method of claim 10, wherein the gain factor undergoes noise coring.

13. The method of claim 11, wherein the degree of noise coring is halved with increasing pyramid level.

14. The method of claim 10, wherein a saturation mask is applied to the gain factor.

15. The method of claim 1, further comprising the step of generating saturation and ROI mask pyramids.

16. The method of claim 1, further comprising the step of applying a non-uniformity correction (NUC) to each of the plurality of images to produce corrected images.

17. The method of claim 16, further comprising the step of applying at least one of linear stretching, uniform histogram equalization, Rayleigh shape histogram equalization, Gaussian shape histogram equalization, and gamma correction to each of the plurality of images to produce preprocessed images.

18. The method of claim 17, further comprising the step of warping the preprocessed images to align the preprocessed images to each other.

19. The method of claim 1, wherein one of the plurality of images has a modality which encompasses at least a portion of the visible range of frequencies.

20. The method of claim 19, wherein the one of the plurality of images is separated into a luminance component and chrominance components, wherein the luminance component is contrast normalized and undergoes image fusion, and the chrominance component is combined with the fused image to produce a color enhanced fused image.

21. The method of claim 20, further comprising the steps of:

applying a luminance damping function to the luminance component; and

applying a gamma correction to the chrominance components.

22. The method of claim 21, further comprising the step of orthogonalizing the chrominance components.

23. An apparatus for fusing images taken by a plurality of cameras, comprising:

a processor configured for executing instructions comprising the steps of;

receiving a plurality of images of the same scene taken by the plurality of cameras;

generating Laplacian pyramid images for each source image of the plurality of images;

applying contrast normalization to the Laplacian pyramids images;

performing pixel-level fusion on the Laplacian pyramid images based on a local salience measure that reduces aliasing artifacts to produce one salience-selected Laplacian pyramid image for each pyramid level; and

combining the salience-selected Laplacian pyramid images into a fused image.

24. The apparatus of claim 23, wherein performing pixel-level fusion further comprises, for each level of the Laplacian pyramid images corresponding to each source image of the plurality of images:

passing a Laplacian pyramid image through a filter which average each pixel with surrounding pixels;

convolving the filtered Laplacian pyramid image with the absolute value of the Laplacian pyramid image to produce an energy image;

generating a selection mask wherein a pixel at a given location is selected based on comparing the energy of the pixels originating from energy images corresponding each of the source images; and

multiplying the selection mask by a filter to reduce aliasing artifacts by smoothing the contributions to the selection mask from each of the energy images to produce a contribution image.

25. The apparatus of claim 24, wherein the processor is further configured for executing instructions comprising the step of summing contribution images corresponding to each of the source images to produce the salience-selected Laplacian pyramid image at one level of a Laplacian pyramid.

26. The apparatus of claim 24, wherein the pixel that is selected at a given location is the pixel among the energy images corresponding to each source image which has the highest energy when only one pixel has the highest energy;

otherwise, partial contributions from each of the pixels from each of the sources which have about the same energy are selected.

27. The apparatus of claim 26, wherein the processor is further configured for executing instructions comprising the step of incrementing a selected pixel by a hysteresis factor to reduce flickering artifacts.

28. The apparatus of claim 23, wherein applying contrast normalization further comprises, for each Laplacian image at a given level:

obtaining an energy image from the Laplacian image;

determining a gain factor that is based on at least the energy image and a target contrast; and

multiplying the Laplacian image by a gain factor to produce a normalized Laplacian image.

29. The apparatus of claim 23, wherein the one of the plurality of images is separated into a luminance component and chrominance components, wherein the luminance component is contrast normalized and undergoes image fusion, and the chrominance component is combined with the fused image to produce a color enhanced fused image.

30. The apparatus of claim 29, wherein the processor is further configured for executing instructions comprising the steps of:

applying a luminance damping function to the luminance component; and

applying a gamma correction to the chrominance components.

31. A computer-readable medium carrying one or more sequences for fusing images taken by a plurality of cameras, wherein execution of the one of more sequences of instructions by one or more processors causes the one or more processors to perform the steps comprising:

receiving a plurality of images of the same scene taken by the plurality of cameras;

generating Laplacian pyramid images for each source image of the plurality of images;

applying contrast normalization to the Laplacian pyramids images;

performing pixel-level fusion on the Laplacian pyramid images based on a local salience measure that reduces aliasing artifacts to produce one salience-selected Laplacian pyramid image for each pyramid level; and

combining the salience-selected Laplacian pyramid images into a fused image.

32. The computer readable medium of claim 31, wherein the step of performing pixel-level fusion further comprises the steps of, for each level of the Laplacian pyramid images corresponding to each source image of the plurality of images:

passing a Laplacian pyramid image through a filter which average each pixel with surrounding pixels;

convolving the filtered Laplacian pyramid image with the absolute value of the Laplacian pyramid image to produce an energy image;

generating a selection mask wherein a pixel at a given location is selected based on comparing the energy of the pixels originating from energy images corresponding each of the source images; and

multiplying the selection mask by a filter to reduce aliasing artifacts by smoothing the contributions to the selection mask from each of the energy images to produce a contribution image.

33. The computer readable medium of claim 32, wherein the one or more processors are further configured to perform the step comprising summing contribution images corresponding to each of the source images to produce the salience-selected Laplacian pyramid image at one level of a Laplacian pyramid.

34. The computer readable medium of claim 32, wherein the pixel that is selected at a given location is the pixel among the energy images corresponding to each source image which has the highest energy when only one pixel has the highest energy;

otherwise, partial contributions from each of the pixels from each of the sources which have about the same energy are selected.

35. The computer readable medium of claim 34, wherein the one or more processors are further configured to perform the step comprising incrementing a selected pixel by a hysteresis factor to reduce flickering artifacts.

36. The computer readable medium of claim 31, wherein the step of applying contrast normalization further comprises the steps of, for each Laplacian image at a given level:

obtaining an energy image from the Laplacian image;

determining a gain factor that is based on at least the energy image and a target contrast; and

multiplying the Laplacian image by a gain factor to produce a normalized Laplacian image.

37. The computer readable medium of claim 31, wherein the one of the plurality of images is separated into a luminance component and chrominance components, wherein the luminance component is contrast normalized and undergoes image fusion, and the chrominance component is combined with the fused image to produce a color enhanced fused image.

38. The computer readable medium of claim 37, wherein the one or more processors are further configured to perform the steps comprising:

applying a luminance damping function to the luminance component; and

applying a gamma correction to the chrominance components.