MULTI-SCALE ADAPTIVE FUSION WITH CONTRAST NORMALIZATION

• US 20140064636A1
• Filed: 02/22/2013
• Published: 03/06/2014
• Est. Priority Date: 11/29/2007
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: ZHANG, CHAO, BURT, PETER JEFFREY, van der Wal, GOOITZEN SIEMAN
• Granted Patent: US 8,873,883 B2
• Time in Patent Office: Days
• US Class Current: 382/284
• CPC Class Codes: G06T 2200/28 involving image processing hardware
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 3/4061 by injecting details from different spectral ranges
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/90 Dynamic range modification of images or parts thereof
• CPC Class Codes: H04N 25/672 between adjacent sensors or output registers for reading a single image

Abstract

A computer implemented method for fusing images taken by a plurality of cameras is disclosed, comprising the steps of: receiving a plurality of images of the same scene taken by the plurality of cameras; generating Laplacian pyramid images for each source image of the plurality of images; applying contrast normalization to the Laplacian pyramids images; performing pixel-level fusion on the Laplacian pyramid images based on a local salience measure that reduces aliasing artifacts to produce one salience-selected Laplacian pyramid image for each pyramid level; and combining the salience-selected Laplacian pyramid images into a fused image. Applying contrast normalization further comprises, for each Laplacian image at a given level: obtaining an energy image from the Laplacian image; determining a gain factor that is based on at least the energy image and a target contrast; and multiplying the Laplacian image by a gain factor to produce a normalized Laplacian image.

Claims

1. A method for fusing images comprising:

generating Laplacian pyramid images from a plurality of images of the same scene captured by an image sensing device, by generating a Gaussian pyramid for at least one of the plurality of images and applying noise coring to said Gaussian pyramid;

applying contrast normalization to the Laplacian pyramids images;

performing pixel-level fusion on the Laplacian pyramid images to produce a set of salience-selected Laplacian pyramid images corresponding to a set of puramid levels; and

combining the salience-selected Laplacian pyramid images into a fused image.

2. The method of claim 1 further comprising performing the pixel level fusion based on a local salience measure that reduces aliasing artifacts.

3. The method of claim 1 wherein performing pixel-level fusion further comprises, for each level of the Laplacian pyramid images:

corresponding to each of the plurality of received images;

passing a Laplacian pyramid image through a filter for averaging each pixel with surrounding pixels;

convolving the filtered Laplacian pyramid image with the absolute value of the Laplacian pyramid image to produce an energy image;

generating a selection mask wherein a pixel at a given location is selected based on comparing the energy of the pixels originating from energy images corresponding each of the received plurality of images; and

multiplying the selection mask by a filter to reduce aliasing artifacts by smoothing the contributions to the selection mask from each of the energy images to produce a contribution image.

4. The method of claim 3, further comprising the step of summing contribution images corresponding to each of the received plurality of images to produce the salience-selected Laplacian pyramid image at one level of a Laplacian pyramid.

5. The method of claim 3, wherein the pixel that is selected at a given location is the pixel among the energy images corresponding to one image of the plurality of received images with the highest energy.

6. The method of claim 3, further comprising the step of multiplying the energy image with a region of interest (ROI) mask to remove a static overlay in a received image or remove the border induced by warping or by a different source size.

7. The method of claim 5, further comprising the step of incrementing a selected pixel by a hysteresis factor to reduce flickering artifacts.

8. The method of claim 1, wherein said step of applying noise coring further comprising the steps of:

processing at least the Level 0 Gaussian image substantially simultaneously through a plurality of derivative (gradient) filters to produce a first set of gradient images;

applying a noise coring function to each of the gradient images;

processing the noise cored gradient images substantially simultaneously through a second set of derivative (gradient) filters; and

negatively summing the resulting filtered/cored images to produce a the noise-cored Laplacian image.

9. A method for fusing images comprising:

generating Laplacian pyramid images from a plurality of images received by a image sensing device;

applying contrast normalization to the Laplacian pyramids images by, for each Laplacian image at a given level;

obtaining an energy image from the Laplacian image;

determining a gain factor based on at least the energy image and a target contrast;

and

multiplying the Laplacian image by the gain factor to produce a normalized Laplacian image;

performing pixel-level fusion on the Laplacian pyramid images to produce a set of salience-selected Laplacian pyramid images corresponding to a set of pyramid levels; and

combining the salience-selected Laplacian pyramid images into a fused image

10. The method of claim 9, wherein the target contrast is halved with increasing pyramid level.

11. The method of claim 9, wherein the gain factor undergoes noise coring.

12. The method of claim 9, wherein the degree of noise coring is halved with increasing pyramid level.

13. The method of claim 9, wherein a saturation mask is applied to the gain factor.

14. The method of claim 9, wherein performing pixel-level fusion further comprises, for each level of the Laplacian pyramid images:

corresponding to each of the plurality of received images;

passing a Laplacian pyramid image through a filter for averaging each pixel with surrounding pixels;

convolving the filtered Laplacian pyramid image with the absolute value of the Laplacian pyramid image to produce an energy image;

generating a selection mask wherein a pixel at a given location is selected based on comparing the energy of the pixels originating from energy images corresponding each of the received plurality of images; and

multiplying the selection mask by a filter to reduce aliasing artifacts by smoothing the contributions to the selection mask from each of the energy images to produce a contribution image.

15. A method for fusing images comprising:

generating Laplacian pyramid images from a plurality of images received by a image sensing device;

generating region of interest (ROI) mask pyramids of the generated Laplacian pyramid images;

applying contrast normalization to the Laplacian pyramids images based on the ROI mask pyramids;

performing pixel-level fusion on the Laplacian pyramid images to produce a set of salience-selected Laplacian pyramid images corresponding to a set of pyramid levels; and

combining the salience-selected Laplacian pyramid images into a fused image.

16. The method of claim 15 further comprising performing the pixel level fusion based on a local salience measure that reduces aliasing artifacts.

17. The method of claim 15 wherein performing pixel-level fusion further comprises, for each level of the Laplacian pyramid images:

corresponding to each of the plurality of received images;

passing a Laplacian pyramid image through a filter for averaging each pixel with surrounding pixels;

convolving the filtered Laplacian pyramid image with the absolute value of the Laplacian pyramid image to produce an energy image;

generating a selection mask wherein a pixel at a given location is selected based on comparing the energy of the pixels originating from energy images corresponding each of the received plurality of images; and

multiplying the selection mask by a filter to reduce aliasing artifacts by smoothing the contributions to the selection mask from each of the energy images to produce a contribution image.

18. The method of claim 17, further comprising the step of summing contribution images corresponding to each of the received plurality of images to produce the salience-selected Laplacian pyramid image at one level of a Laplacian pyramid.

19. The method of claim 17, wherein the pixel that is selected at a given location is the pixel among the energy images corresponding to one image of the plurality of received images with the highest energy.

20. The method of claim 17, further comprising the step of multiplying the energy image with a region of interest (ROI) mask to remove a static overlay in a received image or remove the border induced by warping or by a different source size.