Method and system for combining multiple exposure images having scene and camera motion

• US 20060177150A1
• Filed: 02/01/2005
• Published: 08/10/2006
• Est. Priority Date: 02/01/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Uyttendaele, Matthew T., Szeliski, Richard S., Eden, Ashley Michelle
• Granted Patent: US 7,239,805 B2
• Time in Patent Office: Days
• US Class Current: 382/284
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/009 Global, i.e. based on properties of the image as a whole applied in cameras using an electronic image sensor H04N23/80, H04N23/70
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/92 based on global image properties

Abstract

A panoramic high-dynamic range (HDR) image method and system of combining multiple images having different exposures and at least partial spatial overlap wherein each of the images may have scene motion, camera motion, or both. The major part of the panoramic HDR image method and system is a two-pass optimization-based approach that first defines the position of the objects in a scene and then fills in the dynamic range when possible and consistent. Data costs are created to encourage radiance values that are both consistent with object placement (defined by the first pass) and of a higher signal-to-noise ratio. Seam costs are used to ensure that transitions occur in regions of consistent radiances. The result is a high-quality panoramic HDR image having the full available spatial extent of the scene along with the full available exposure range.

Claims

1. A method for processing digital images, comprising:

inputting an input image set containing a plurality of images capturing a scene, the input image set having different exposures and at least partial spatial overlap;

combining at least some of the plurality of images to create a reference panorama; and

generating from the reference panorama a panoramic high-dynamic range (HDR) image having a full available spatial extent of the scene and a full available dynamic exposure range of the scene.

2. The method of claim 1, further comprising using pixel labeling to fill in and augment an exposure detail of the reference panorama using the plurality of images from the input image set.

3. The method of claim 2, wherein pixel labeling further comprises simultaneously minimizing data costs and seams costs of pixels across the panoramic HDR image to determine optimal pixel labelings.

4. The method of claim 3, further comprising defining the data costs in terms of signal-to-noise ratio (SNR) weightings and pixel probabilities for pixels from the input image set.

5. The method of claim 3, further comprising defining seam costs associated with pixel neighbor pairs from the input image set.

6. The method of claim 5, further comprising defining the seam costs using radiance values of the pixel neighbor pairs.

7. The method of claim 5, further comprising radiometrically aligning the input image set to cause each of the images in the input image set to be consistently color balanced with one another.

8. The method of claim 7, further comprising:

selecting a reference image from the input image set;

determining a gain for each color channel of the reference image; and

adjusting a gain of each color channel for non-reference images in the input image set to match the gain of the corresponding color channel of the reference image.

9. The method of claim 7, further comprising adjusting gains for a certain color channel of each image in the input image set to make the gains as similar as possible.

10. A computer-readable medium having computer-executable instructions for performing the method recited in claim 1.

11. A computer-implemented process for generating a panoramic high-dynamic range (HDR) image, comprising:

an input step for obtaining an input image set containing a plurality of images capturing different exposure levels and at least partial spatial overlap of a scene;

a reference panorama step for combining at least some of the plurality of images in the input image set to generate a reference panorama containing at least a majority of a spatial extent of the scene; and

an exposure augmentation step for filling in and augmenting exposure details of the reference panorama using exposure detail available from the input image set to generate the panoramic HDR image.

12. The computer-implemented process as set forth in claim 11, wherein the exposure augmentation step further comprises simultaneously minimizing data costs and seam costs across the panoramic HDR image to determine optimal pixel labelings.

13. The computer-implemented process as set forth in claim 12, further comprising:

defining the data costs in terms of signal-to-noise ratio (SNR) weightings and pixel probabilities for pixels in the input image set; and

defining the seam costs associated with pixel neighbor pairs of the input image set using radiance values of the pixel neighbor pairs.

14. The computer-implemented process as set forth in claim 13, further comprising defining the data costs, DC(i), for selecting image i from the input image set as the optimal pixel labelings of the panoramic HRD image at pixel p by the equation:

DC(i)=λ

/W(M_p,i)P(L_p,i, L_p,ref)

wherein W is the SNR weighting, P is the pixel probabilities, and λ

is a parameter used to regularize Wand P.

15. The computer-implemented process as set forth in claim 13, further comprising defining the seam costs, C, over each neighbor pixel, p′

, of pixel p at a location (x,y), by the equation;

$C<br><br>\left(p,p^{\mathrm{\prime <br><br>}},i,j\right)=\frac{<br><br>L_{p,i}-L_{p,j}<br><br>+<br><br>L_{p^{\mathrm{\prime <br><br>}},i}-L_{p^{\mathrm{\prime <br><br>}},j}<br><br>}{L_{p,i}+L_{p,j}}$

wherein L is pixel radiance.

16. The computer-implemented process as set forth in claim 12, further comprising:

selecting a reference image from the input image set and designating the non-selected images from the input image set as non-reference images;

determining a gain for a certain color channel of the reference image;

adjusting each non-reference image'"'"'s gain of the certain color channel to approximately match the reference image'"'"'s gain of the certain color channel; and

repeating the above steps for each color channel of the reference image.

17. The computer-implemented process as set forth in claim 16, further comprising defining an error term, E, representing an error between a gain of a non-reference image and a desired gain of the reference image of the certain color channel, by the equation:

$E=\underset{p<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{\varepsilon <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{valid}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{overlap}<br><br>\left(1,R\right)}{\sum <br><br>}<br><br>{\left(L_{p,1}^{\mathrm{\prime <br><br>}}<br><br>G_{1<br><br>R}-L_{p,R}^{\mathrm{\prime <br><br>}}\right)}^2+\underset{p<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{\varepsilon <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{valid}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{overlap}<br><br>\left(1,3\right)}{\sum <br><br>}<br><br>{\left(L_{p,1}^{\mathrm{\prime <br><br>}}<br><br>G_{1<br><br>R}-L_{p,3}^{\mathrm{\prime <br><br>}}<br><br>G_{3<br><br>R}\right)}^2+\mathrm{\dots <br><br>}<br><br>\text{\hspace{1em}<br><br>},$

18. A system for creating a panoramic high-dynamic range (HDR) image containing a full available spatial extent of a scene and a full available exposure range of the scene, comprising:

an input image set containing images capturing the scene at various exposure levels and capturing a spatial extent of the scene, wherein at least some of the images overlap in the spatial domain;

a reference panorama generation module that generates a reference panorama containing a full available spatial extent of the scene using a subset of the input image set; and

a pixel labeling module that fills in and augments exposure detail of the reference panorama by simultaneously minimizing data costs and seam costs to obtain optimal pixel labeling for pixels in the input image set and outputs the panoramic HDR image.

19. The system of claim 18, wherein the data costs are simultaneously minimized using a graph-cut optimization technique.

20. The system of claim 18, further comprising a radiometric alignment module that normalizes the input image set by adjusting color channel gains to take into account exposure differences between images in the input image set.