Method and apparatus for producing digital images having extended dynamic ranges

• US 5,828,793 A
• Filed: 05/06/1996
• Issued: 10/27/1998
• Est. Priority Date: 05/06/1996
• Status: Expired due to Fees

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Mann, Steve
• Primary Examiner(s): Johns, Andrew W.
• Assistant Examiner(s): Mariam, Daniel G.
• Application Number:
  US08/643,473
• Time in Patent Office: 904 Days
• Field of Search: 382/132, 382/294, 382/284, 356/448, 356/522, 396/60, 396/61, 396/62, 348/222, 348/229
• US Class Current: 382/284
• CPC Class Codes: G06T 2207/10144 Varying exposure
• CPC Class Codes: G06T 2207/20212 Image combination
• CPC Class Codes: G06T 5/40 using histogram techniques
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/90 Dynamic range modification of images or parts thereof
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors

Abstract

Different exposures of an electronically represented, pictorial scene are combined into a final image having expanded dynamic range yet still capturing subtle differences in exposure. The final image provides increased shadow, midtone and highlight detail notwithstanding the limited response of the system (electronic or photographic) that produced the component images. The invention features automatic identification of optimally exposed regions (on a pixel-by-pixel basis) from the component images, as well as automatic combination into a final image. The invention may also be used to make a camera, of possibly unknown response function, behave as an array of photometric measurement instruments (e.g., an array of light meters) to within a constant scale factor.

Claims

1. A method of combining a plurality of differently exposed source images of the same object or scene, the method comprising the steps of:

a. establishing a response function characterizing the source images in terms of values of a response parameter and exposures producing the values; and

b. based on the response function, combining the source images into a composite image comprising image detail from each of the source images and having a greater range of image detail than any of the source images

wherein

c. the source and composite images each comprise an ordered array of pixels;

d. the pixels of the composite image correspond to the pixels of the source images; and

e. the images are combined by assigning, to each composite-image pixel, a pixel-parameter value representing a weighted average of the pixel parameter of corresponding pixels from the source images obtained by;

i. for each source image, generating a certainty function characterizing a tonal range of greatest image detail; and

ii. for each composite-image pixel, (A) weighing the corresponding pixels from the source images in accordance with the certainty functions, (B) combining the pixel-parameter values of the corresponding source image pixels in accordance with the weighting, and (C) assigning the combined pixel-parameter value to the composite-image pixel.

2. The method of claim 1 wherein the response parameter is luminance.

3. The method of claim 1 further comprising the step of registering the source images.

4. The method of claim 3 wherein the registering step comprises:

a. featurelessly approximating parameters of a projective coordinate transformation that spatially relates, in first and second images source, pixels corresponding to common subject matter therebetween;

b. applying the parameters to the first source image to thereby transform it into a processed image, the common subject matter encoded by pixels in the processed image being substantially spatially consistent with the common subject matter encoded by pixels in the second source image; and

c. aligning the images by combining the pixels corresponding to the common subject matter; and

d. repeating steps (a) to (c) until all source images are aligned.

5. The method of claim 1 further comprising the step of obtaining a user input, the pixel parameters being weighted according to said input.

6. The method of claim 1 wherein the composite image has highlight, midtone and shadow areas, and the weighted average is varied for pixels in at least one of the areas.

7. The method of claim 1 wherein the response function is established according to steps comprising:

a. identifying a shadow-region pixel from a first source image, the shadow-region pixel having an image location and an associated pixel parameter;

b. identifying, in a second source image, a corresponding second-image pixel having the same image location as the shadow-region pixel and having an associated pixel parameter;

c. identifying, in the first source image, a first-image pixel having an associated pixel parameter equal to that of the second-image pixel;

d. identifying, in the second source image, a corresponding second-image pixel having the same image location as the first-image pixel;

e. repeating steps (c) and (d); and

f. deriving the response function through comparison of the pixel parameters of the pixels identified in the first source image with the pixel parameters of the pixels identified in the second source image.

8. The method of claim 1 wherein the step of generating the certainty function comprises differentiating the response function.

9. The method of claim 1 wherein the images correspond to different color channels of the same image, and the composite image is rendered in a single color.

10. Apparatus for combining a plurality of differently exposed source images of the same object or scene, the apparatus comprising:

a. analysis means for establishing a response function characterizing at least one of the source images in terms of values of a response parameter and exposures producing the values;

b. combining means for combining, based on the response function, the source images into a composite image comprising image detail from each of the source images and having a greater range of image detail than any of the source images; and

c. a computer memory for storing images as ordered sets of pixels each having at least one associated pixel parameter;

wherein

d. the analysis means establishes a response function characterizing at least one of the source images in terms of values of a pixel parameter and exposures producing the values, and is configured to generate, for each source image, a certainty function characterizing a tonal range of greatest image detail; and

e. the combining means is configured to (i) weight the corresponding pixels from the source images in accordance with the certainty functions, (ii) combine the pixel-parameter values of the corresponding source image pixels in accordance with the weighting, and (iii) store the combined pixel-parameter value in the computer memory as a displayable value.

11. The apparatus of claim 10 wherein the pixel parameter is luminance.

12. The apparatus of claim 10 wherein the analysis means is further configured to register the source images.

13. The apparatus of claim 10 wherein the analysis means is configured to establish the response function by:

a. identifying a shadow-region pixel from a first source image, the shadow-region pixel having an image location and an associated pixel parameter;

b. identifying, in a second source image, a corresponding second-image pixel having the same image location as the shadow-region pixel and having an associated pixel parameter;

c. identifying, in the first source image, a first-image pixel having an associated pixel parameter equal to that of the second-image pixel;

d. identifying, in the second source image, a corresponding second-image pixel having the same image location as the first-image pixel;

e. repeating steps (c) and (d); and

f. deriving the response function through comparison of the pixel parameters of the pixels identified in the first source image with the pixel parameters of the pixels identified in the second source image.

14. The apparatus of claim 10 wherein the analysis means generates the certainty function by differentiating the response function.

15. The apparatus of claim 10 wherein the analysis means is further configured to generate an image of exposure intermediate between two of the source images by (i) generating a response function intermediate to response functions associated with the two source images and (ii) modifying one of the source images to conform to the intermediate response function.

16. The apparatus of claim 10 further comprising a selectably actuable filter for modifying the composite image.