Combining multiple exposure images to increase dynamic range

• US 20070242900A1
• Filed: 04/13/2006
• Published: 10/18/2007
• Est. Priority Date: 04/13/2006
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Chen, Mei, Lin, Qian, Lim, Suk
• Granted Patent: US 7,623,683 B2
• Time in Patent Office: Days
• US Class Current: 382/294
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

Methods, machines, and machine-readable media for processing multiple exposure source images, including a reference image and one or more non-reference images, are described. In one aspect, respective sets of motion vectors that establish correspondences between respective pairs of the source images are calculated. Ones of the source images are warped to a reference coordinate system based on the sets of motion vectors. For each of the non-reference ones of the warped images, a respective set of one or more measures of alignment confidence between the non-reference warped image and the reference image in the reference coordinate system is computed. Saturated pixels and unsaturated pixels are identified in the warped images. An output image with pixel value contributions from unsaturated pixels in the warped images is generated based on the computed alignment confidence measures.

Claims

1. A machine-implemented method of processing source images including a reference image and one or more non-reference images, the method comprising:

calculating respective sets of motion vectors establishing correspondences between respective pairs of the source images;

warping ones of the source images to a reference coordinate system based on the sets of motion vectors;

for each of the non-reference ones of the warped images, computing a respective set of one or more measures of alignment confidence between the non-reference warped image and the reference image in the reference coordinate system;

identifying saturated pixels and unsaturated pixels in the warped images; and

generating an output image with pixel value contributions from unsaturated pixels in the warped images based on the computed alignment confidence measures.

2. The method of claim 1, wherein the computing comprises computing measures of correlation between pixels of the reference image in the reference coordinate system and corresponding pixels of the non-reference ones of the warped images.

3. The method of claim 1, further comprising computing saliency measures describing local texture content and contrast in local regions of the warped images, and assigning the saliency measures to corresponding ones of the pixels in the warped images, wherein the generating comprise classifying pixels in the non-reference ones of the warped images as reliable or unreliable based on comparisons of the corresponding alignment confidence measures with an alignment confidence threshold, and nullifying pixel value contributions to the output image from unreliable pixels in the non-references ones of the warped images.

4. The method of claim 3, wherein the generating comprises identifying in each set of corresponding reliable pixels across the warped images a pixel associated with a highest exposure level, and assigning to pixels of the output image values of corresponding ones of the highest exposure level pixels identified in the sets of corresponding pixels.

5. The method of claim 1, further comprising computing saliency measures describing local texture content and contrast in local regions of the warped images, and assigning the saliency measures to corresponding ones of the pixels in the warped images, wherein the generating comprises determining for each of the pixels in the output image a weighted combination of values of corresponding pixels across the warped images.

6. The method of claim 1, further comprising constructing from the warped images respective multiresolution pyramids of images at different resolution levels ranging from a coarse resolution level to a fine resolution level.

7. The method of claim 6, wherein the computing comprises computing measures of alignment confidence between the non-references ones of the warped images and the reference image in the reference coordinate system for each image in each of the multiresolution pyramids, and computing the alignment confidence measures for images above the coarse resolution level based on alignment confidence measures of lower resolution images in the corresponding multiresolution pyramids.

8. The method of claim 1, wherein the output image has pixels located in the reference coordinate system at an output image resolution level and the source images have pixels located in respective source image coordinate systems at a source image resolution level lower than the output image resolution level, wherein the warping comprises up-projecting the source images and the sets of motion vectors from the source image resolution level to the output image resolution level.

9. The method of claim 8, wherein the warping comprises re-mapping ones of the up-projected source images to the reference coordinate system based on the up-projected sets of motion vectors, and the generating comprises computing for each pixel of the output image a respective combination of pixel value contributions from unsaturated pixels in the up-projected and re-mapped source images weighted based on the sets of alignment confidence measures.

10. The method of claim 9, further comprising computing saliency measures describing local texture content and contrast in local regions of the up-projected re-mapped source images and assigning the saliency measures to corresponding ones of the pixels in the up-projected re-mapped source images, wherein the generating comprises additionally weighting the pixel value contributions from the unsaturated pixels in the up-projected and re-mapped source images based on the assigned saliency measures.

11. The method of claim 9, wherein the generating comprises additionally weighting the pixel value contributions from the unsaturated pixels in the up-projected and re-mapped source images based on measures of temporal distance between the reference image and respective ones of the up-projected and re-mapped source images.

12. The method of claim 1, wherein the output image has pixels located in an output image coordinate system at an output image resolution level and registered in relation to the reference coordinate system, and the source images have pixels located in respective source image coordinate systems at a source image resolution level lower than the output image resolution level, wherein the warping comprises re-mapping the source images to a common reference coordinate system at the source image resolution level, and up-projecting the re-mapped source images and the sets of motion vectors from the source image resolution level to the output image resolution level.

13. The method of claim 12, further comprising up-projecting the re-mapped images and the sets of alignment confidence measures from the source image resolution level to the output image resolution level, wherein the generating comprises computing for each pixel of the output image a respective combination of pixel value contributions from unsaturated pixels in the up-projected images based on the up-projected sets of alignment confidence measures.

14. The method of claim 13, further comprising computing saliency measures describing local texture content and contrast in local regions of the up-projected images and assigning the saliency measures to corresponding ones of the pixels in the up-projected images, wherein the generating comprises additionally weighting the pixel value contributions from the unsaturated pixels in the up-projected images based on the assigned saliency measures.

15. The method of claim 13, wherein the generating comprises additionally weighting the pixel value contributions from the unsaturated pixels in the up-projected images based on measures of temporal distance between the reference image and respective ones of the warped images.

16. The method of claim 1, further comprising determining signal-to-noise measures for respective ones of the pixels in the warped images, wherein the generating comprises determining for each of the pixels in the output image a weighted combination of values of corresponding pixels across the warped images, and the determining comprises weighing pixel value contributions to the weighted combination from pixels in the warped images in accordance with the corresponding alignment confidence measures, pixel saturation measures and the determined signal-to-noise measures.

17. A system for processing source images including a reference image and one or more non-reference images, the system comprising at least one data processing module configured to perform operations comprising:

calculating respective sets of motion vectors establishing correspondences between respective pairs of the source images;

warping ones of the source images to a reference coordinate system based on the sets of motion vectors;

for each of the non-reference ones of the warped images, computing a respective set of one or more measures of alignment confidence between the non-reference warped image and the reference image in the reference coordinate system;

identifying saturated pixels and unsaturated pixels in the warped images; and

generating an output image with pixel value contributions from unsaturated pixels in the warped images based on the computed alignment confidence measures.

18. A machine-implemented method of processing source images including a reference image and one or more non-reference images, the method comprising:

computing saliency measures describing local texture content and contrast in local regions of the source images, and assigning the saliency measures to corresponding ones of the pixels in the source images;

identifying saturated pixels and unsaturated pixels in the source images; and

generating an output image with pixel value contributions from unsaturated pixels in the source images based on the computed saliency measures.

19. The method of claim 18, wherein the generating comprises identifying a highest saliency pixel in each set of corresponding pixels across the source images that is assigned a highest saliency measure among the pixels in the set of corresponding pixels, and assigning to pixels of the output image values of corresponding ones of the highest saliency pixels identified in the sets of corresponding pixels.

20. The method of claim 18, wherein the generating comprises determining for each of the pixels in the output image a weighted combination of values of corresponding pixels across the warped images, wherein the determining comprises weighing pixel value contributions to the weighted combination from reliable pixels in the warped images in accordance with the corresponding alignment confidence measures and the assigned saliency measures.