Global alignment for high-dynamic range image generation

• US 8,711,248 B2
• Filed: 02/25/2011
• Issued: 04/29/2014
• Est. Priority Date: 02/25/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Jandhyala, Radhika, Wu, Yongjun, Sun, Shijun
• Primary Examiner(s): Tran, Nhan T
• Application Number:
  US13/035,883
• Publication Number:
  US 20120218442A1
• Time in Patent Office: 1,159 Days
• Field of Search: 348/239, 348/362, 348/208.4, 348/208.1, 348/208.12, 348/415.1, 348/416.1, 348/218.1, 375/240.16
• US Class Current: 348/239
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors

Abstract

Techniques and tools for high dynamic range (“HDR”) image generation and rendering are described herein. In several described embodiments, images having distinct exposure levels are aligned. In particular embodiments, the alignment of a reference image to a non-reference image is based at least in part on motion vectors that are determined using covariance computations. Furthermore, in certain embodiments, saturated areas, underexposed areas, and/or moving objects are ignored or substantially ignored during the image alignment process. Moreover, in certain embodiments, a hierarchical pyramid block-based scheme is used to perform local motion estimation between the reference image and the non-reference image.

Claims

1. One or more non-transitory computer-readable media storing computer-executable instructions which when executed by a computer cause the computer to perform a method, the method comprising:

receiving a reference image, the reference image comprising a digital image at a first exposure level and at a first time;

receiving a non-reference image, the reference image comprising a digital image at a second exposure level and at a second time, the second exposure level being different than the first exposure level, and the second time being different than the first time;

estimating two or more local motion vectors for the non-reference image relative to the reference image, the estimating of the two or more local motion vectors comprising;

downsampling the non-reference image and the reference image one or more times to form a plurality of versions of the non-reference image and the reference image at a plurality of resolutions, wherein a lowest resolution version of the non-reference image has a plurality of blocks associated as a pyramid structure to a successively larger number of blocks at corresponding locations in each higher resolution version of the non-reference image, and wherein a lowest resolution version of the reference image has a plurality of blocks associated as a pyramid structure to a successively larger number of blocks at corresponding locations in each higher resolution version of the reference image;

for the lowest resolution version of the non-reference image, estimating motion vectors of the blocks of the non-reference image by performing a search for motion vectors in the associated lowest resolution version of the reference image, wherein the search comprises searching for matching blocks at nearby block locations then expanding the search; and

for each successively higher resolution version of the non-reference image, estimating motion vectors of the blocks of the successively higher resolution version of the non-reference image using a refinement search starting from the motion vector estimated for the associated block of the preceding lower resolution version of the non-reference image according to the pyramid structure.

2. The one or more computer-readable media of claim 1, wherein the method further comprises estimating a global motion transform based at least in part on the two or more local motion vectors.

3. The one or more computer-readable media of claim 2, wherein the method further comprises performing image warping on the non-reference image based at least in part on the global motion transform in order to compensate for motion between the non-reference image and the reference image.

4. The one or more computer-readable media of claim 2, wherein the estimating a global motion transform comprises selecting a subset of the two or more motion vectors for use in estimating the global motion transform, the selecting the subset of motion vector comprising one or more of (a) excluding motion vectors on image boundaries;

(b) excluding motion vectors for blocks having an associated covariance value that fails to exceed a covariance threshold;

or (c) excluding motion vectors for block that have an image content variation value that fails to exceed a content variation threshold.

5. The one or more computer-readable media of claim 2, wherein the estimating the global motion transform comprises using a random sample consensus method for removing outlier motion vectors.

6. The one or more computer-readable media of claim 1, wherein the performing the full search for motion vectors comprises computing covariance values between a block in the lowest resolution version of the reference image and two or more candidate blocks in the associated lowest resolution version of the non-reference image, or computing covariance values between a block in the lowest resolution version of the non- reference image and two or more candidate blocks in the associated lowest resolution version of the reference image.

7. The one or more computer-readable media of claim 1, wherein the estimating two or more local motion vectors is performed using a metric that measures a degree to which pixel values of a block in the non-reference image change together with pixel values of a block in the reference image.

8. A digital image processing device for generating HDR images, the digital image processing device comprising:

a memory storing an image alignment program;

at least one digital signal processing component;

a processing unit operable to execute the image alignment program, wherein said execution of the image alignment program causes the processing unit to;

input a reference image, the reference image comprising a digital image at a first exposure level and at a first time;

input a non-reference image, the reference image comprising a digital image at a second exposure level and at a second time, the second exposure level being different than the first exposure level, and the second time being different than the first time; and

compute a local motion vector for a block of pixel sample values in the non-reference image indicating relative motion of the block to a corresponding block of pixel sample values in the reference image, the local motion vector being determined at least in part using a metric measuring how the pixel sample values in the block in the non-reference image change together with the pixel sample values in the block in the reference image.

9. The digital image processing device of claim 8, wherein the metric is the covariance between the pixel sample values in the block in the non-reference image and the pixel sample values in the block in the reference image.

10. A method, the method comprising:

by computing hardware;

receiving a reference image, the reference image comprising a digital image at a first exposure level and at a first time;

receiving a non-reference image, the reference image comprising a digital image at a second exposure level and at a second time, the second exposure level being different than the first exposure level, and the second time being different than the first time;

estimating two or more local motion vectors for the non-reference image relative to the reference image, the estimating of the two or more local motion vectors comprising;

downsampling the non-reference image and the reference image one or more times to form a plurality of versions of the non-reference image and the reference image at a plurality of resolutions, wherein a lowest resolution version of the non-reference image has a plurality of blocks associated as a pyramid structure to a successively larger number of blocks at corresponding locations in each higher resolution version of the non-reference image, and wherein a lowest resolution version of the reference image has a plurality of blocks associated as a pyramid structure to a successively larger number of blocks at corresponding locations in each higher resolution version of the reference image;

for the lowest resolution version of the non-reference image, estimating motion vectors of the blocks of the non-reference image by performing a search for motion vectors in the associated lowest resolution version of the reference image, wherein the search comprises searching for matching blocks at nearby block locations then expanding the search; and

for each successively higher resolution version of the non-reference image, estimating motion vectors of the blocks of the successively higher resolution version of the non-reference image using a refinement search starting from the motion vector estimated for the associated block of the preceding lower resolution version of the non-reference image according to the pyramid structure.

11. The method of claim 10, wherein the method further comprises estimating a global motion transform based at least in part on the two or more local motion vectors.

12. The method of claim 11, wherein the method further comprises performing image warping on the non-reference image based at least in part on the global motion transform in order to compensate for motion between the non-reference image and the reference image.

13. The method of claim 11, wherein the estimating a global motion transform comprises selecting a subset of the two or more motion vectors for use in estimating the global motion transform, the selecting the subset of motion vector comprising one or more of (a) excluding motion vectors on image boundaries;

(b) excluding motion vectors for blocks having an associated covariance value that fails to exceed a covariance threshold;

or (c) excluding motion vectors for block that have an image content variation value that fails to exceed a content variation threshold.

14. The method of claim 11, wherein the estimating the global motion transform comprises using a random sample consensus method for removing outlier motion vectors.

15. The method of claim 10, wherein the performing the search for motion vectors comprises computing covariance values between a block in the lowest resolution version of the reference image and two or more candidate blocks in the associated lowest resolution version of the non-reference image, or computing covariance values between a block in the lowest resolution version of the non-reference image and two or more candidate blocks in the associated lowest resolution version of the reference image.

16. The method of claim 10, wherein the estimating two or more local motion vectors is performed using a metric that measures a degree to which pixel values of a block in the non-reference image change together with pixel values of a block in the reference image.

17. A digital image processing method for generating HDR images, the method comprising:

by computing hardware;

inputting a reference image, the reference image comprising a digital image at a first exposure level and at a first time;

inputting a non-reference image, the reference image comprising a digital image at a second exposure level and at a second time, the second exposure level being different than the first exposure level, and the second time being different than the first time; and

computing a local motion vector for a block of pixel sample values in the non-reference image indicating relative motion of the block to a corresponding block of pixel sample values in the reference image, the local motion vector being determined at least in part using a metric measuring how the pixel sample values in the block in the non-reference image change together with the pixel sample values in the block in the reference image.

18. The digital image processing method of claim 17, wherein the metric is the covariance between the pixel sample values in the block in the non-reference image and the pixel sample values in the block in the reference image.