High dynamic range image generation and rendering

• US 9,978,130 B2
• Filed: 11/18/2013
• Issued: 05/22/2018
• Est. Priority Date: 02/04/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Sun, Shijun
• Primary Examiner(s): Rush, Eric
• Application Number:
  US14/083,246
• Publication Number:
  US 20140079335A1
• Time in Patent Office: 1,646 Days
• Field of Search: 382169, 382168, 382274, 345589, 345590, 345690
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/90 Dynamic range modification of images or parts thereof
• CPC Class Codes: G06T 5/94 based on local image properties, e.g. for local contrast enhancement
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G09G 5/10 Intensity circuits
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors

Abstract

Techniques and tools for high dynamic range (HDR) image rendering and generation. An HDR image generating system performs motion analysis on a set of lower dynamic range (LDR) images and derives relative exposure levels for the images based on information obtained in the motion analysis. These relative exposure levels are used when integrating the LDR images to form an HDR image. An HDR image rendering system tone maps sample values in an HDR image to a respective lower dynamic range value, and calculates local contrast values. Residual signals are derived based on local contrast, and sample values for an LDR image are calculated based on the tone-mapped sample values and the residual signals. User preference information can be used during various stages of HDR image generation or rendering.

Claims

1. One or more computer-readable hardware media storing instructions that, when executed by a computing device, perform a method of generating a high dynamic range digital image, the method comprising:

performing motion analysis relative to a reference image in a set of plural digital images depicting a scene, the motion analysis comprising determining image differences, relative to the reference image, for respective non-reference images in a plurality of non-reference images in the set;

for the respective non-reference images;

determining a plurality of corresponding locations between the reference image and the non-reference image based at least in part on the motion analysis;

for each of the plurality of corresponding locations, comparing a sample value of the reference image and a sample value of the non-reference image; and

based on the comparing of the sample values for each of the plurality of corresponding locations, deriving a single relative exposure level for the non-reference image as a whole relative to the reference image; and

based at least in part on the relative exposure levels of the non-reference images, integrating the set of plural digital images to form a higher dynamic range digital image depicting the scene.

2. The computer-readable hardware media of claim 1, wherein the method further comprises:

selecting the reference image from a set of candidate images by determining which image in the set of candidate images has a greater number of sample values that fall within an acceptable exposure range.

3. The computer-readable hardware media of claim 1, wherein the motion analysis comprises using normalized cross correlation to determine the image differences relative to the reference image.

4. The computer-readable hardware media of claim 1, wherein the motion analysis comprises selecting a subset of sample values in a non-reference image to determine the image differences relative to the reference image, the subset being selected based on whether the sample values fall within a range of exposure values.

5. The computer-readable hardware media of claim 1, wherein the motion analysis comprises global motion analysis in which the one or more non-reference images are analyzed for image differences with respect to the reference image that are present throughout an image, followed by local motion refinement in which the one or more non-reference images are analyzed for image differences that are present in a portion of an image.

6. The computer-readable hardware media of claim 1, wherein the respective single relative exposure levels are derived based on an average sample ratio for a respective non-reference image.

7. The computer-readable hardware media of claim 1, further comprising:

based at least in part on the single relative exposure levels, scaling sample values in the respective non-reference images to a normalization level.

8. The computer-readable hardware media of claim 7, wherein the integrating comprises calculating, for respective sample positions in the higher dynamic range digital image, a weighted average of a sample value corresponding to the sample position in the reference image and a scaled sample value corresponding to the sample position in the respective non-reference images.

9. The computer-readable hardware media of claim 1, further comprising selecting the reference image in the set of plural digital images.

10. The computer-readable hardware media of claim 9, wherein the selecting is based on at least one of:

user preference information or an exposure range of the reference image.

11. The computer-readable hardware media of claim 1, wherein the motion analysis comprises determining a motion vector for one or more pixels in each of the one or more non-reference images.

12. The computer-readable hardware media of claim 1, further comprising converting sample values in the set of plural digital images to a linear domain.

13. The computer-readable hardware media of claim 1, wherein the single relative exposure levels are derived based on at least one of:

sample values in a linear domain or sample ratios.

14. The computer-readable hardware media of claim 1, further comprising preprocessing one or more of the plural digital images to remove one or more image artifacts prior to the motion analysis.

15. A computer-implemented method of generating a high dynamic range digital image, the method comprising:

for a set of digital images depicting a scene, determining image sample displacement between samples in a reference image in the set and corresponding samples in respective non-reference images in the set;

for the respective non-reference images;

determining, for a plurality of samples of the non-reference image, a ratio of a sample value of the non-reference image and a sample value of a corresponding location in the reference image, the corresponding location based at least in part on the determined image sample displacement; and

deriving a single relative exposure level for the non-reference image relative to the reference image based at least in part on the sample value ratios; and

based at least in part on the relative exposure levels of the non-reference images, integrating the set of digital images to form a high dynamic range digital image depicting the scene, wherein the high dynamic range image has a higher dynamic range than the images in the set of digital images.

16. The method of claim 15, wherein integrating the set of digital images comprises calculating weighted averages of corresponding sample values from the reference image and the respective non-reference images.

17. The method of claim 15, further comprising:

for the respective non-reference images, scaling the sample values of the image using the relative exposure level of the image.

18. A high dynamic range digital image rendering system comprising:

a motion analyzer configured to;

detect motion in a set of images that represent a scene at different time instances by determining image differences between a plurality of non-reference images and a reference image in the set of images, and

quantify the detected motion by calculating motion vectors for subsections of the respective non-reference images in the plurality of non-reference images, wherein a motion vector indicates a displacement of the subsection of the non-reference image relative to the reference image;

a relative exposure deriver configured to derive relative exposures for the plurality of non-reference images by, for the respective non-reference images;

using the calculated motion vectors, determining sample values for corresponding subsections in the non-reference image and the reference image;

determining, for the respective corresponding subsections, a ratio of the sample value for the subsection of the non-reference image and the sample value for the subsection of the reference image; and

determining a single relative exposure level representing the non-reference image based on the determined ratios;

an image integrator configured to create a high dynamic range image representing the scene based at least in part on the determined relative exposures of the respective non-reference images and the sample values of the subsections of the respective images in the set of images, wherein the high dynamic range image has a higher dynamic range than the images in the set of images; and

at least one processor configured to communicate with the motion analyzer, relative exposure deriver, and image integrator to perform the respective functions.

19. The system of claim 18, wherein the subsections of non-reference images and corresponding subsections of the reference image are at least one of:

individual pixels, subsampled pixels, or groups of pixels.

20. The system of claim 18, wherein the image integrator is further configured to integrate the sample values corresponding to a subsection for both the reference image and the respective non-reference images in the plurality of non-reference images to determine a sample value for the subsection in the high dynamic range image.