High dynamic range image combining

• US 8,339,475 B2
• Filed: 12/19/2008
• Issued: 12/25/2012
• Est. Priority Date: 12/19/2008
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Atanassov, Kalin M., Velarde, Ruben M., Li, Hsiang-Tsun, Hwang, Hau
• Primary Examiner(s): Jerabek, Kelly L
• Application Number:
  US12/340,474
• Publication Number:
  US 20100157078A1
• Time in Patent Office: 1,467 Days
• Field of Search: 348/222.1, 348/229.1, 348/230.1, 348/255, 348/256, 382/274, 382/284
• US Class Current: 348/229.1
• CPC Class Codes: G06T 2207/10144 Varying exposure
• CPC Class Codes: G06T 2207/20004 Adaptive image processing
• CPC Class Codes: G06T 2207/20012 Locally adaptive
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/40 using histogram techniques
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/90 Dynamic range modification of images or parts thereof

Abstract

Systems and methods of high dynamic range image combining are disclosed. In a particular embodiment, a device includes a global mapping module configured to generate first globally mapped luminance values within a region of an image, a local mapping module configured to generate second locally mapped luminance values within the region of the image, and a combination module configured to determine luminance values within a corresponding region of an output image using a weighted sum of the first globally mapped luminance values and the second locally mapped luminance values. A weight of the weighted sum is at least partially based on a luminance variation within the region of the image.

Claims

1. A computer based method of storing a High Dynamic Range (HDR) image to an image storage, comprising:

receiving first image data from an image sensor, the first image data generated using a first exposure time;

receiving second image data from the image sensor, the second image data generated using a second exposure time that is greater than the first exposure time;

scaling luminance values of the first image data based on a ratio of the second exposure time to the first exposure time to create scaled first image data;

selectively combining the scaled first image data and the second image data based on an overlap between a first effective dynamic range of the scaled first image data and a second effective dynamic range of the second image data to generate a high dynamic range (HDR) image;

generating a tone mapping, including determining a pixel-by-pixel weighting of a global tone mapping value and a local tone mapping value based on a luminance variation within a neighborhood of each pixel of the HDR image;

generating a compressed dynamic range HDR image based, at least in part, on the tone mapping; and

storing the compressed dynamic range HDR image to the image storage.

2. The method of claim 1, wherein the first effective dynamic range is determined based on a first saturation level, a first dark level, and a first noise floor of the first image data, and wherein the second effective dynamic range is determined based on a second saturation level, a second dark level, and a second noise floor of the second image data.

3. The method of claim 1, wherein the scaled first image data and the second image data are selectively combined based on a first sigmoid function substantially centered at the first noise floor and a second sigmoid function substantially centered at the second saturation level minus the second dark level.

4. The method of claim 1, wherein the tone mapping includes a weighted sum of the global tone mapping value and the local tone mapping value.

5. The method of claim 1, further comprising providing the compressed dynamic range HDR image to be displayed at a display device.

6. A computer based method of storing image data to an image storage, comprising:

receiving first image data from an input device;

adjusting first luminance values of first image data based on a luminance distribution of the first image data to generate second image data;

adjusting second luminance values of a region of the second image data based on a local luminance distribution of the region of the second image data to generate locally adjusted image data;

generating third image data having a smaller dynamic range than the first image data,

wherein luminance values of the third image data corresponding to the region of the second image data are determined by combining the locally adjusted image data and the second image data within the region of the second image data based on a luminance variation of the second image data within the region; and

storing the third image data to the image storage.

7. The method of claim 6, wherein adjusting the first luminance values comprises:

determining a histogram of pixel luminance values of the first image data;

capping a range of the histogram based on a uniform distribution level;

normalizing the capped histogram;

determining a cumulative distribution function (CDF) of the normalized histogram; and

applying the CDF as a tone mapping function to the first image data to remap the first luminance values.

8. The method of claim 6, wherein adjusting the second luminance values comprises:

dividing the second image data into multiple regions;

computing a histogram of pixel luminance values of each of the multiple regions;

determining a local tone mapping function corresponding to each of the multiple regions; and

performing a bilinear interpolation between the local tone mapping functions to compute a tone mapping of a particular pixel of a particular region of the multiple regions, wherein the local tone mapping functions correspond to regions of the multiple regions that are adjacent to the particular region.

9. The method of claim 6, further comprising generating a flatness map by determining, for each particular pixel within the region, a ratio of luminance variance to luminance mean using the second image data corresponding to a neighborhood of pixels including the particular pixel.

10. The method of claim 6, wherein the third image data is generated to have at least a same amount of detail as the first image data.

11. The method of claim 6, wherein the locally adjusted image data and the second image data within the region of the second image data are combined using a weighted sum.

12. A wireless device comprising:

a processor; and

a memory accessible to the processor,

wherein the processor is configured to;

adjust first luminance values of first image data based on a luminance distribution of the first image data to generate second image data;

adjust second luminance values of a region of the second image data based on a local luminance distribution of the region of the second image data to generate locally adjusted image data;

generate third image data having a smaller dynamic range than the first image data, wherein luminance values of the third image data corresponding to the region of the second image data are determined by combining the locally adjusted image data and the second image data within the region of the second image data based on a luminance variation of the second image data within the region; and

wherein the processor is configured to provide the third image data for display.

13. The wireless device of claim 12, wherein the processor is further configured to store the third image data at the memory.

14. The wireless device of claim 12, further comprising a wireless interface coupled to the processor, and wherein the processor is further configured to provide the third image data to be communicated via the wireless interface.

15. The wireless device of claim 12, wherein the first image data includes high dynamic range (HDR) image data.

16. The wireless device of claim 15, wherein the high dynamic range (HDR) image data is created by combining multiple exposures from a camera.

17. The wireless device of claim 16, wherein the processor is further configured to:

process the HDR image data to generate a flatness map that indicates luminance variation within the neighborhood of a particular pixel; and

determine a tone mapped pixel value of the particular pixel using a weighted sum of globally mapped image data and locally mapped image data, wherein a weight of the weighted sum is determined based on the luminance variation within the neighborhood of the particular pixel.

18. The wireless device of claim 12, further comprising a camera.

19. The wireless device of claim 12, further comprising a display device.

20. The wireless device of claim 12, wherein the generation of the third image combines the locally adjusted image data and the second image data within the region of the second image data using a weighted sum.

21. An apparatus comprising:

means for generating tone mapped image data including a tone mapped pixel value of each particular pixel of a plurality of pixels of image data based on a luminance variation within a neighborhood of the particular pixel, the image data corresponding to an image captured at a camera, and adjusting first luminance values of first image data based on a luminance distribution of the first image data to generate second image data;

means for adjusting second luminance values of a region of the second image data based on a local luminance distribution of the region of the second image data to generate locally adjusted image data;

means for generating third image data having a smaller dynamic range than the first image data, wherein luminance values of the third image data corresponding to the region of the second image data are determined by combining the locally adjusted image data and the second image data within the region of the second image data based on a luminance variation of the second image data within the region; and

means for providing the tone mapped third image data for display.

22. The apparatus of claim 21, further comprising means for wirelessly communicating the third image data.

23. The apparatus of claim 21, wherein the first image data includes high dynamic range (HDR) image data that is generated by combining a first set of image data received from the camera using a first exposure time and a second set of image data received from the camera using a second exposure time.

24. The apparatus of claim 21, wherein the means for generating third image data uses a weighted sum to combine the locally adjusted image data and the second image data within the region of the second image data.

25. A computer readable medium comprising executable instructions that, when executed, cause a processor to:

receive first image data;

generate second image data based on a luminance distribution of the first image data;

generate locally adjusted image data based on a local luminance distribution within a region of the second image data; and

generate third image data having a smaller dynamic range than the first image data, wherein luminance values of a region of the third image data corresponding to the region of the second image data are determined by combining the locally adjusted image data and the second image data based on a luminance variation of the second image data within the region of the second image data.

26. The computer readable medium of claim 25, wherein combining the locally adjusted image data and the second image data includes using a weighted sum of the locally adjusted image data and the second image data within the region, and wherein a weight of the weighted sum is based on the luminance variation.

27. The computer readable medium of claim 25, wherein a luminance value of a particular pixel within the region of the third image data is determined using a weighted sum of a first value of the second image data corresponding to the particular pixel and a second value of the locally adjusted image data corresponding to the particular pixel.

28. The computer readable medium of claim 25, wherein the instructions that cause a processor to generate third image data combine the locally adjusted image data and the second image data using a weighted sum.

29. A device comprising:

a global mapping module configured to generate first globally mapped luminance values within a region of an image;

a local mapping module configured to generate second locally mapped luminance values within the region of the image; and

a combination module configured to determine luminance values within a corresponding region of an output image using a weighted sum of the first globally mapped luminance values and the second locally mapped luminance values, wherein a weight of the weighted sum is at least partially based on a luminance variation within the region of the image.

30. The device of claim 29, further comprising a variation module configured to generate a flatness map corresponding to a portion of the image, wherein the flatness map indicates the luminance variation within the region of the image.

31. The device of claim 30, wherein an entry of the flatness map corresponding to the particular pixel is proportional to a ratio of a luminance variance within a neighborhood of a particular pixel to a luminance mean within the neighborhood of the particular pixel.