Methods and systems for inverse tone mapping

• US 9,607,364 B2
• Filed: 11/19/2014
• Issued: 03/28/2017
• Est. Priority Date: 11/22/2013
• Status: Active Grant

• Current Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Original Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Inventors: Kunkel, Timo, Xu, Ning, Chen, Tao, Crenshaw, James E., Lee, Bongsun
• Primary Examiner(s): Hung, Yubin
• Application Number:
  US15/035,477
• Publication Number:
  US 20160358319A1
• Time in Patent Office: 860 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20028 Bilateral filtering
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/40 using histogram techniques
• CPC Class Codes: G06T 5/90 Dynamic range modification of images or parts thereof
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: H04N 1/4072 dependent on the contents of the original
• CPC Class Codes: H04N 1/6027 Correction or control of colour gradation or colour contrast H04N1/6058 takes precedence

Abstract

Novel methods and systems for inverse tone mapping are disclosed. A scene can be analyzed to obtain highlight detection from bright light sources and specular reflections. An inverse tone mapping curve can be calculated based on the lower dynamic range and higher dynamic range displays. Multi-scale filtering can be applied to reduce noise or artifacts.

Claims

1. A method to perform inverse tone mapping, the method comprising:

providing, by a computer, a luminance of an image;

detecting, by a computer, highlights in the luminance, thereby obtaining a highlight detection parameter;

estimating, by a computer, a mid-value of the luminance, thereby obtaining a mid-value luminance parameter;

providing, by a computer, minimum and maximum luminance values for a lower dynamic range display and for a higher dynamic range display;

calculating, by a computer, an inverse tone mapping curve, based on the highlight detection parameter, the mid-value luminance parameter, and the minimum and maximum luminance values for the lower dynamic range display and the higher dynamic range display; and

applying, by a computer, the inverse tone mapping curve to the luminance as a global mapping operator, thereby obtaining an inversely mapped luminance.

2. The method of claim 1, wherein the detecting highlights comprises detecting highlights from brighter light sources and from specular reflections.

3. The method of claim 1, wherein the detecting highlights comprises estimating a probability of highlighted pixels in the image.

4. The method of claim 1, wherein the highlight detection parameter and the mid-value luminance parameter are scene-adaptive.

5. The method of claim 1, wherein the image is a frame in a video.

6. The method of claim 5, further comprising recursive filtering between adjacent frames in the video.

7. The method of claim 1, wherein the lower dynamic range display or the higher dynamic range display is a virtual display, and wherein the virtual display has a different maximum luminance value from a corresponding source display.

8. The method of claim 7, further comprising:

adding, by a computer, virtual highlights to a luminance of the image, wherein the virtual highlights have a higher luminance level than a luminance level of the virtual display;

displaying, by a computer, the image with virtual highlights on the virtual display; and

removing, by a computer, the virtual highlights.

9. The method of claim 1, wherein the detecting highlights is based on a dark channel information and two threshold values.

10. The method of claim 1, wherein the detecting highlights is based on heuristic criteria, perception-based criteria, or a multi-scale approach.

11. The method of claim 1, wherein the calculating comprises:

obtaining, by a computer, a mid-value luminance of a lower dynamic range image; and

estimating, by a computer, a mid-value luminance of a higher dynamic range image, based on the mid-value luminance of the lower dynamic range image.

12. The method of claim 1, further comprising applying a multi-scale filter to the inversely mapped luminance, thereby obtaining a filtered inversely mapped luminance.

13. The method of claim 12, wherein the multi-scale filter comprises a Gaussian filter, a bilateral filter, or a guided filter.

14. The method of claim 12, further comprising blending, by a computer, the inversely mapped luminance with the filtered inversely mapped luminance, thereby obtaining a blended inversely mapped luminance.

15. The method of claim 14, wherein the blending comprises:

defining, by a computer, a first threshold parameter and a second threshold parameter, wherein the second threshold parameter is greater than the first threshold parameter;

iterating, by a computer, between pixels of the image, wherein the iterating is for at least one pixel and wherein the iterating executes one of the following steps

i. if a slope of the inverse tone mapping curve at a luminance value of the at least one pixel is less than the first threshold parameter, the blending for the at least one pixel considers only the inversely tone mapped luminance for the at least one pixel;

ii. if a slope of the inverse tone mapping curve at a luminance value of the at least one pixel is more than the second threshold parameter, the blending for the at least one pixel considers only the filtered inversely mapped luminance for the at least one pixel;

or

iii. if a slope of the inverse tone mapping curve at a luminance value of the at least one pixel is more than the first threshold parameter and less than the second threshold parameter, the blending for the at least one pixel is linearly interpolated between the inversely tone mapped luminance and the filtered inversely mapped luminance for the at least one pixel.

16. The method of claim 15, wherein the image comprises high frequency luminance pixels, the high frequency luminance pixels having frequencies higher than a fixed value and wherein the method further comprises mapping of the high frequency luminance pixels separately.

17. A system comprising:

a decoder, configured to receive an encoded lower dynamic range image or video and output a decoded lower dynamic range image or video;

an inverse tone mapping module, configured to receive the decoded lower dynamic range image or video and output an inversely tone mapped image or video, wherein the inverse tone mapping is carried out according to the method of claim 1.

18. A system comprising:

at least one lower dynamic range camera, configured to output a lower dynamic range image or video;

at least one higher dynamic range camera, configured to output a higher dynamic range image or video;

an inverse tone mapping module, configured to receive the lower dynamic range image or video and output an inversely tone mapped image or video, wherein the inverse tone mapping is carried out according to the method of claim 1;

a higher dynamic range display, configured to receive the higher dynamic range image or video and the inversely tone mapped image or video.