Encoding, decoding, and representing high dynamic range images

• US 8,248,486 B1
• Filed: 04/17/2012
• Issued: 08/21/2012
• Est. Priority Date: 04/15/2011
• Status: Active Grant

• Current Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Original Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Inventors: Ward, Gregory John, Ninan, Ajit, Ten, Arkady, Jia, Wenhui, Wang, Gaven
• Primary Examiner(s): Tran, Nhan T
• Application Number:
  US13/449,119
• Time in Patent Office: 126 Days
• Field of Search: 348/223.1, 348/222.1, 348/234, 348/235, 348/557, 348/663, 348/712, 348/273, 348/280, 382/254, 382/274, 382/162, 345/589, 345/590, 375/240.25, 375/240.26
• US Class Current: 348/223.1
• CPC Class Codes: H04N 1/4072 dependent on the contents of the original
• CPC Class Codes: H04N 1/60 Colour correction or control H04N1/54 takes precedence
• CPC Class Codes: H04N 1/6066 dependent on the gamut of the image to be reproduced
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 23/85 for matrixing
• CPC Class Codes: H04N 9/67 for matrixing camera processing pipelines for matrixing of colour signals H04N23/85
• CPC Class Codes: H04N 9/77 Circuits for processing the brightness signal and the chrominance signal relative to each other, e.g. adjusting the phase of the brightness signal relative to the colour signal, correcting differential gain or differential phase circuits for matrixing H04N9/67

Abstract

Techniques are provided to encode and decode image data comprising a tone mapped (TM) image with HDR reconstruction data in the form of luminance ratios and color residual values. In an example embodiment, luminance ratio values and residual values in color channels of a color space are generated on an individual pixel basis based on a high dynamic range (HDR) image and a derivative tone-mapped (TM) image that comprises one or more color alterations that would not be recoverable from the TM image with a luminance ratio image. The TM image with HDR reconstruction data derived from the luminance ratio values and the color-channel residual values may be outputted in an image file to a downstream device, for example, for decoding, rendering, and/or storing. The image file may be decoded to generate a restored HDR image free of the color alterations.

Claims

1. A method, comprising:

receiving a high dynamic range (HDR) image;

receiving a tone-mapped (TM) image generated based on the HDR image, the TM image comprising one or more color alterations that are not recoverable from the TM image with a luminance ratio image;

computing luminance ratio values, on an individual pixel basis, by dividing luminance values of the HDR image with luminance values of the TM image on the individual pixel basis;

applying the luminance ratio values to the HDR image to create a re-mapped image;

determining residual values in color channels of a color space based on the re-mapped image and the TM image; and

outputting the TM image with HDR reconstruction data, the HDR reconstruction data being derived from the luminance ratio values and the residual values.

2. The method of claim 1, further comprising converting at least one of the re-mapped image and the TM image from a different color space to the color space of the residual.

3. The method of claim 1, wherein the color space is an YCbCr color space, and wherein the color channels of the color space comprise a Cb color channel and a Cr color channel.

4. The method of claim 3, wherein the residual values in the color channels of the color space are calculated as differences between first pixel values, as derived from the re-mapped image, in the color channels and second pixel values, as derived from the TM image, in the color channels.

5. The method of claim 1, wherein at least one of the one or more color alterations in the TM image is caused by one of clippings or alterations of hues, at one or more pixels.

6. The method of claim 1, wherein the HDR reconstruction data comprises a residual image with quantized values derived from the luminance ratio values and the residual values in the color channels of the color space.

7. The method of claim 1, wherein the HDR reconstruction data is stored in an application segment of an image file with the TM image as a base image in the image file.

8. The method of claim 1 further comprising creating an image file, wherein the image file is in a JPEG-HDR format.

9. The method of claim 1, further comprising allowing any tone mapping operations with any tone mapping operator and/or any color alterations on any number of pixels in the TM image.

10. The method of claim 1, further comprising replacing zero, one, or more color-channel zero-values in the TM image with zeros, or values smaller than a threshold value.

11. The method of claim 1, further comprising applying a color space conversion to at least one of the HDR image, the TM image, or the re-mapped image.

12. The method of claim 1, wherein luminance residual values between the TM image and the re-mapped image are all zeros.

13. The method of claim 1, wherein the HDR image is one of a fixed-point image or a floating-point image.

14. The method of claim 1, further comprising performing a preprocessing operation required by a tone mapping operator on the HDR image, wherein the preprocessing operation is one of a color space conversion, a gamma encoding, a gamma decoding, a downsampling, or an upsampling.

15. The method of claim 1, further comprising receiving information relating to gamma encoding performed in the TM image, and performing an inverse operation of the gamma encoding on the TM image to generate an intermediate TM image before determining the residual values in the color channels of the color space.

16. The method of claim 1, wherein the HDR image is encoded in one of JPEG, JPEG-2000, MPEG, AVI, TIFF, BMP, GIFF, or another image format.

17. A method, comprising:

parsing an image file comprising a tone-mapped (TM) base image and HDR reconstruction data, the HDR reconstruction data comprising quantized luminance ratio values and quantized residual values in color channels of a color space;

the TM base image comprising one or more color alterations that are not recoverable from the TM base image with a luminance ratio image;

extracting quantization parameters relating to the quantized luminance ratio values and the quantized residual values in the color channels of the color space;

converting, based at least in part on the quantization parameters, the quantized luminance ratio values and the quantized residual values into luminance ratio values and residual values in the color channels of the color space; and

reconstructing an HDR image using the TM base image and the luminance ratio values and residual values in the color channels of the color space.

18. The method of claim 17, wherein the image file is parsed with a standard based image decoder.

19. The method of claim 17, wherein the quantized luminance ratios and the quantized residual values are stored in a residual image, and wherein the residual image and the TM base image are de-quantized and decompressed using a common procedure.

20. The method of claim 17, wherein the TM base image comprises results of any tone mapping operations with any tone mapping operator and/or any color alterations on any number of pixels.

21. The method of claim 17, wherein the HDR image is one of a fixed-point image or a floating-point image.

22. A method for performing white balance correction on initial image data, said initial image data comprising tone-mapped (TM) image data and residual image data, said residual image data further comprising high dynamic range (HDR) image data, effective linear sensor values and white balance multipliers, the steps of said method comprising:

recovering the effective linear sensor values and the white balance multipliers from said residual image data;

producing sensor color space values from said effective linear sensor values and said initial image data to form a first intermediate image data;

for each color channel, applying white balance multipliers to said first intermediate image data to produce a second intermediate image data;

clamping said second intermediate image data to minimum of the maximum color channel values if there is no HDR image data available from said initial image data to form a third intermediate image data; and

if clamping was not performed, transforming said second intermediate image data into a target monitor color space to form a final image data; and

if clamping was performed, transforming said third intermediate image data into a target monitor color space to form the final image data.

23. The method as recited in claim 22 wherein said initial image data is formatted in JPEG-HDR format.

24. The method as recited in claim 23 wherein said initial image data comprises upper sensor range for the captured HDR in each of the RGB color channels.

25. The method as recited in claim 24 wherein said upper sensor range comprises white balance multipliers and a matrix, said matrix further comprising effective linear sensor values.

26. The method as recited in claim 25 wherein said final image data is input into a HDR path of an image encoder.

27. A method for processing initial image data, the steps of said method comprising:

inputting said initial image data;

inputting sensor color space data and white balance multipliers;

if a new white point for said initial image data is assumed, computing new white balance multipliers to produce current white balance multipliers;

for each pixel in said initial image data, transforming said initial image data into a first intermediate image data back into said sensor color space;

for each color channel in said intermediate image data, applying the current white balance multipliers to produce a second intermediate image data;

clamping said second intermediate image data to the minimum value amount all color channel maximum values to produce a third intermediate image data; and

transforming said third intermediate image data into a desired monitor color space to produce a fourth intermediate image data.

28. The method as recited in claim 27 wherein said fourth intermediate image data is input into a tone mapping operator.

29. The method as recited in claim 27 wherein said initial image data is captured by an image capturing device and further wherein a camera RAW data of said image capturing device is not available.

30. The method as recited in claim 29 wherein said image capturing device is capable of capturing image data in a plurality of color channels and further wherein at least two said color channels saturate at a different luminance value along a black-to-white gradient.