COMPATIBLE COMPRESSION OF HIGH DYNAMIC RANGE, VISUAL DYNAMIC RANGE, AND WIDE COLOR GAMUT VIDEO

• US 20110194618A1
• Filed: 04/21/2011
• Published: 08/11/2011
• Est. Priority Date: 03/13/2009
• Status: Active Grant

• Current Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Original Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Inventors: Vogt, Christopher, Gish, Walter, Li, Zhen
• Granted Patent: US 8,982,963 B2
• Time in Patent Office: Days
• US Class Current: 375/240.25
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G09G 2320/0271 Adjustment of the gradation levels within the range of the gradation scale, e.g. by redistribution or clipping
• CPC Class Codes: G09G 5/10 Intensity circuits
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/36 Scalability techniques involving formatting the layers as a function of picture distortion after decoding, e.g. signal-to-noise [SNR] scalability
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards

Abstract

A first image stream has a first dynamic range and a first color space. First and the second image streams are received in a layered codec. The second image stream has a second dynamic range, which is higher than the first dynamic range. The first image stream is in the codec'"'"'s base layer; the second image stream is in its enhancement layer. The first image stream is encoded to obtain an encoded image stream, which is decoded to obtain a decoded image stream. The decoded image stream is converted from the first non-linear or linear color space to a second, different color space to obtain a color converted image stream. A higher dynamic range image representation of the color converted image stream is generated to obtain a transformed image stream. Inverse tone mapping parameters are generated based on the transformed image stream and the second image stream.

Claims

1. A method for generating inverse tone mapping parameters (724), comprising:

receiving a first image stream (704) of a first dynamic range and of a first color space and a second image stream (702) of a second dynamic range in a layered codec;

wherein;

the first dynamic range is lower than the second dynamic range;

the first image stream (704) is in a base layer and the second image stream (702) is in an enhancement layer;

the layered codec comprises a decoder (708) and an encoder (708); and

one or more of the base layer or enhancement layer has a sampling precision that comprises one or more of;

an intensity and color sampling pattern of 4;

2;

2;

an intensity and color sampling pattern of 4;

2;

4;

an intensity and color sampling pattern that at least meets or exceeds 4;

2;

0;

a bit depth that at least meets or exceeds 8 bits;

or

a bit depth of 10-14 bits;

encoding the first image stream (704) in the base layer using the encoder (708) to obtain an encoded image stream (738);

decoding the encoded image stream (738) using the decoder (708) to obtain a decoded image stream (710);

converting the decoded image stream (710) from the first color space to a second color space different from the first color space to obtain a color converted image stream;

wherein one or more of the first color space or the second color space comprises a linear or a non-linear color space;

generating a higher dynamic range image representation of the color converted image stream to obtain a transformed image stream (714); and

generating the inverse tone mapping parameters (724) based on the transformed image stream (714) and the second image stream (702).

2. The method as recited in claim 1, wherein the first image stream (704) is obtained by performing tone mapping on the second image stream (702).

3. The method as recited in claim 2, further comprising inserting data for one or more details from an original image in the second image stream that were lost as a result of creating the first image stream from the original image.

4. The method as recited in claim 1, wherein the generating a higher dynamic range image representation comprises performing at least one transformation (230, 240) and at least one quantization (230, 240).

5. The method as recited in claim 1, further comprising:

applying at least one de-contouring operation to the transformed image stream (714) to obtain a de-contoured image stream (720);

wherein the generating the inverse tone mapping parameters (724) is based on the de-contoured image stream (720) and the second image stream (714).

6. The method as recited in claim 1, wherein the generating the inverse tone mapping parameters (724) comprises generating a function of one or more of:

a parameterized global tone mapping operator;

a parameterized local tone mapping operator;

a parameterized inverse global tone mapping operator;

or

a parameterized inverse local tone mapping operator.

7. The method as recited in claim 1, wherein the generating the inverse tone mapping parameters (724) comprises generating a function of one or more of:

a monotonic inverted tone mapping operator;

an inverted local tone mapping operator;

a function that comprises a plurality of quadratic curves;

or

one or more parameters that corresponds to one or more of a dodging operation or a burning operation.

8. The method as recited in claim 1, wherein the generating the inverse tone mapping parameters (724) is based on the transformed image stream (714), the second image stream (702), and a functional form (815), wherein the functional form provides a relationship between the inverse tone mapping parameters (724), the transformed image stream (714), and the second image stream (702).

9. The method of claim 8, wherein the generating the inverse tone mapping parameters (724) comprises calculating least-square errors for each pixel in the transformed image stream (714) and each pixel in the second image stream (702) to estimate the inverse tone mapping parameters (724).

10. The method of claim 8, wherein the functional form (815) comprises a non-linear or linear form.

11. The method of claim 1, wherein the generating the inverse tone mapping parameters (724) is based on at least one pixel but less than an entirety of pixels in each of the of the transformed image stream (714) and the second image stream (702).

12. The method as recited in claim 1, further comprising:

upon generating inverse tone mapping parameters (724), performing inverse tone mapping (TTM) on the transformed image stream (714) to obtain a predicted image stream (726) of the second dynamic range;

producing a residual (1103) between the predicted image stream (714) and the second image stream (702); and

processing the residual (1103) to obtain a residual bit stream (1120),

wherein the residual (1103) comprises a difference between the predicted image stream (714) and the second image stream (702) and is adapted for signaling to a codec decoder.

13. The method as recited in claim 12, wherein the processing the residual comprises:

performing low-pass filtering (1105) on the residual to obtain a filtered residual;

performing down-sampling (1110) on the filtered residual to obtain a down-sampled residual; and

encoding (1115) the down-sampled residual to obtain the residual bit stream.

14. The method as recited in claim 12, further comprising:

encoding the residual (1103) to obtain a residual bit stream (742); and

encoding the inverse tone mapping parameters (724) to obtain an inverse tone mapping parameter bit stream (740);

wherein the encoded image stream (738), the residual bit stream (742), and the inverse tone mapping parameter bit stream (740) are adapted to be signaled to a codec decoder (748).

15. The method as recited in claim 12, further comprising:

encoding the residual (1103) to obtain a residual bit stream (742);

encoding the inverse tone mapping parameters (724) to obtain an inverse tone mapping parameter bit stream (740); and

formatting the encoded image stream (738), the residual bit stream (742), and the inverse tone mapping parameter bit stream (740) with a formatter block (745) to obtain a compressed bit stream (750),

wherein the compressed bit stream (750) is adapted to be signaled to a codec decoder (748).

16. The method as recited in claim 15, further comprising producing an output image stream (795) of the second dynamic range from the codec decoder (748), which decodes the compressed bit stream (750), wherein the producing the output image stream (795) comprises:

receiving the compressed bit stream (750);

parsing (752) the compressed encoded bit stream (750) into a third image stream (754) of the first dynamic range, a second tone mapping image stream (780), and a second internal residual image stream (758);

decoding the third image stream (754) to obtain a second decoded internal image stream (784),

decoding the second tone mapping image stream (780) to obtain a second decoded tone mapping image stream (724), and

decoding the second internal residual image stream (758) to obtain a second decoded internal residual image stream (798);

converting the second decoded internal image stream (784) from the first color space to the second color space to obtain a second decoded color converted image stream;

generating a higher dynamic range image representation of the second decoded color converted image stream to obtain a second transformed image stream (790);

performing inverse tone mapping (793) on the second transformed image stream (790) based on the second decoded tone mapping image stream (724) to obtain a second predicted image stream of the second dynamic range; and

generating the output image stream (795) based on the second decoded internal residual image stream (798) and the second predicted image stream.

17. The method as recited in claim 16, further comprising:

applying at least one de-contouring operation to the second transformed image stream (790) to obtain a second de-contoured image stream (792),

wherein the performing inverse tone mapping (793) is performed on the second de-contoured image stream (792) based on the second decoded tone mapping image stream (724).

22. The method as recited in claim 1, wherein the converting comprises performing at least one transformation (210, 240).

23. The method as recited in claim 22, wherein the first color space comprises at least one of a gamma-encoded R′

G′

B′

color space or a linear color space and the second color space is selected from the group consisting of a CIE XYZ color space and a YCrCb color space.

24. The method as recited in claim 23, wherein a first transformation (210) is from the at least one gamma-encoded R′

G′

B′

color space to an RGB color space or the linear color space is transformed to a second linear color space and a second transformation (220) is from the RGB color space to the second color space.

25. The method as recited in claim 22, wherein the first color space is an RGB color space and the second color space is a selected from the group consisting of a CIE XYZ color space and a YCrCb color space.

26. The method as recited in claim 25, wherein the at least one transformation comprises transforming from the RGB color space to the second color space.

27. The method as recited in claim 1, wherein the generating a higher dynamic range image representation comprises performing projective transformation on one or more chroma components of an output from the converting.

18. A method for generating an output bit stream (795) from an input bit stream (754), comprising:

receiving an enhancement layer that comprises the input bit stream (754), an inverse tone mapping bit stream (780), and a residual bit stream (758), wherein the input bit stream (754) is of a first dynamic range and of a first color space;

decoding the bit stream (754) to obtain a decoded bit stream (784);

decoding the inverse tone mapping bit stream (780) to obtain a decoded inverse tone mapping bit stream (724);

decoding the residual bit stream (758) to obtain a decoded residual bit stream;

converting the decoded bit stream (784) from the first color space to a second color space to obtain a decoded color converted image stream;

generating a higher dynamic range image representation of the decoded color converted image stream to obtain a transformed image stream (790);

performing inverse tone mapping (793) on the transformed image stream (790) based on the decoded inverse tone mapping bit stream (780) to obtain a predicted bit stream of a second dynamic range; and

generating a base layer that comprises the output bit stream (795), wherein the output bit stream (795) is of the second dynamic range, based on the decoded residual bit stream (798) and the predicted bit stream,

wherein the first dynamic range is lower than the second dynamic range; and

wherein one or more of the base layer or enhancement layer has a sampling precision that comprises one or more of;

an intensity and color sampling pattern of 4;

2;

2;

an intensity and color sampling pattern of 4;

2;

4;

an intensity and color sampling pattern that at least meets or exceeds 4;

2;

0;

a bit depth that at least meets or exceeds 8 bits;

or

a bit depth of 10-14 bits.

19. The method as recited in claim 18, further comprising:

applying at least one de-contouring operation to the transformed image stream (790) to obtain a de-contoured image stream (792),

wherein the performing inverse tone mapping (793) is performed on the de-contoured image stream (792) based on the decoded inverse tone mapping bit stream (724).

20. The method as recited in claim 18, wherein the performing inverse tone mapping (793) comprises performing a function of one or more of:

a parameterized global tone mapping operator;

a parameterized local tone mapping operator;

a parameterized inverse global tone mapping operator;

or

a parameterized inverse local tone mapping operator.

21. The method as recited in claim 18, wherein the performing inverse tone mapping (793) comprises performing a function of one or more of:

a monotonic inverted tone mapping operator;

an inverted local tone mapping operator;

a function that comprises a plurality of quadratic curves;

or

one or more parameters that corresponds to one or more of a dodging operation or a burning operation.

32. The method as recited in claim 18, wherein the converting comprises performing at least one transformation (210, 240).

33. The method as recited in claim 18, wherein the generating a higher dynamic range image representation comprises performing projective transformation on one or more chroma components of an output from the converting.

28. A system for generating inverse tone mapping parameters (724), comprising:

means for receiving a first image stream (704) of a first dynamic range and of a first color space and a second image stream (702) of a second dynamic range in a layered codec;

wherein;

the first dynamic range is lower than the second dynamic range;

the first image stream (704) is in a base layer and the second image stream (702) is in an enhancement layer; and

the layered codec comprises a decoder (708) and an encoder (708);

means for encoding the first image stream (704) in the base layer using the encoder (708) to obtain an encoded image stream (738);

means for decoding the encoded image stream (738) using the decoder (708) to obtain a decoded image stream (710);

means for converting the decoded image stream (710) from the first color space to a second color space different from the first color space to obtain a color converted image stream;

wherein one or more of the first color space or the second color space comprises a linear or a non-linear color space;

means for generating a higher dynamic range image representation of the color converted image stream to obtain a transformed image stream (714); and

means for generating the inverse tone mapping parameters (724) based on the transformed image stream (714) and the second image stream (702).

29. A video encoder (744) for generating inverse tone mapping parameters (724), comprising:

a receiver that receives a first image stream (704) of a first dynamic range and of a first color space and a second image stream (702) of a second dynamic range in a layered codec;

wherein;

the first dynamic range is lower than the second dynamic range; and

the first image stream (704) is in a base layer and the second image stream (702) is in an enhancement layer;

an encoder function (708) that encodes the first image stream (704) in the base layer to obtain an encoded image stream (738);

a decoder function (708) that decodes the encoded image stream (738) to obtain a decoded image stream (710);

a converter that converts the decoded image stream (710) from the first color space to a second color space that is different from the first color space to obtain a color converted image stream;

wherein one or more of the first color space or the second color space comprises a linear or a non-linear color space;

a representation generator (712) that generates a higher dynamic range image representation of the color converted image stream to obtain a transformed image stream (714); and

a parameter generator (722) that generates the inverse tone mapping parameters (724) based on the transformed image stream (714) and the second image stream (702).

30. A computer apparatus, comprising:

a bus;

one or more processors coupled to the bus; and

a computer readable storage medium comprising code, which when executed with the one or more processors, causes the one or more processors to perform a process for generating inverse tone mapping parameters (724), the process comprising;

receiving a first image stream (704) of a first dynamic range and of a first color space and a second image stream (702) of a second dynamic range in a layered codec;

wherein;

the first dynamic range is lower than the second dynamic range;

the first image stream (704) is in a base layer and the second image stream (702) is in an enhancement layer;

the layered codec comprises a decoder (708) and an encoder (708); and

one or more of the base layer or enhancement layer has a precision that comprises one or more of;

an intensity and color sampling pattern of 4;

2;

2;

an intensity and color sampling pattern of 4;

2;

4;

an intensity and color sampling pattern that at least meets or exceeds 4;

2;

0;

a bit depth that at least meets or exceeds 8 bits;

or

a bit depth of 10-14 bits;

encoding the first image stream (704) in the base layer using the encoder (708) to obtain an encoded image stream (738);

decoding the encoded image stream (738) using the decoder (708) to obtain a decoded image stream (710);

converting the decoded image stream (710) from the first color space to a second color space different from the first color space to obtain a color converted image stream;

wherein one or more of the first color space or the second color space comprises a linear or a non-linear color space;

generating a higher dynamic range image representation of the color converted image stream to obtain a transformed image stream (714); and

generating the inverse tone mapping parameters (724) based on the transformed image stream (714) and the second image stream (702).

31. A computer readable storage medium comprising code, which when executed with the one or more processors of at least one of a system, a computer apparatus or a video encoder, causes the one or more processors to perform a process for generating inverse tone mapping parameters (724), the process comprising:

receiving a first image stream (704) of a first dynamic range and of a first color space and a second image stream (702) of a second dynamic range in a layered codec;

wherein;

the first dynamic range is lower than the second dynamic range;

the first image stream (704) is in a base layer and the second image stream (702) is in an enhancement layer;

the layered codec comprises a decoder (708) and an encoder (708); and

one or more of the base layer or enhancement layer has a precision that comprises one or more of;

an intensity and color sampling pattern of 4;

2;

2;

an intensity and color sampling pattern of 4;

2;

4;

an intensity and color sampling pattern that at least meets or exceeds 4;

2;

0;

a bit depth that at least meets or exceeds 8 bits;

or

a bit depth of 10-14 bits;

encoding the first image stream (704) in the base layer using the encoder (708) to obtain an encoded image stream (738);

decoding the encoded image stream (738) using the decoder (708) to obtain a decoded image stream (710);

converting the decoded image stream (710) from the first color space to a second color space different from the first color space to obtain a color converted image stream;

wherein one or more of the first color space or the second color space comprises a linear or a non-linear color space;

generating a higher dynamic range image representation of the color converted image stream to obtain a transformed image stream (714); and

generating the inverse tone mapping parameters (724) based on the transformed image stream (714) and the second image stream (702).