SYSTEM FOR CODING HIGH DYNAMIC RANGE AND WIDE COLOR GAMUT SEQUENCES

• US 20170026646A1
• Filed: 07/22/2016
• Published: 01/26/2017
• Est. Priority Date: 07/22/2015
• Status: Active Grant

• Current Assignee: ARRIS Enterprises LLC (CommScope Holding Company, Inc.)
• Original Assignee: ARRIS Enterprises LLC (CommScope Holding Company, Inc.)
• Inventors: Minoo, Koohyar, Gu, Zhouye, Baylon, David M., Luthra, Ajay
• Granted Patent: US 10,116,938 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/174 the region being a slice, e.g. a line of blocks or a group of blocks
• CPC Class Codes: H04N 19/179 the unit being a scene or a shot
• CPC Class Codes: H04N 19/182 the unit being a pixel
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/36 Scalability techniques involving formatting the layers as a function of picture distortion after decoding, e.g. signal-to-noise [SNR] scalability
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence

Abstract

A method of encoding a digital video data applies adaptive pre-processing to data representing high dynamic range (HDR) and/or wide color gamut (WCG) image data prior to encoding and complementary post-processing to the data after decoding in order to allow at least partial reproduction of the HDR and/or WCG data. The example methods apply one or more color space conversions, and a perceptual transfer functions to the data prior to quantization. The example methods apply inverse perceptual transfer functions and inverse color space conversions after decoding to recover the HDR and/or WCG data. The transfer functions are adaptive so that different transfer functions may be applied to video data sets including different groups of frames, frames or processing windows in a single frame. Information on the data set and information on the applied transfer function is passed as metadata from the encoder to the decoder.

Claims

1. A method of encoding a digital video, comprising:

receiving a digital video data set including at least one of high dynamic range (HDR) and wide color gamut (WCG) video data;

identifying a characteristic of a portion of the digital video data set;

modifying a perceptual transfer function according to the identified characteristic;

applying the modified perceptual transfer function to the portion of the digital video data set to generate a perceptually modified portion of the digital video data set;

applying a perceptual normalization including at least one of a gain factor or an offset to the perceptually modified digital video data set to generate a perceptually normalized portion of the digital video data set;

encoding the perceptually normalized portion of the video data set to generate a bit stream;

transmitting, to a decoder, the bit stream and metadata that indicates the modification of the perceptual transfer function, that identifies the perceptual normalization, and that identifies the portion of the video data set.

2. The method of claim 1, wherein the transmitting includes transmitting the metadata as supplemental enhancement information (SEI) data in the bit stream.

3. The method of claim 1, wherein the data set includes a sequence of image frames and the identifying the characteristic of the portion of the digital video data set includes identifying the characteristic of the portion of the data set selected from a group consisting of multiple video frames, a single video frame and a portion of a single video frame.

4. The method of claim 1, wherein the encoding includes:

predictively encoding the perceptually normalized portion of the video data set based on one or more reference frames;

regenerating the perceptually normalized portion of the video data set;

applying an inverse perceptual normalization to the perceptually normalized portion of the video data set to regenerate the digital video data set including at least one HDR and WCG video data; and

applying the modified perceptual transfer function to the portion of the digital video data set to regenerate the perceptually modified portion of the digital video data set as a part of a further reference frame to be used in predictively encoding further video data sets.

5. The method of claim 1, further comprising modifying the digital video data processed by the modified perceptual transfer function by applying a tone map to the digital video data processed by the modified perceptual transfer function prior to encoding the processed video data.

6. The method of claim 1 further comprising:

converting the portion of the digital video data set from an input color space to an intermediate color space to generate intermediate color converted video data and generating metadata identifying the input color space, the intermediate color space and the portion of the digital video data set;

applying a compression transfer function to the intermediate color converted video data to generate compressed video data and generating metadata characterizing the compression transfer function and identifying the portion of the digital video data set;

converting the compressed video data from the intermediate color space to a final color space to generate final color converted video data and generating metadata identifying the intermediate color space, the final color space and the portion of the digital video data set; and

combining the metadata identifying the input color space and the intermediate color space, the metadata characterizing the compression transfer function and the metadata identifying the final color space with the metadata that indicates the modification of the perceptual transfer function to generate combined metadata;

wherein the portion of the digital video data to which the perceptual transfer function is applied includes the final color converted video data; and

wherein the transmitting transmits the bit stream and the combined metadata to the decoder.

7. The method of claim 6 further comprising performing a chrominance downsampling operation on the final color converted video data prior to applying the modified perceptual transfer function to the second color converted video data.

8. The method of claim 6 further comprising performing a chrominance downsampling on the processed portion of the video data set prior to encoding the processed portion of the video data set.

9. The method of claim 1 further comprising:

processing the portion of the video data set to generate a tone map to translate the video data set to video data consistent with standard dynamic range (SDR) video data;

combining data representing the tone map with the metadata to generate combined metadata;

wherein the transmitting transmits the bit stream and the combined metadata to the decoder.

10. A digital video encoding system for encoding a digital video data set including at least one of a high dynamic range (HDR) and wide color gamut (WCG) video data, the digital video encoding system comprising:

a perceptual transfer function process configured to;

identify a characteristic of a portion of the digital video data set;

modify a perceptual transfer function according to the identified characteristic; and

apply the modified perceptual transfer function to the portion of the digital video data set to generate a perceptually modified portion of the digital video data set;

apply a perceptual normalization that includes at least one of a gain factor or an offset to the perceptually modified portion of the digital video data set to generate a perceptually normalized portion of the digital video data set;

an encoder for encoding the processed portion of the video data set to generate a bit stream; and

a transmitter for transmitting, to a decoder, the bit stream and metadata that indicates the modification of the perceptual transfer function, that identifies the perceptual normalization, and that identifies the portion of the video data set.

11. The digital video encoding system of claim 10, wherein the encoder is a Main 10 High Efficiency Video Coder (HEVC) encoder and the encoder system formats the metadata as supplemental enhancement information (SEI) data in the bit stream.

12. The digital video encoding system of claim 10, wherein the perceptual transfer function processor is configured to identify the characteristic of the portion of the digital video data as one of a group of multiple video frames, a single video frame or a portion of a single video frame.

13. The digital video encoding system of claim 10, wherein the encoder is configured to:

predictively encode the perceptually normalized portion of the video data set using one or more reference frames;

regenerate the perceptually normalized portion of the video data set;

apply an inverse perceptual normalization to the perceptually normalized portion of the video data set to regenerate the digital video data set including at least one of HDR and WCG video data; and

apply the modified perceptual transfer function to the portion of the digital video data set to regenerate the perceptually modified portion of the digital video data set as a part of a further reference frame to be used by the encoder to predictively encode further video data sets.

14. The digital video encoding system of claim 10 further comprising:

an intermediate color conversion process configured to convert the portion of the digital video data set from an input color space to an intermediate color space to generate intermediate color converted video data and to generate metadata identifying the input color space, the intermediate color space and the portion of the digital video data set;

a compression transfer function process configured to apply a compression transfer function to the intermediate color converted video data to generate compressed video data and to generate metadata characterizing the compression transfer function and identifying the portion of the digital video data set; and

a final color conversion process configured to convert the compressed video data from the intermediate color space to a final color space to generate final color converted video data and to generate metadata identifying the intermediate color space, the final color space and the portion of the digital video data set;

wherein the encoder is configured to combine the metadata identifying the input color space and the intermediate color space, the metadata characterizing the compression transfer function and the metadata identifying the final color space with the metadata that indicates the modification of the perceptual transfer function to generate combined metadata;

wherein the portion of the digital video data to which the perceptual transfer function is applied includes the final color converted video data; and

wherein the transmitting transmits the bit stream and the combined metadata to the decoder.

15. A digital video decoding system for decoding bit stream in a format that does not accommodate a digital video data set including high dynamic range (HDR) and wide color gamut (WCG) video data, to reconstruct an output digital video data set including at least one of HDR and WCG video data, the digital video decoding system comprising:

a decoder for decoding the bit stream to recover a digital video data set from the bit stream;

an inverse perceptual transfer function process configured to;

receive video data corresponding to a portion of the digital video data set;

retrieve metadata from the bit stream identifying a transfer function to be applied to the received video data;

retrieve metadata from the bit stream identifying an inverse perceptual normalization function, including at least one of a gain factor or an offset, to be applied to the received video data;

generate an inverse perceptual transfer function based on the identified transfer function; and

apply the generated inverse perceptual transfer function and the inverse perceptual normalization function to the portion of the digital video data set to generate an inverse transformed portion of the digital video data set;

an output color conversion process configured to process the inverse transformed portion of the digital video data set to reconstruct the output digital video data set including at least one of HDR and WCG video data.

16. The digital video decoding system of claim 15, wherein the bit stream is a Main 10 High Efficiency Video Coder (HEVC) bit stream and the inverse perceptual transfer function process is configured to extract the metadata from supplemental enhancement information (SEI) data in the bit stream.

17. The digital video decoding system of claim 15, wherein the inverse perceptual transfer function process is configured to receive the portion of the digital video data as one of a group of multiple video frames, a single video frame or a portion of a single video frame.

18. The digital video decoding system of claim 15 wherein the decoder is configured to:

predictively decode the bit stream using one or more reference frames;

apply a color conversion process to the output digital video data set, the color conversion process being an inverse of the output color conversion process to generate a color converted video data set;

generate a perceptual transfer function process from the metadata identifying the inverse perceptual transfer function to be applied to the received video data;

apply the perceptual transfer function to the color converted video data set to generate a transformed video data set;

generate a perceptual normalization function using the metadata identifying the inverse perceptual normalization function;

apply the perceptual normalization function to the transformed video data set to generate a perceptually normalized video data set;

generate a further reference frame for use in predictively decoding the bit stream from the perceptually normalized video data set.

19. The digital video decoding system of claim 15 further comprising:

an intermediate color conversion process configured to;

extract intermediate color conversion metadata from the bit stream, the intermediate color conversion metadata identifying an input color space of the digital video data set extracted from the bit stream and an intermediate color space to which the digital video data set is to be converted; and

convert the inverse transformed portion of the digital video data set from the input color space to the intermediate color space to produce intermediate color converted digital video data;

an inverse compression transfer function process configured to extract compression metadata from the bit stream and to apply an inverse compression transfer function to the intermediate color converted video data to generate the digital video data set to be applied to the output color conversion process.

20. A non-transitory computer readable medium containing computer instructions, the instructions causing a computer to:

receive a digital video data set including at least one of high dynamic range (HDR) and wide color gamut (WCG) video data;

identify a characteristic of a portion of the digital video data set;

modify a perceptual transfer function according to the identified characteristic;

apply the modified perceptual transfer function to the portion of the digital video data set;

encode the processed portion of the video data set to generate a bit stream; and

transmit, to a decoder, the bit stream and metadata that indicates the modification of the perceptual transfer function and that identifies the portion of the video data set.