Adaptive transfer function for video encoding and decoding

• US 10,812,801 B2
• Filed: 02/25/2015
• Issued: 10/20/2020
• Est. Priority Date: 02/25/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Tourapis, Alexandros, Singer, David
• Primary Examiner(s): Rahman, Mohammad J
• Application Number:
  US14/631,410
• Publication Number:
  US 20150245050A1
• Time in Patent Office: 2,064 Days
• Field of Search: None
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/92 based on global image properties
• CPC Class Codes: G09G 2320/0261 in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: G09G 2320/0271 Adjustment of the gradation levels within the range of the gradation scale, e.g. by redistribution or clipping
• CPC Class Codes: G09G 2320/0613 The adjustment depending on the type of the information to be displayed
• CPC Class Codes: G09G 2320/062 Adjustment of illumination source parameters
• CPC Class Codes: G09G 2320/0626 for control of overall brightness
• CPC Class Codes: G09G 2320/066 for control of contrast
• CPC Class Codes: G09G 2320/0666 for control of colour parameters, e.g. colour temperature
• CPC Class Codes: G09G 2320/0673 for control of gamma adjustment, e.g. selecting another gamma curve
• CPC Class Codes: G09G 2320/0686 with two or more screen areas displaying information with different brightness or colours
• CPC Class Codes: G09G 2320/0693 Calibration of display systems
• CPC Class Codes: G09G 2320/08 Arrangements within a display terminal for setting, manually or automatically, display parameters of the display terminal
• CPC Class Codes: G09G 2320/103 Detection of image changes, e.g. determination of an index representative of the image change
• CPC Class Codes: G09G 2340/02 Handling of images in compressed format, e.g. JPEG, MPEG
• CPC Class Codes: G09G 2360/144 the light being ambient light
• CPC Class Codes: G09G 2370/042 for monitor identification
• CPC Class Codes: G09G 3/2007 Display of intermediate tones
• CPC Class Codes: G09G 5/005 Adapting incoming signals to the display format of the display terminal
• CPC Class Codes: G09G 5/02 : characterised by the way in which colour is displayed details of colour display specific for CRTs G09G1/28; specific for flat matrix panels other than liquid crystal displays G09G3/2003; specific for liquid crystal displays G09G3/3607
• CPC Class Codes: G09G 5/10 : Intensity circuits
• CPC Class Codes: H04N 1/6066 : dependent on the gamut of the image to be reproduced
• CPC Class Codes: H04N 1/64 : Systems for the transmission or the storage of the colour picture signal; Details therefor, e.g. coding or decoding means therefor H04N19/00 takes precedence
• CPC Class Codes: H04N 19/102 : characterised by the element, parameter or selection affected or controlled by the adaptive coding
• CPC Class Codes: H04N 19/124 : Quantisation
• CPC Class Codes: H04N 19/136 : Incoming video signal characteristics or properties
• CPC Class Codes: H04N 19/137 : Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/14 : Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/154 : Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/17 : the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/172 : the region being a picture, frame or field
• CPC Class Codes: H04N 19/176 : the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/177 : the unit being a group of pictures [GOP]
• CPC Class Codes: H04N 19/182 : the unit being a pixel
• CPC Class Codes: H04N 19/184 : the unit being bits, e.g. of the compressed video stream
• CPC Class Codes: H04N 19/186 : the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/188 : the unit being a video data packet, e.g. a network abstraction layer [NAL] unit
• CPC Class Codes: H04N 19/30 : using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/33 : in the spatial domain
• CPC Class Codes: H04N 19/44 : Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/463 : by compressing encoding parameters before transmission
• CPC Class Codes: H04N 19/52 : by predictive encoding
• CPC Class Codes: H04N 19/86 : involving reduction of coding artifacts, e.g. of blockiness
• CPC Class Codes: H04N 19/98 : Adaptive-dynamic-range coding [ADRC]
• CPC Class Codes: H04N 21/42202 : environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/4223 : Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4318 : by altering the content in the rendering process, e.g. blanking, blurring or masking an image region image enhancement or restoration in general G06T5/00
• CPC Class Codes: H04N 21/44008 : involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4402 : involving reformatting operations of video signals for household redistribution, storage or real-time display details of conversion of video standards at pixel level H04N7/01; video transcoding H04N19/40; adapting incoming signals to the display format of the display terminal G09G5/005; media handling at the source in data packet switching networks H04L65/764
• CPC Class Codes: H04N 21/44218 : Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/4854 : for modifying image parameters, e.g. image brightness, contrast
• CPC Class Codes: H04N 23/741 : by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 5/20 : Circuitry for controlling amplitude response
• CPC Class Codes: H04N 5/505 : Invisible or silent tuning
• CPC Class Codes: H04N 9/641 : Multi-purpose receivers, e.g. for auxiliary information H04N9/642 takes precedence
• CPC Class Codes: H04N 9/67 : for matrixing camera processing pipelines for matrixing of colour signals H04N23/85

Abstract

A video encoding and decoding system that implements an adaptive transfer function method internally within the codec for signal representation. A focus dynamic range representing an effective dynamic range of the human visual system may be dynamically determined for each scene, sequence, frame, or region of input video. The video data may be cropped and quantized into the bit depth of the codec according to a transfer function for encoding within the codec. The transfer function may be the same as the transfer function of the input video data or may be a transfer function internal to the codec. The encoded video data may be decoded and expanded into the dynamic range of display(s). The adaptive transfer function method enables the codec to use fewer bits for the internal representation of the signal while still representing the entire dynamic range of the signal in output.

Claims

1. A system comprising:

at least one processor;

wherein the at least one processor implements an encoder module configured to process video data according to an encoding format to generate encoded video data, wherein the encoded video data is represented using code values of an internal transfer function at a bit depth of C bits within the encoder module; and

wherein the at least one processor implements an adaptive transfer function module configured to;

receive input video data using code values of a transfer function at a bit depth of N bits, wherein N is greater than C;

determine a focus dynamic range for the input video data based at least in part according to one or more characteristics of the input video data, wherein the focus dynamic range is less than a total dynamic range of the input video data such that one or more code values of the input video data are outside the focus dynamic range, and wherein the focus dynamic range is representable by the code values of the internal transfer function at the bit depth of C bits;

crop the input video data to the focus dynamic range to generate cropped N-bit video data such that the cropped N-bit video data excludes the one or more code values of the input video data outside the focus dynamic range;

map the cropped N-bit video data according to the internal transfer function of the encoder module to generate C-bit video data; and

output the C-bit video data to the encoder module for processing.

2. The system as recited in claim 1, wherein the adaptive transfer function module is configured to perform said receive, said determine, said crop, said map, and said output for each of one or more video frames or for each of one or more sequences of two or more video frames.

3. The system as recited in claim 1, wherein the adaptive transfer function module is configured to perform said receive, said determine, said crop, said map, and said output for two or more regions within each of one or more video frames.

4. The system as recited in claim 1, wherein the encoder module is further configured to include format metadata in the encoded video data, the format metadata indicating one or more parameters used in said crop.

5. The system as recited in claim 4, further comprising:

a decoder module configured to process the encoded video data generated by the encoder module to generate decoded C-bit video data and extracted format metadata; and

an inverse adaptive transfer function module configured to expand the decoded C-bit video data according to the extracted format metadata to generate D-bit video data covering full dynamic range of a target device.

6. The system as recited in claim 5, wherein the target device is a high dynamic range (HDR) enabled device.

7. The system as recited in claim 1, wherein the internal transfer function is the same as the transfer function used to represent the input video data.

8. The system as recited in claim 1, wherein the internal transfer function is different than the transfer function used to represent the input video data.

9. The system as recited in claim 1, wherein the internal transfer function is a discontinuous transfer function representation that includes one of more portions of the transfer function used to represent the input video data and that uses a different transfer function for the remainder of the transfer function representation.

10. The system as recited in claim 1, wherein, to determine the focus dynamic range for the input video data, the adaptive transfer function module is configured to determine the focus dynamic range for the input video data at least in part according to one or more characteristics of reference video data from one or more video frames that were previously processed by the encoder module or from one or more previously processed portions of a current video frame being processed by the encoder module.

11. The system as recited in claim 1, wherein the one or more characteristics used to determine the focus dynamic range for the input video data include one or more of brightness, motion, texture, or color characteristics of the input video data.

12. The system as recited in claim 1, wherein the focus dynamic range represents an effective dynamic range of the human visual system for a current scene, sequence, frame, or region of a frame.

13. The system as recited in claim 1, wherein, to process a current video frame according to an encoding format to generate encoded video data, the encoder module is configured to access reference data from one or more previously processed video frames, wherein the system further comprises a reformatting module configured to convert the reference data from the focus dynamic range used in processing the respective video frames to the focus dynamic range determined for the current frame.

14. The system as recited in claim 1, wherein the encoding format is one of H.264/Advanced Video Coding (AVC) format or H.265 High Efficiency Video Coding (HEVC) format.

19. The system as recited in claim 1, wherein, to map the code values of the input video data within the focus dynamic range to code values of the internal transfer function to generate C-bit video data, the adaptive transfer function module is configured to restrict the input video data to the focus dynamic range to generate cropped N-bit video data and map the code values of the cropped N-bit video data to code values of the internal transfer function to generate C-bit video data values.

15. A method, comprising:

iteratively performing, by an encoder module;

determining a focus dynamic range for input video data according to one or more characteristics of the input video data, wherein the input video data is represented using code values of a transfer function at a bit depth of N bits, and wherein the focus dynamic range is less than a total dynamic range of the input video data such that one or more code values of the input video data are outside the focus dynamic range, and wherein the focus dynamic range is representable by code values of a transfer function of the encoder module at a bit depth of C bits;

cropping the input video data to the focus dynamic range to generate cropped N-bit video data such that the cropped N-bit video data excludes the one or more code values of the input video data outside the focus dynamic range;

mapping the code values of the cropped N-bit video data to the code values of the transfer function of the encoder module at the bit depth of C bits to generate C-bit video data, wherein N is greater than C; and

processing the C-bit video data according to an encoding format to generate encoded video data as output.

16. The method as recited in claim 15, further comprising:

iteratively performing, by a decoder module;

decoding input encoded video data to generate decoded C-bit video data; and

expanding the decoded C-bit video data according to format metadata to generate D-bit video data covering full dynamic range of a high dynamic range (HDR)-enabled device, wherein the format metadata indicates one or more parameters used in said cropping and said mapping.

17. The method as recited in claim 15, wherein the focus dynamic range represents an effective dynamic range of the human visual system for a current scene, sequence, frame, or region of a frame.

18. An apparatus, comprising:

an encoder module configured to;

determine a focus dynamic range for input video data according to one or more characteristics of the input video data, wherein the input video data is represented using code values of a transfer function at a bit depth of N bits, and wherein the focus dynamic range is less than a total dynamic range of the input video data such that one or more code values of the input video data are outside the focus dynamic range, and wherein the focus dynamic range is representable by code values of a transfer function of the encoder module at a bit depth of C bits;

crop the input video data to the focus dynamic range to generate cropped N-bit video data such that the cropped N-bit video data excludes the one or more code values of the input video data outside the focus dynamic range;

quantize the cropped N-bit video data according to the transfer function of the encoder module to generate C-bit video data, wherein N is greater than C; and

map the code values of the cropped N-bit video data to code values of a transfer function of the encoder module at a bit depth of C bits to generate C-bit video data, wherein N is greater than C; and

process the C-bit video data according to an encoding format to generate encoded video data as output; and

a decoder module configured to;

decode the encoded video data to generate decoded C-bit video data; and

expand the decoded C-bit video data according to a transfer function of a display device to generate D-bit video data covering full dynamic range of the display device.