Perceptual optimization for model-based video encoding

• US 10,097,851 B2
• Filed: 11/18/2016
• Issued: 10/09/2018
• Est. Priority Date: 03/10/2014
• Status: Active Grant

• Current Assignee: Euclid Discoveries LLC
• Original Assignee: Euclid Discoveries LLC
• Inventors: Lee, Nigel, Park, Sangseok, Tun, Myo, Kottke, Dane P., Lee, Jeyun, Weed, Christopher
• Primary Examiner(s): Vaughn, Jr., William C
• Assistant Examiner(s): UHL, LINDSAY JANE KILE
• Application Number:
  US15/356,142
• Publication Number:
  US 20170070745A1
• Time in Patent Office: 690 Days
• Field of Search: None
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/13 Adaptive entropy coding, e.g. adaptive variable length coding [AVLC] or context adaptive binary arithmetic coding [CABAC]
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/147 according to rate distortion criteria rate-distortion as a criterion for motion estimation H04N19/567
• CPC Class Codes: H04N 19/159 Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/182 the unit being a pixel
• CPC Class Codes: H04N 19/184 the unit being bits, e.g. of the compressed video stream
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 19/56 Motion estimation with initialisation of the vector search, e.g. estimating a good candidate to initiate a search
• CPC Class Codes: H04N 19/567 Motion estimation based on rate distortion criteria
• CPC Class Codes: H04N 19/61 in combination with predictive coding

Abstract

Perceptual statistics are used to compute importance maps that indicate which regions of a video frame are important to the human visual system. Importance maps may be generated from encoders that produce motion vectors and employ motion estimation for inter-prediction. The temporal contrast sensitivity function (TCSF) may be computed from the encoder'"'"'s motion vectors. Quality metrics may be used to construct a true motion vector map (TMVM), which refines the TCSF. Spatial complexity maps (SCMs) can be calculated from simple metrics (e.g. block variance, block luminance, SSIM, and edge detection). Importance maps with TCSF, TMVM, and SCM may be used to modify the standard rate-distortion optimization criterion for selecting the optimum encoding solution. Importance maps may modify encoder quantization. The spatial information for the importance maps may be provided by a lookup table based on block variance, where negative and positive spatial QP offsets for block variances are provided.

Claims

1. A method of encoding a plurality of video frames having non-overlapping target blocks, the method comprising:

encoding the plurality of video frames using importance maps, such that the importance maps modify quantization affecting encoding quality of each target block being encoded in each video frame, the importance maps being formed by;

configuring the importance maps using temporal information and spatial information; and

computationally causing the importance maps to indicate which parts of a video frame in the plurality of video frames are most noticeable to human perception, wherein;

(i) in target blocks where the importance maps take on high values that are higher than an average value in a value range of the importance map based on perceptual statistics, reducing a block quantization parameter (QP) of each high-value target block relative to a frame quantization parameter (QP_frame) of the video frame, resulting in increasing quality for the high-value target blocks, and

(ii) in target blocks where the importance maps take on low values that are lower than an average value in a value range of the importance map based on perceptual statistics, increasing a block quantization parameter (QP) of each low-value target block relative to the frame quantization parameter (QP_frame), resulting in decreasing quality for the low-value target blocks, and

(iii) representing each reduction in block QP of high-value target blocks or increase in block QP of the low-value target blocks in the importance map as a QP offset;

wherein the spatial information for the importance maps is provided by a lookup table based on block variance, the lookup table indicating spatial QP offsets including negative spatial QP offsets for block variances lower than 200 and positive spatial QP offsets for block variances above 400;

wherein the temporal information for the importance maps is provided by an algorithm that determines encoding importance of each target block of the video frame for inter-prediction in future video frames, the algorithm assigning the target blocks spatial QP offsets, including assigning high-value target blocks negative temporal QP offsets; and

wherein total QP offset for a given target block is equal to spatial QP offset of the given target block plus temporal QP offset of the given target block, clipped to maximum and minimum allowable QP values in the video frame.

2. The method as in claim 1, further comprising:

for each target block, adjusting block variance of the target block by edge strength, comprising;

calculating edge strength for each pixel in the target block by taking differences with neighboring pixels, and calculating edge strength for the target block by averaging the calculated edge strengths of each pixel in the target block;

classifying the target block as either a flat macroblock, a clean edge, a complex edge, or a complex macroblock based on the calculated edge strength for the target block, block variance for the target block, and the edge strength of neighboring target blocks;

if the target block is classified as a clean edge, adjusting block variance of the target block downward; and

if the target block is classified as a complex edge, adjusting block variance of the target block upward.

3. The method as in claim 2, further comprising:

setting the spatial QP offset of the target block to 0, if original block variance of the target block is greater than 400 and block variance of the target block adjusted based on edge strength is less than 200.

4. The method as in claim 2, further comprising:

reducing a positive spatial QP offset for the target block by a factor of 2, if the target block is classified as either a clean edge or a complex edge.

5. The method as in claim 1, wherein a minimum allowable QP in the video frame (QP_min) is determined based on complexity of content of the video frame, tightness of bit budget, and frame type.

6. The method as in claim 1, further comprising:

setting an initial estimate of block QP for each target block (QP_block-init) to 28 for I-frames, 30 for P-frames, and 33 for B-frames, if block variance of the target block is less than 60 and to QP_frame, if the block variance of the target block is greater than or equal to 60.

7. The method as in claim 6, further comprising:

calculating a final block QP of each target block (QP_block) by summing QP_block-init and the total QP offset of the target block; and

if the calculated QP_block is less than QP_min, setting QP_block to QP_min.

8. The method as in claim 1, further comprising:

applying luminance level matching to the video frame by considering an intra encoding mode involving a decision of a skip mode;

comparing rate-distortion cost of the intra mode and rate-distortion cost of the skip mode; and

if the rate-distortion cost of the intra mode is lower than the rate-distortion cost of the skip mode, choosing the intra mode.

9. A computer system encoding a plurality of video frames having non-overlapping target blocks, the computer system comprising:

at least one processor executing an encoder;

the encoder encoding the plurality of video frames using importance maps, such that the importance maps modify quantization affecting encoding quality of each target block being encoded in each video frame, the importance maps being formed by;

configuring the importance maps using temporal information and spatial information; and

computationally causing the importance maps to indicate which parts of a video frame in the plurality of video frames are most noticeable to human perception, wherein;

(i) in target blocks where the importance maps take on high values that are higher than an average value in a value range of the importance map based on perceptual statistics, reducing a block quantization parameter (QP) of each high-value target block relative to a frame quantization parameter (QP_frame) of the video frame, resulting in increasing quality for the high-value target blocks,

(ii) in target blocks where the importance maps take on low values that are lower than an average value in a value range of the importance map based on perceptual statistics, increasing a block quantization parameter (QP) of each low-value target block relative to the frame quantization parameter (QP_frame), resulting in decreasing quality for the low-value target blocks, and

(iii) representing each reduction in block QP of high-value target blocks or increase in block QP of the low-value target blocks in the importance map as a QP offset;

wherein the spatial information for the importance maps is provided by a lookup table based on block variance, the lookup table indicating spatial QP offsets, including negative spatial QP offsets for block variances lower than 200 and positive spatial QP offsets for block variances above 400;

wherein the temporal information for the importance maps is provided by an algorithm that determines encoding importance of each target block of the video frame for inter-prediction in future video frames, the algorithm assigning target blocks spatial QP offsets, including assigning high-value target blocks negative temporal QP offsets; and

wherein total QP offset for a given target block is equal to spatial QP offset of the given target block plus temporal QP offset of the given target block, clipped to the maximum and minimum allowable QP values in the video frame.

10. The computer system as in claim 9, wherein, for each target block, the encoder adjusts block variance by edge strength by;

calculating edge strength for each pixel in the target block by taking differences with neighboring pixels, and calculating edge strength for the target block by averaging the calculated edge strengths of each pixel in the target block;

classifying the target block as either a flat macroblock, a clean edge, a complex edge, or a complex macroblock based on the calculated edge strength for the target block, block variance for the target block, and the edge strength of neighboring target blocks;

if the target block is classified as a clean edge, adjusting block variance of the target block downward; and

if the target block is classified as a complex edge, adjusting block variance of the target block upward.

11. The computer system as in claim 10, wherein the encoder sets the spatial QP offset of the target block to 0 if original block variance of the target block is greater than 400 and block variance of the target block adjusted based on edge strength is less than 200.

12. The computer system as in claim 10, wherein the encoder reduces a positive spatial QP offset for the target block by a factor of 2, if the target block is classified as either a clean edge or a complex edge.

13. The computer system as in claim 9, wherein the encoder determines a minimum allowable QP in the video frame (QP_min) based on complexity of content of the video frame, tightness of bit budget, and frame type.

14. The computer system as in claim 9, wherein the encoder sets an initial estimate of block QP for each target block (QP_block-init) to 28 for I-frames, 30 for P-frames, and 33 for B-frames, if block variance of the target block is less than 60 and to QP_frame, if the block variance of the target block is greater than or equal to 60.

15. The computer system as in claim 14, wherein the encoder calculates a final block QP of each target block (QP_block) by summing QP_block-init and the total QP offset of the target block, and

if the calculated QP_block is less than QP_min, the encoder sets QP_block to QP_min.

16. The computer system as in claim 9, wherein the encoder choses an intra encoding mode by:

applying luminance level matching by considering an intra encoding mode involving a decision of a skip mode;

comparing rate-distortion cost of the intra mode and rate-distortion cost of the skip mode; and

if the rate-distortion cost of the intra mode is lower than the rate-distortion cost of the skip mode, choosing the intra encoding mode.

17. A computer program product having computer readable program code stored on a non-transitory storage medium, the computer readable program code causing a plurality of video frames having non-overlapping target blocks to be encoded, the computer comprising:

the computer readable program code implementing an encoder encoding the plurality of video frames using importance maps, such that the importance maps modify quantization affecting encoding quality of each target block to be encoded in each video frame, the importance maps being formed by the encoder;

configuring the importance maps using temporal information and spatial information; and

computationally causing the importance maps to indicate which parts of a video frame in the plurality of video frames are most noticeable to human perception, wherein;

(i) in target blocks where the importance maps take on high values that are higher than an average value in a value range of the importance map based on perceptual statistics, reducing a block quantization parameter (QP) of each high-value target block relative to a frame quantization parameter (QP_frame), resulting in increasing quality for the high-value target blocks, and

(ii) in target blocks where the importance maps take on low values that are lower than an average value in a value range of the importance map based on perceptual statistics, increasing a block quantization parameter (QP) of each low-value target block relative to the frame quantization parameter (QP_frame), resulting in decreasing quality for the low-value target blocks, and

(iii) representing each reduction in block QP of high-value target blocks or increase in block QP of the low-value target blocks in the importance map as a QP offset;

wherein the spatial information for the importance maps is provided by a lookup table based on block variance, the lookup table indicating spatial QP offsets including negative spatial QP offsets for block variances lower than 200 and positive spatial QP offsets for block variances above 400;

wherein the temporal information for the importance maps is provided by an algorithm that determines encoding importance of each target block of the video frame for inter-prediction in future video frames, the algorithm assigning the target blocks spatial QP offsets, including assigning high-value target blocks negative temporal QP offsets; and

wherein total QP offset for a given target block is equal to spatial QP offset of the given target block plus temporal QP offset of the given target block, clipped to maximum and minimum allowable QP values in the video frame.