Video Compression and Transmission Techniques

• US 20090086816A1
• Filed: 09/08/2008
• Published: 04/02/2009
• Est. Priority Date: 09/28/2007
• Status: Active Grant

• Current Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Original Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Inventors: Tourapis, Alexandros, Leontaris, Athanasios
• Granted Patent: US 9,445,110 B2
• Time in Patent Office: Days
• US Class Current: 375/240.30
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/126 Details of normalisation or weighting functions, e.g. normalisation matrices or variable uniform quantisers
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/142 Detection of scene cut or scene change
• CPC Class Codes: H04N 19/149 by estimating the code amount by means of a model, e.g. mathematical model or statistical model
• CPC Class Codes: H04N 19/15 by monitoring actual compressed data size at the memory before deciding storage at the transmission buffer
• CPC Class Codes: H04N 19/152 by measuring the fullness of the transmission buffer
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/159 Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/174 the region being a slice, e.g. a line of blocks or a group of blocks
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/194 involving only two passes
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/615 using motion compensated temporal filtering [MCTF]
• CPC Class Codes: H04N 19/80 Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence

Abstract

Embodiments feature families of rate allocation and rate control methods that utilize advanced processing of past and future frame/field picture statistics and are designed to operate with one or more coding passes. At least two method families include: a family of methods for a rate allocation with picture look-ahead; and a family of methods for average bit rate (ABR) control methods. At least two other methods for each method family are described. For the first family of methods, some methods may involve intra rate control. For the second family of methods, some methods may involve high complexity ABR control and/or low complexity ABR control. These and other embodiments can involve any of the following: spatial coding parameter adaptation, coding prediction, complexity processing, complexity estimation, complexity filtering, bit rate considerations, quality considerations, coding parameter allocation, and/or hierarchical prediction structures, among others.

Claims

1. A method for estimating a complexity of a picture, the method comprising:

receiving a metric of a complexity of a picture generated from a motion-compensated processor or analyzer, a motion compensator, a spatial processor, a filter, or from a result generated from a previous coding pass, wherein the complexity comprises a temporal, a spatial, or a luminance characteristic; and

estimating the metric of the complexity of the picture comprising;

determining if the picture is correlated with a future or past picture; and

determining if the picture or an area of the picture masks artifacts more effectively than areas of the picture or the future or past picture that do not mask the artifacts.

2. The method of claim 1, further comprising providing local complexity information for a picture via sending information for blocks to be considered in isolation.

3. The method of claim 1, further comprising providing global complexity information for a picture via sending information for blocks to be accumulated for the picture.

4. The method of claim 1, further comprising coding the picture with an encoder, where the encoder is configured to determine a complexity measure using any combination of edge information or the temporal, spatial, or luminance characteristics.

5. The method of claim 1, further comprising determining a sensitivity of one or more regions of the picture to the compression artifacts;

and, if lower quality is determined, coding the picture or the area of the picture to increase the picture quality.

6. The method of claim 1, further comprising:

selecting one or more coding parameters for the picture; and

distributing the coding parameters within the picture.

7. The method of claim 1, wherein the coding parameters comprise quantization parameters, wherein the method further estimates a complexity of any of a slice or a frame.

8. The method of claim 1, wherein determining if the picture or the area of the picture masks artifacts more effectively than areas of the picture or the future or past picture that do not mask the artifacts comprises:

using coding statistics to compare masking artifacts in the area of the picture with at least one other area of the picture, at least one other area of a past picture, or at least one other area of a future picture, or

using coding statistics to compare masking artifacts in the picture with the past picture or the future picture; and

estimating the metric for the complexity using the coding statistics.

9. A method to generate high quality coded video, the method comprising:

assessing complexity information between video pictures, wherein the complexity information comprises temporal, or spatial information, wherein the video pictures comprise video frames;

using the complexity information to determine a frame type and to analyze parameters, wherein the parameters comprise parameters for scene changes, fade-ins, fade-outs, cross fades, local illumination changes, camera pan, or camera zoom; and

filtering an amount of statistics or complexity information between the video frames by using the analyzed parameters to remove outliers and avoid abrupt fluctuations in the parameters or video quality between the video frames.

10. The method of claim 9, further comprising using a lookahead window with encoded video for transcoding the video.

11. The method of claim 9, further comprising reducing the complexity information via spatial or temporal downsampling;

wherein any combination of the statistics or complexity are used to determine an abruptness of fluctuations in the coding parameters or video quality.

12. A method of estimating complexity for pictures, the method comprising:

determining if the pictures are to be coded in a hierarchical structure, wherein the hierarchical structure comprises multiple picture levels, and wherein the hierarchical structure further comprises bits or coding parameters at different picture levels; and

upon the determination that a picture is assigned to a certain hierarchical level, coding a picture based on an importance of the picture, wherein the coding comprises controlling a quality level of the picture, and wherein the coding comprises varying at least one of the coding parameters of the picture based on the importance.

13. The method of claim 12, wherein the coding parameters comprise quantization parameters, the method further comprising allocating the bits or quantization parameters for a picture by segmenting regions of a scene in the picture, wherein the regions comprise simple regions, complex regions, or regions that include information that is important for coding other pictures, and wherein the method further estimates complexity for slices or frames.

14. The method of claim 13, further comprising estimating complexity employing multiple metrics or metrics in parallel, wherein the metrics comprise a Mean Squared Error (MSE), a Summed Squared Error (SSE), a Mean Absolute Difference (MAD), or a Summed Absolute Difference (SAD).

15. A method to code video data, the method comprising:

coding parameters for the video data on a macroblock basis, wherein the coding comprises accounting for variations in spatial and temporal statistics;

generating a complexity measure;

determining an importance of the complexity measure;

mapping the complexity measure to a coding parameter set; and

using the complexity measure to adjust the coding parameter set to increase a level of quality to the video data by making an image region in the video data more or less important in the video data.

16. The method of claim 15, wherein the complexity measure comprises any of spatial measures, temporal measures, edge measures, luminance measures, chrominance measures, or texture measures.

17. A method for encoding a picture, the method comprising:

receiving a current frame;

setting a bit rate target and a number of bits for the current frame;

determining a plurality of complexities for the picture, wherein the determining complexities comprises;

determining, in parallel, coding parameters for respective complexities; and

after the coding parameters are determined for respective complexities, coding respective pictures using the respective complexities;

selecting a final coded picture from the coded respective pictures; and

updating the plurality of complexities using the final coded picture selection.

18. The method of claim 17, wherein the plurality of complexities comprises N complexities, wherein a coding parameter set is determined for each of the N complexities.

19. The method of claim 17, further comprising increasing a picture quality level by using the respective complexities with picture look-ahead information.

20. A rate allocation method for video, the method comprising:

receiving information for a picture look-ahead buffer;

in a first coding pass, performing rate allocation to set a bit target for a picture, the rate allocation comprising utilizing the picture look-ahead buffer to determine a complexity for the picture; and

selecting a coding parameter set for the bit target using a rate control model.

21. The method of claim 20, wherein the complexity comprises a motion-compensated sum of absolute differences (SAD), a SAD with weighting prediction, or a combination of temporal and spatial picture statistics.

22. The method of claim 20, further comprising determining a total complexity with a summation of complexities of a number of pictures, or a combination of linear or non-linear complexities of a number of pictures, wherein the summation comprises a weighted summation.

23. The method of claim 20, further comprising determining the complexity by determining if the picture is a predictive picture or a non-predictive picture.

24. The method of claim 20, further comprising determining a number of bits allocated to the picture with a function that considers a number of bits remaining in a total bit budget.

25. The method of claim 20, further comprising determining a total complexity with a weighted summation of complexities of a number of pictures, or determining the complexity by determining if the picture is a intra-picture prediction or a inter-picture prediction, wherein the coding parameter set comprises a quantization parameter.

26. A method for rate allocation for video coding, the method comprising:

initializing a quantization parameter and a number of remaining bits for a picture;

determining a total complexity for picture look-ahead frames;

determining a slice type for the picture comprising an I-coded picture, a P-coded picture, or a periodic I-coded picture, wherein the determining comprises;

for the I-coded picture, determining a number of bits allocated to an inter-coded picture, and employing a first rate control model to use the quantization parameter to code the picture;

for the P-coded frame, determining a number of bits allocated to a predictive coded picture, and employing a second rate control model using the quantization parameter to code the picture; and

for the periodic I-coded picture, using a previous quantization parameter to code the picture; and

after the slice type is determined, coding the picture for the determined slice type.

27. The method of claim 26, further comprising:

assessing a proximity to a picture bit target for coded picture bits; and

modifying the quantization parameter if coded picture bits are not close to the proximity of the picture bit target.

28. The method of claim 26, further comprising determining a number of bits for encoded intra-coded pictures before determining the slice type for the picture.

29. The method of claim 26, further comprising:

determining if there are other pictures to code; and

if there are other pictures to code,

updating the rate control model based on the selected slice type; and

updating the previous quantization parameter and the number of bits remaining.

30. A rate control method for video coding, the rate control method comprising:

receiving coding statistics for previous pictures in a video system;

receiving look-ahead information for future pictures;

using a coding parameter set to code a current picture, wherein the coding parameter set comprises coding parameters, wherein the coding parameters comprise a base coding parameter set and a modifier to achieve a target bit rate for the previous pictures and the current picture, wherein the current and previous pictures comprise weights to adjust picture quality and bit rate allocation; and

adjusting the weights to modify the picture quality of the current and previous pictures, wherein the picture quality is dependent on a rate factor for the quantization parameter, and wherein the adjustment of the weights modifies the bit rate allocation.

31. The method of claim 30, further comprising employing a video coder with complexity measures, wherein the complexity measures employ the look-ahead information for the future pictures and past coding statistics for the previous pictures to adjust the picture quality, further comprising employing a video coder using past coding statistics to adjust the picture quality, wherein the coding parameter set comprises a quantization parameter (QP).

32. A method for rate control, comprising:

initializing values for a set of coding parameters and a rate factor, wherein the set of coding parameters comprises a quantization parameter;

determining a bit target, a number of bits used, and a coding parameter modifier;

determining the rate factor with the bit target;

determining a slice type comprising a level-greater-than-zero frame, a predicted coded frame at level zero, an intra coded frame at level zero, and a periodic intra coded frame at level zero; and

selecting the determined slice type.

33. The method of claim 32, further comprising:

for the level-greater-than-zero frame, determining a previous quantization parameter value and a modifier of a coding parameter before coding the frame;

for the predicted coded frame at level zero or the intra coded frame at level zero, determining a quantization parameter as a function of a complexity and the rate factor before coding the frame; and

for the periodic intra coded frame at level zero, using the previous quantization parameter before coding the frame.

34. A method for rate control, comprising:

initializing values for a coding parameter and a rate factor;

determining a bit target, a number of bits used, and an overflow parameter;

determining the rate factor with a function of the bit target and a sum of a complexity;

determining a slice type comprising a level-greater-than-zero frame, a predicted coded frame at level zero, an intra coded frame at level zero, and a periodic intra coded frame at level zero; and

selecting the determined slice type.

35. The method of claim 34, further comprising:

for the level-greater-than-zero frame, determining a previous quantization parameter value and a modifier of a coding parameter before coding the frame;

for the predicted coded frame at level zero or the intra coded frame at level zero, determining a quantization parameter as a function of a complexity and the rate factor before coding the frame; and

for the periodic intra coded frame at level zero, using the previous quantization parameter before coding the frame.

36. The method of claim 34, wherein the coding parameter comprises a quantization parameter.

37. The method of claim 35, wherein the method further comprises an estimation of complexity.

38. A computer program product, encoded on a computer-readable medium, operable to cause data processing apparatus to perform operations for estimating a complexity of a picture, comprising:

receiving a metric of a complexity of a picture generated from a motion-compensated processor or analyzer, a motion compensator, a spatial processor, a filter, or from a result generated from a previous coding pass, wherein the complexity comprises a temporal, spatial, or luminance characteristic; and

estimating the metric of the complexity of the picture comprising;

determining if the picture is correlated with a future or past picture; and

determining if the picture or an area of the picture masks artifacts more effectively than areas of the picture or the future or past picture that do not mask the artifacts.

39. A computer program product, encoded on a computer-readable medium, operable to cause data processing apparatus to perform operations for generating high quality coded video, comprising:

assessing complexity information between video pictures, wherein the complexity information comprises temporal, spatial, or luminance information, wherein the video pictures comprise video frames;

using the complexity information to determine a frame type and to analyze parameters, wherein the parameters comprise parameters for scene changes, fade-ins, fade-outs, cross fades, local illumination changes, camera pan, or camera zoom; and

filtering an amount of statistics or complexity information between the video frames by using the analyzed parameters to remove outliers and avoid abrupt fluctuations in the parameters or video quality between the video frames.

40. A computer program product, encoded on a computer-readable medium, operable to cause data processing apparatus to perform operations for estimating complexity for pictures, comprising:

determining if the pictures are to be coded in a hierarchical structure, wherein the hierarchical structure comprises multiple picture levels, and wherein the hierarchical structure further comprises bits or coding parameters at different picture levels; and

upon the determination that a picture is assigned to a certain hierarchical level, coding a picture based on an importance of the picture, wherein the coding comprises controlling a quality level of the picture, and wherein the coding comprises varying at least one of the coding parameters of the picture based on the importance.

41. A computer program product, encoded on a computer-readable medium, operable to cause data processing apparatus to perform operations for coding video data, comprising:

coding parameters for the video data on a macroblock basis, wherein the coding comprises accounting for variations in spatial and temporal statistics;

generating a complexity measure;

determining an importance of the complexity measure;

mapping the complexity measure to a coding parameter set; and

using the complexity measure to adjust the coding parameter set to increase a level of quality to the video data by making an image region in the video data more or less important in the video data.

42. A computer program product, encoded on a computer-readable medium, operable to cause data processing apparatus to perform operations for encoding a picture, comprising:

receiving a current frame;

setting a bit rate target and a number of bits for the current frame;

determining a plurality of complexities for the picture, wherein the determining complexities comprises;

determining, in parallel, coding parameters for respective complexities; and

after the coding parameters are determined for respective complexities, coding respective pictures using the respective complexities;

selecting a final coded picture from the coded respective pictures; and

updating the plurality of complexities using the final coded picture selection.

43. A computer program product, encoded on a computer-readable medium, operable to cause data processing apparatus to perform operations for rate allocation for video, comprising:

receiving information for a picture look-ahead buffer;

in a first coding pass, performing rate allocation to set a bit target for a picture, the rate allocation comprising utilizing the picture look-ahead buffer to determine a complexity for the picture; and

selecting a coding parameter set for the bit target using a rate control model.

44. A computer program product, encoded on a computer-readable medium, operable to cause data processing apparatus to perform operations for rate allocation for video coding, comprising:

initializing a quantization parameter and a number of remaining bits for a picture;

determining a total complexity for picture look-ahead frames;

determining a slice type for the picture comprising an I-coded picture, a P-coded picture, or a periodic I-coded picture, wherein the determining comprises;

for the I-coded picture, determining a number of bits allocated to an inter-coded picture, and employing a first rate control model to use the quantization parameter to code the picture;

for the P-coded frame, determining a number of bits allocated to a predictive coded picture, and employing a second rate control model using the quantization parameter to code the picture; and

for the periodic I-coded picture, using a previous quantization parameter to code the picture; and

after the slice type is determined, coding the picture for the determined slice type.

45. A computer program product, encoded on a computer-readable medium, operable to cause data processing apparatus to perform operations for rate control for video coding, comprising:

receiving coding statistics for previous pictures in a video system;

receiving look-ahead information for future pictures;

using a coding parameter set to code a current picture, wherein the coding parameter set comprises coding parameters, wherein the coding parameters comprise a base coding parameter set and a modifier to achieve a target bit rate for the previous pictures and the current picture, wherein the current and previous pictures comprise weights to adjust picture quality and bit rate allocation; and

adjusting the weights to modify the picture quality of the current and previous pictures, wherein the picture quality is dependent on a rate factor for the quantization parameter, and wherein the adjustment of the weights modifies the bit rate allocation.

46. A computer program product, encoded on a computer-readable medium, operable to cause data processing apparatus to perform operations for rate control, comprising:

initializing values for a set of coding parameters and a rate factor, wherein the set of coding parameters comprises a quantization parameter;

determining a bit target, a number of bits used, and a coding parameter modifier;

determining the rate factor with the bit target;

determining a slice type comprising a level-greater-than-zero frame, a predicted coded frame at level zero, an intra coded frame at level zero, and a periodic intra coded frame at level zero; and

selecting a frame to code for the determined slice type.

47. A system comprising one or more devices for video processing to:

receive a metric of a complexity of a picture generated from a motion-compensated processor or analyzer, a motion compensator, a spatial processor, a filter, or from a result generated from a previous coding pass, wherein the complexity comprises a temporal, spatial, or luminance characteristic; and

estimate the metric of the complexity of the picture comprising;

determine if the picture is correlated with a future or past picture; and

determine if the picture or an area of the picture masks artifacts more effectively than areas of the picture or the future or past picture that do not mask the artifacts.

48. A system comprising one or more devices to:

assess complexity information between video pictures, wherein the complexity information comprises temporal, spatial, or luminance information, wherein the video pictures comprise video frames;

use the complexity information to determine a frame type and to analyze parameters, wherein the parameters comprise parameters for scene changes, fade-ins, fade-outs, cross fades, local illumination changes, camera pan, or camera zoom; and

filtering an amount of statistics or complexity information between the video frames by using the analyzed parameters to remove outliers and avoid abrupt fluctuations in parameters or video quality between the video frames.

49. A system comprising one or more devices to:

determining if the pictures are to be coded in a hierarchical structure, wherein the hierarchical structure comprises multiple picture levels, and wherein the hierarchical structure further comprises bits or coding parameters at different picture levels; and

upon the determination that a picture is assigned to a certain hierarchical level, coding a picture based on an importance of the picture, wherein the coding comprises controlling a quality level of the picture, and wherein the coding comprises varying at least one of the coding parameters of the picture based on the importance.

50. A system comprising one or more devices for video processing of pictures operable to:

code parameters for video data on a macroblock basis, wherein the coding comprises accounting for variations in spatial and temporal statistics;

generate a complexity measure;

determine an importance of the complexity measure;

map the complexity measure to a coding parameter set; and

use the complexity measure to adjust the coding parameter set to increase a level of quality to the video data by making an image region in the video data more or less important in the video data.

51. A system comprising one or more devices to:

receive a current frame;

set a bit rate target and a number of bits for the current frame;

determine a plurality of complexities for a picture, wherein the determining complexities comprises;

determine, in parallel, coding parameters for respective complexities; and

after the coding parameters are determined for respective complexities, code respective pictures using the respective complexities;

select a final coded picture from the coded respective pictures; and

update the plurality of complexities using the final coded picture selection.

52. A system comprising one or more devices to:

receive information for a picture look-ahead buffer;

in a first coding pass, perform rate allocation to set a bit target for a picture, the rate allocation comprising utilizing the picture look-ahead buffer to determine a complexity for the picture; and

select a coding parameter set for the bit target using a rate control model.

53. A system comprising one or more devices to:

initialize a quantization parameter and a number of remaining bits for a picture;

determine a total complexity for picture look-ahead frames;

determine a slice type for the picture comprising an I-coded picture, a P-coded picture, or a periodic I-coded picture, wherein the determining comprises;

for the I-coded picture, determine a number of bits allocated to an inter-coded picture, and employ a first rate control model to use the quantization parameter to code the picture;

for the P-coded frame, determine a number of bits allocated to a predictive coded picture, and employ a second rate control model using the quantization parameter to code the picture; and

for the periodic I-coded picture, use a previous quantization parameter to code the picture; and

after the slice type is determined, code the picture for the determined slice type.

54. A system comprising one or more devices operable perform rate control of video data, the one or more devices are configured to:

receive coding statistics for previous pictures in a video system;

receive look-ahead information for future pictures;

use a coding parameter set to code a current picture, wherein the coding parameter set comprises coding parameters, wherein the coding parameters comprise a base coding parameter set and a modifier to achieve a target bit rate for the previous pictures and the current picture, wherein the current and previous pictures comprise weights to adjust picture quality and bit rate allocation; and

adjust the weights to modify the picture quality of the current and previous pictures, wherein the picture quality is dependent on a rate factor for the quantization parameter, and wherein the adjustment of the weights modifies the bit rate allocation.

55. A system comprising one or more devices operable for rate control of video, the one or more devices are configured to:

initialize values for a set of coding parameters and a rate factor;

determine a bit target, a number of bits used, and a coding parameter modifier, wherein the set of coding parameters comprises a quantization parameter;

determine the rate factor with the bit target;

determine a slice type comprising a level-greater-than-zero frame, a predicted coded frame at level zero, an intra coded frame at level zero, and a periodic intra coded frame at level zero; and

select a frame to code for the determined slice type.