H.264/AVC coder incorporating rate and quality controller

• US 7,986,731 B2
• Filed: 03/30/2004
• Issued: 07/26/2011
• Est. Priority Date: 02/06/2004
• Status: Expired due to Fees

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Wu, Hsi-Jung, Dumitras, Adriana, Normile, James, Haskell, Barin, Puri, Atul, Nie, Xiaochun
• Primary Examiner(s): Czekaj; David
• Assistant Examiner(s): HOLDER, ANNER N
• Application Number:
  US10/811,983
• Publication Number:
  US 20050175092A1
• Time in Patent Office: 2,674 Days
• Field of Search: 375/240.03
• US Class Current: 375/240.03
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/142 Detection of scene cut or scene change
• CPC Class Codes: H04N 19/149 by estimating the code amount by means of a model, e.g. mathematical model or statistical model
• CPC Class Codes: H04N 19/152 by measuring the fullness of the transmission buffer
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/196 being specially adapted for the computation of encoding parameters, e.g. by averaging previously computed encoding parameters processing of motion vectors H04N19/513
• CPC Class Codes: H04N 19/198 including smoothing of a sequence of encoding parameters, e.g. by averaging, by choice of the maximum, minimum or median value
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/82 involving filtering within a prediction loop

Abstract

A rate control system is disclosed for video coding applications. The rate controller assigns a quantization parameter for video data in a picture in response to complexity indicators indicative of spatial complexity, motion complexity and/or bits per pel of the picture. A virtual buffer based quantizer parameter is proposed based on a virtual buffer fullness analysis and a target rate estimate, which is derived from the complexity indicators. A second quantizer parameter is proposed from a linear regression analysis of quantizer parameters used to code previously coded pictures of similar type (e.g., I pictures, P pictures or B pictures). A coding policy decision unit defines a final quantizer parameter from a comparison of the two proposed quantizer parameters.

Claims

1. Rate and quality control system for an AVC-based video coder, comprising:

a picture analyzer, to generate complexity indicators from each picture of an input video sequence;

a first quantizer estimator to generate a first quantizer estimate for each picture based on the complexity indicators, a target coding rate calculated for each picture and a transmit buffer fullness indicator representing a quantity of stored previously-coded video data;

a second quantizer estimator, to generate a second quantizer estimate for each picture, the second quantizer estimates for I and P pictures based on a linear regression analysis of quantizers and coding rates of previously-coded pictures;

a quantizer selector to generate a quantizer parameter for each picture from the first and second quantizer estimates; and

a coding policy unit operative according to a rate control policy, wherein the rate control policy is selected based at least on a comparison of the first and second quantizer estimates.

2. The rate and quality control system of claim 1, wherein the AVC video coder comprises:

an integer approximated transform circuit, to generate transform coefficients from input pixel data based on the input pictures,

a forward quantizer to divide the transform coefficients according to the quantizer parameter,

a transform scaler, coupled to the forward quantizer,

a forward scan unit, coupled to the transform scaler,

a variable length coder, coupled to the forward scan unit, and

a formatter, coupled to the variable length coder.

3. The rate and quality control system of claim 2, wherein the coding policy unit determines when it becomes necessary to eliminate non-zero quantized transform coefficients according to the rate control policy, and

wherein the AVC coder further comprises a coefficient zeroer provided between the forward quantizer and the transform scaler, responsive to control from the coding policy unit, to eliminate selected quantized transform coefficients.

4. The rate and quality control system of claim 2 wherein the coding policy unit determines when it becomes necessary to eliminate pictures from the video sequence from being coded according to the rate control policy, and

a video preprocessing unit, responsive to control from the coding policy unit, to perform frame decimation before pictures are input to the AVC coder.

5. The rate and quality control system of claim 2 wherein the coding policy unit determines when it becomes necessary to eliminate motion vectors according to the rate control policy, and

wherein the AVC coder includes a prediction circuit that generates motion vectors for prediction of video data of macroblocks in the input pictures and of video data for sub-blocks therein of various sizes, the prediction circuit responsive to control from the coding policy unit, to eliminate selected motion vectors from an output coded bitstream.

6. The rate and quality control system of claim 2, wherein

the AVC coder includes a prediction chain that decodes coded video output from the forward scan unit, the prediction chain including a deblocking filter, and

wherein the coding policy unit controls the deblocking filter .

7. The rate and quality control system of claim 6, wherein the coding policy unit calculates alpha and beta control parameters to be used by an H.264 deblocking filter.

8. The rate and quality control system of claim 2 wherein the AVC coder includes a prediction chain, comprising:

a spatial predictor that predicts video data for a block of input data according to intra prediction techniques,

a temporal predictor that predicts video data for the block of input data according to temporal predictions between a current picture and one or more previously coded reference frames, and

a mode selector that selects an output from one of the spatial predictor or the temporal predictor for each block of input data,

wherein the mode selector performs its selection based on mode decision control signals from the coding policy unit.

9. The rate and quality control system of claim 1, wherein the complexity indicator includes an indicator of spatial complexity within the picture.

10. The rate and quality control system of claim 1, wherein the complexity indicator includes an indicator of motion complexity of the picture with respect to previously coded pictures.

11. The rate and quality control system of claim 1, wherein the complexity indicator includes an indicator of a number of bits used to represent each pixel in the picture.

48. The rate and quality control system of claim 1 wherein, for P and B pictures, the target coding rate is a predetermined target coding rate if a scene change detection signal indicates a scene change.

12. Rate and quality control system for an AVC-based video coder, comprising:

a content characteristics and coding rate analyzer, responsive to pictures from an input video sequence, to generate complexity indicators representative thereof,

a target bits computer, responsive to the complexity indicators and to a picture type signal, to calculate a target coding rate for each picture in the video sequence,

a buffer based quantizer computer, responsive to the target coding rates, to a transmit buffer indicator signal and to the picture type signal, to generate a buffer-based quantizer estimate for each picture, and

an activity based quantizer computer to calculate activity of each picture in the video sequence and modify the buffer-based quantizer estimate in response thereto,

an AVC coder including a forward quantizer operative according to the modified buffer-based quantizer estimate.

13. The rate and quality control system of claim 12, wherein the complexity indicator includes an indicator of spatial complexity within the picture.

14. The rate and quality control system of claim 12, wherein the complexity indicator includes an indicator of motion complexity of the picture with respect to previously coded pictures.

15. The rate and quality control system of claim 12, wherein the complexity indicator includes an indicator of a number of bits used to represent each pixel in the picture.

16. The rate and quality control system of claim 12, wherein the AVC coder comprises:

an integer approximated transform circuit, to generate transform coefficients from input pixel data,

the forward quantizer to divide the transform coefficients according to the modified buffer-based quantizer estimate,

a transform scaler, coupled to the forward quantizer,

a forward scan unit, coupled to the transform scaler,

a variable length coder, coupled to the forward scan unit, and

a formatter, coupled to the variable length coder.

17. The rate and quality control system of claim 16, further comprising

a coding policy unit, to determine when it becomes necessary to eliminate non-zero quantized transform coefficients according to a rate control policy, and

wherein the AVC coder further comprises a coefficient zeroer provided between the forward quantizer and the transform scaler, responsive to control from the coding policy unit, to eliminate selected quantized transform coefficients.

18. The rate and quality control system of claim 16, further comprising

a coding policy unit, to determine when it becomes necessary to eliminate pictures from the video sequence from being coded according to a rate control policy, and

a video preprocessing unit, responsive to control from the coding policy unit, to perform frame decimation before pictures are input to the AVC coder.

19. The rate and quality control system of claim 16, further comprising

a coding policy unit, to determine when it becomes necessary to eliminate motion vectors according to a rate control policy, and

wherein the AVC coder includes a prediction circuit that generates motion vectors for prediction of video data of macroblocks in the input pictures and of video data for sub-blocks therein of various sizes, the prediction circuit responsive to control from the coding policy unit, to eliminate selected motion vectors from an output coded bitstream.

20. The rate and quality control system of claim 16, wherein

the AVC coder includes a prediction chain that decodes coded video output from the forward scan unit, the prediction chain including a deblocking filter, and

the rate and quality control system further comprises a coding policy unit, to control the deblocking filter according to a rate control policy.

21. The rate and quality control system of claim 20, wherein the coding policy unit calculates alpha and beta control parameters to be used by an H.264 deblocking filter.

22. The rate and quality control system of claim 16, further comprising

a coding policy unit operative according to a rate control policy, and

wherein the AVC coder includes a prediction chain, comprising:

a spatial predictor that predicts video data for a block of input data according to intra prediction techniques,

a temporal predictor that predicts video data for the block of input data according to temporal predictions between a current picture and one or more previously coded reference frames, and

a mode selector that selects an output from one of the spatial predictor or the temporal predictor for each block of input data,

wherein the mode selector performs its selection based on mode decision control signals from the coding policy unit.

49. The rate and quality control system of claim 12 wherein the target bits computer is further responsive to a scene detection signal, and wherein, for P and B pictures, the target coding rate is a predetermined target coding rate if the scene change detection signal indicates a scene change.

23. Rate and quality control system for an AVC-based video coder, comprising:

a content characteristics and coding rate analyzer, responsive to pictures from an input video sequence, to generate complexity indicators representative thereof,

a rate model quantizer estimator, responsive to quantizers and coding rates of previously-coded pictures and to picture type indicators of input pictures, to estimate quantizer parameters of the input pictures according to a linear regression analysis, wherein linear regression coefficients of input I pictures are selected according to the complexity indicators for such I pictures,

an AVC coder including a forward quantizer operative according to the quantizer estimates.

24. The rate and quality control system of claim 23, wherein the complexity indicator includes an indicator of spatial complexity within the picture.

25. The rate and quality control system of claim 23, wherein the complexity indicator includes an indicator of motion complexity of the picture with respect to previously coded pictures.

26. The rate and quality control system of claim 23, wherein the complexity indicator includes an indicator of a number of bits used to represent each pixel in the picture.

27. The rate and quality control system of claim 23, wherein the AVC coder comprises:

an integer approximated transform circuit, to generate transform coefficients from input pixel data,

the forward quantizer to divide the transform coefficients according to the modified buffer-based quantizer estimate,

a transform scaler, coupled to the forward quantizer,

a forward scan unit, coupled to the transform scaler,

a variable length coder, coupled to the forward scan unit, and

a formatter, coupled to the variable length coder.

28. The rate and quality control system of claim 27, further comprising

a coding policy unit, to determine when it becomes necessary to eliminate non-zero quantized transform coefficients according to a rate control policy, and

wherein the AVC coder further comprises a coefficient zeroer provided between the forward quantizer and the transform scaler, responsive to control from the coding policy unit, to eliminate selected quantized transform coefficients.

29. The rate and quality control system of claim 27, further comprising

a coding policy unit, to determine when it becomes necessary to eliminate pictures from the video sequence from being coded according to a rate control policy, and

a video preprocessing unit, responsive to control from the coding policy unit, to perform frame decimation before pictures are input to the AVC coder.

30. The rate and quality control system of claim 27, further comprising

a coding policy unit, to determine when it becomes necessary to eliminate motion vectors according to a rate control policy, and

wherein the AVC coder includes a prediction circuit that generates motion vectors for prediction of video data of macroblocks in the input pictures and of video data for sub-blocks therein of various sizes, the prediction circuit responsive to control from the coding policy unit, to eliminate selected motion vectors from an output coded bitstream.

31. The rate and quality control system of claim 27, wherein

the AVC coder includes a prediction chain that decodes coded video output from the forward scan unit, the prediction chain including a deblocking filter, and

the rate and quality control system further comprises a coding policy unit, to control the deblocking filter according to a rate control policy.

32. The rate and quality control system of claim 31, wherein the coding policy unit calculates alpha and beta control parameters to be used by an H.264 deblocking filter.

33. The rate and quality control system of claim 27, further comprising

a coding policy unit operative according to a rate control policy, and

wherein the AVC coder includes a prediction chain, comprising:

a spatial predictor that predicts video data for a block of input data according to intra prediction techniques,

a temporal predictor that predicts video data for the block of input data according to temporal predictions between a current picture and one or more previously coded reference frames, and

a mode selector that selects an output from one of the spatial predictor or the temporal predictor for each block of input data,

wherein the mode selector performs its selection based on mode decision control signals from the coding policy unit.

34. A video coding system, comprising:

a rate controller having an input coupled to a source of video data and generating a quantizer selection on a picture-by-picture basis, the selection based on a first quantizer estimate, said first quantizer estimate based on a measure of complexity for a picture, an assigned coding type for the picture, a target coding rate for the picture, or a measure of fullness of a transmit buffer, and a second quantizer estimate, said second quantizer estimate based on a linear regression analysis of quantizers and coding rates of previously coded pictures,

a video prediction chain to generate predicted video data on a block-by-block basis,

a block-based video coding chain including;

a subtractor coupled to the source video data and to the video prediction chain,

a transform circuit, to receive data output from the subtractor, and

a quantizer to receive data output from the transform circuit, the quantizer operative according to a quantizer parameter output from the rate controller,

wherein the video coding chain deletes motion vectors under control of the rate controller.

35. The video coding system of claim 34, wherein the video coding chain further deletes transform coefficients under control of the rate controller.

36. The video coding system of claim 34, wherein the video prediction chain comprises a deblocking filter whose mode of operation is controlled by the rate controller.

37. The video coding system of claim 34, wherein video prediction chain comprises a prediction mode decision unit whose mode of operation is controlled by the rate controller.

38. The video coding system of claim 34, further comprising a video preprocessor that performs picture decimation under control of the rate controller.

39. A method of controlling the rate and quality for an AVC-based video coder, comprising:

generating complexity indicators from each picture of an input video sequence;

generating a first quantizer estimate for each picture based on the complexity indicators, a target coding rate calculated for each picture and a transmit buffer fullness indicator representing a quantity of stored previously-coded video data;

generating a second quantizer estimate for each picture, the second quantizer estimates for I and P pictures based on a linear regression analysis of quantizers and coding rates of previously-coded pictures;

generating a quantizer parameter for each picture from the first and second quantizer estimates

determining a rate control policy, wherein the rate control policy is selected based at least on a comparison of the first and second quantizer estimates.

40. The method of claim 39, further comprising:

determining when it becomes necessary to eliminate non-zero quantized transform coefficients according to the rate control policy, and

eliminating selected quantized transform coefficients.

41. The method of claim 39, further comprising:

determining when it becomes necessary to eliminate pictures from the video sequence from being coded according to the rate control policy, and

performing frame decimation before pictures are coded.

42. The method of claim 39, further comprising:

generating motion vectors for prediction of video data of macroblocks in the input pictures and of video data for sub-blocks therein of various sizes,

determining when it becomes necessary to eliminate motion vectors according to the rate control policy, and

eliminating selected motion vectors from an output coded bitstream.

50. The method of claim 39 wherein, for P and B pictures, the target coding rate is a predetermined target coding rate if a scene change detection signal indicates a scene change.

43. A non-transitory computer-readable medium encoded with a set of instructions which, when performed by a computer, perform a method of controlling the rate and quality for an AVC-based video coder, said method comprising:

generating complexity indicators from each picture of an input video sequence;

generating a first quantizer estimate for each picture based on the complexity indicators, a target coding rate calculated for each picture and a transmit buffer fullness indicator representing a quantity of stored previously-coded video data;

generating a second quantizer estimate for each picture, the second quantizer estimates for I and P pictures based on a linear regression analysis of quantizers and coding rates of previously-coded pictures;

generating a quantizer parameter for each picture from the first and second quantizer estimates

determining a rate control policy, wherein the rate control policy is selected based at least on a comparison of the first and second quantizer estimates.

44. The non-transitory computer-readable medium of claim 43, further comprising:

determining when it becomes necessary to eliminate non-zero quantized transform coefficients according to the rate control policy, and

eliminating selected quantized transform coefficients.

45. The non-transitory computer-readable medium of claim 43, further comprising:

determining when it becomes necessary to eliminate pictures from the video sequence from being coded according to the rate control policy, and

performing frame decimation before pictures are coded.

46. The non-transitory computer-readable medium of claim 43, further comprising:

generating motion vectors for prediction of video data of macroblocks in the input pictures and of video data for sub-blocks therein of various sizes,

determining when it becomes necessary to eliminate motion vectors according to the rate control policy, and

eliminating selected motion vectors from an output coded bitstream.

51. The computer-readable medium of claim 43 wherein, for P and B pictures, the target coding rate is a predetermined target coding rate if a scene change detection signal indicates a scene change.

47. Rate and quality control system for an AVC-based video coder, comprising:

a content characteristics and coding rate analyzer, responsive to pictures from an input video sequence, to generate complexity indicators representative thereof,

a target bits computer, responsive to the complexity indicators and to a picture type signal, to calculate a target coding rate for each picture in the video sequence,

a buffer based quantizer computer, responsive to the target coding rates, to a transmit buffer indicator signal and to the picture type signal, to generate a first quantizer estimate for each picture, and

an activity based quantizer computer to calculate activity of each picture in the video sequence and modify the first quantizer estimate in response thereto,

a rate model quantizer estimator, responsive to quantizers and coding rates of previously-coded pictures and to picture type indicators of input pictures, to generate a second quantizer estimate for each picture according to a linear regression analysis, wherein linear regression coefficients of I pictures are selected according to the complexity indicators for such I pictures,

an AVC coder including a forward quantizer, and

a coding policy unit operative according to a rate control policy, wherein the rate control policy is selected based at least on a comparison of the first and second quantizer estimates.

52. The rate and quality control system of claim 47 wherein the target bits computer is further responsive to a scene detection signal, and wherein, for P and B pictures, the target coding rate is a predetermined target coding rate if the scene change detection signal indicates a scene change.