Digital advanced television systems

• US 5,218,435 A
• Filed: 02/20/1991
• Issued: 06/08/1993
• Est. Priority Date: 02/20/1991
• Status: Expired due to Term

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Lim, Jae S., Monta, Peter A.
• Primary Examiner(s): Groody, James J.
• Assistant Examiner(s): LEE, MICHAEL
• Application Number:
  US07/658,192
• Time in Patent Office: 839 Days
• Field of Search: 358/133, 358/136, 358/135, 358/138, 358/12, 358/13, 358/105, 358/140, 358/141, 382/56, 382/41, 382/43, 382/28
• US Class Current: 375/240.16
• CPC Class Codes: H04N 11/24 High-definition television systems
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/126 Details of normalisation or weighting functions, e.g. normalisation matrices or variable uniform quantisers
• CPC Class Codes: H04N 19/129 Scanning of coding units, e.g. zig-zag scan of transform coefficients or flexible macroblock ordering [FMO]
• CPC Class Codes: H04N 19/142 Detection of scene cut or scene change
• CPC Class Codes: H04N 19/179 the unit being a scene or a shot
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/503 involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/60 using transform coding
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/619 the transform being operated outside the prediction loop
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets
• CPC Class Codes: H04N 19/87 involving scene cut or scene change detection in combination with video compression

Abstract

Image quality is improved in high definition television using multi-scale representation of motion compensated residuals. The bandwidths of the subband filters vary with the frequency band and the total number of coefficients in the multi-scale-represented frames is equal to the number of values in the residual. Image initialization in the receivers is achieved using original image leakage, but the leakage factor is varied for different frequency subbands. To free up channel capacity at scene changes, a global (i.e., substantially frame-wide) decision is made as to whether to motion compensate a particular frame. To avoid the unattractive ghost of the previous scene persisting for a short time in a new scene, the motion estimator makes a decision on a block-by-block basis whether to use the original image or the residual. Chrominance resolution is improved by encoding all of the subbands of the chroma residuals, instead of just the low subbands. The chroma residuals are encoded at relatively coarser quantization than the luma residual, but when the energy of the luma residual is low (as, e.g. may occur when there is little motion), chroma quantization is improved, by making an overall (both chroma and luma) reduction in quantization step size. Runlength-amplitude representation and statistical coding are used. Runlength-amplitude representation is applied to entire subbands, and, preferably, different codebooks are used in statistically coding different subbands, to take advantage of the different statistics in the different subbands. The quantization coarseness is adaptively varied based on a computation of the number of bits necessary to statistically code a particular frame, thus guaranteeing for each frame exactly (or approximately, if a small buffer is provided in the decoder) the number of bits available in the channel.

Claims

1. A method of encoding television images comprising the steps of:

forming a residual frame having a number of values by determining the difference between an actual current image frame and a predicted current image frame;

forming the predicted current image frame by motion compensating a prior image frame;

coding the residual frame using subband representation, the subbands being formed by scaling the predicted current image frame relative to the actual current image frame so that a fraction of the current image is injected into the residual frame, the amount of the current image injected varying between subbands; and

quantizing the subband-represented residual frame.

2. The method of claim 1 wherein the fraction of the current image injected into the subbands of the residual frame is lower for higher frequency subbands.

3. The method of claim 2 wherein the fraction of the current image injected into the subbands of the residual frame is lowest for the highest frequency subbands and highest for the lowest frequency subband.

4. The method of claim 1 wherein the subband representation is multi-scale representation in which the subband filtering is performed with filter bandwidths that vary with frequency (the bandwidth being generally narrower at lower frequencies), and in which the total number of resulting coefficients in the multi-scale-represented frame is substantially equal to the number of values in the residual frame.

5. The method of claim 4 wherein the subband representation includes block transform representation such as the DCT.

6. A method of encoding television images, comprising the steps of:

forming a residual frame by determining the difference between an actual current image frame and a predicted current image frame;

forming the predicted current image frame by motion compensating a prior image frame using a frame of motion vectors;

estimating the motion in the image and using the estimates to form the frame of motion vectors;

determining based on the estimated motion whether to globally (i.e., substantially frame wide) code the actual current image frame instead of coding the residual frame;

encoding an indicator (e.g., a single bit) to inform the decoder of the global decision; and

quantizing either the residual frame with motion vectors or the actual current image frame without motion vectors, depending on which has been coded.

7. A method of decoding television images, comprising the steps of:

decoding a received frame to determine the state of an indicator that specifies whether the received frame is an actual image frame or a residual frame;

decoding the frame to form either a residual frame or an actual image frame;

if the frame is a residual frame, forming the current image frame by motion compensating a prior image frame held in memory and adding the residual frame to the motion-compensated frame; and

if the frame is an image frame, using that frame without motion compensation or adding a residual.

8. A method of encoding television images, comprising the steps of:

forming a residual frame by determining the difference between an actual current image frame and a predicted current image frame;

forming the predicted current image frame by motion compensating a prior image frame using a frame of motion vectors;

injecting into the residual frame fractions of the current image, for assisting the decoder in initialization or in reducing the effect of bit errors;

estimating the motion in the image on a block-by-block basis and using the estimates to form the frame of motion vectors, one vector for each block;

deciding on a block-by-block basis whether, for a particular frame, in light of a motion estimate consistent with a scene change or with motion too fast to compensate, that it is better to code the actual current image for the particular block instead of the residual frame;

coding indicators, for use in decoding, that specify on a block-by-block basis whether the residual or the actual image has been encoded; and

quantizing the mixed residual/image frame.

9. A method of decoding television images, comprising the steps of:

decoding the television signal to determine the state of indicators that indicate whether individual blocks of a received frame represent blocks of the image or blocks of a residual;

for those blocks that represent a residual, forming that block of the image by motion compensating corresponding block from a prior image and adding the decoded residual to the motion-compensated block;

for those blocks that represent the actual image, using the decoded block without motion compensation for the addition of a residual; and

relying for initialization of, or reduction of the effect of bit errors in, those blocks that represent a residual on the injection at the encoder of a small fraction of the original image.

10. A method of encoding television images, comprising the steps of:

encoding luma and chroma components of the television image by encoding luma and chroma residuals, each of which comprises the difference between the current component and a motion compensated prediction of the component;

quantizing the luma residual at a relatively finer quantization step size than the chroma residual; and

at times when the luma energy is lower, decreasing the quantization step size of the chroma residual to improve chroma resolution.

11. The method of claim 10 wherein the relative fineness of the luma quantization is greatest for the lower frequency subbands.

12. The method of claim 11 wherein relative quantization step sizes for the luma and chroma residuals are maintained, and wherein, at times when the luma energy is lower, the quantization step size of both luma and chroma residuals is decreased.

13. The method of claim 12 further comprising the step of subband filtering the luma and chroma residuals, and wherein low and high subbands are quantized for both the luma and chroma residuals.

14. The method of claim 13 wherein the quantization step size is decreased adaptively by choosing a step size reduction factor that substantially matches the available bit rate.

15. The method of claim 10 further comprising the step of encoding a quantization step size coefficient for use by the decoder.

16. A method of decoding television images, comprising the steps of:

decoding the television signal to produce luma and chroma residuals, which represent the difference between a current component and a motion compensated prediction of the component, said decoding using a relatively finer quantization step size for the luma residual than for the chroma residual; and

decoding a quantization step size coefficient representing the variation in quantization step size made by the encode to improve chroma resolution at times when the luma energy is lower.

17. The method of claim 16 wherein the relative fineness of the luma quantization is greatest for the lower frequency subbands.

18. The method of claim 17 wherein relative quantization step sizes used for decoding the luma and chroma residuals remain the same, and wherein the quantization step size coefficient is used to raise or lower the quantization step sizes used for both luma and chroma residuals, with the result that chroma resolution is enhanced during intervals of low energy in the luma residual.

19. The method of claim 18 wherein luma and chroma residuals have been subband filtered, and the decoder must perfor the inverse of the filtering to recove the residuals.

20. The method of claim 19 wherein the quantization step size has been decreased adaptively in the encoder by choosing a step size reduction factor that substantially matches the available bit rate.

21. A method of encoding television images, comprising the steps of:

forming a residual frame having a number of values by determining the difference between a current image frame and a predicted version of the current frame;

forming the predicted version of the current frame by motion compensating a prior frame;

coding the residual frame by subband filtering the residual frame;

quantizing the subband-filtered residual frame; and

statistically coding the quantized, subband-filter residual frame by scanning the quantized residual frame and encoding each string of zeros and adjacent non-zero value as a pair of numbers, with one number representing the number of repeated zeros and the other number representing the amplitude of the non-zero value, the direction of scanning in at least some of the low-vertical/high-horizontal subband sis orthogonal to the direction in the high-vertical/low-horizontal subbands at the same frequency level.

22. The method of claim 21 further comprising the step of statistically coding the runlength-amplitude pairs, using a codebook relating runlength-amplitude pairs to bit strings.

23. The method of claim 22 wherein Huffman coding is used for the statistical coding.

24. The method of claim 22 wherein different codebooks are used for different subbands.

25. The method of claim 24 wherein

the subband filtering comprises multi-scale representation, in which the subband filtering is performed with filter bandwidths that vary with frequency (the bandwidth being generally narrower at lower frequencies), and in which the total number of resulting coefficients in the multi-scale-represented frame is substantially equal to the number of values in the residual frame;

and

different codebooks are used for at least some of the different multi-scale representation levels.

26. A method of decoding television images, comprising the steps of:

decoding the television signal using inverse statistical coding to produce decoded frames comprising subband-filtered residual frames having a number of values, the statistical coding producing a pair of numbers, with one number representing the number of repeated zeros and the other number representing the amplitude of the non-zero value, the decoded frames being assemble in a scanning pattern, wherein the direction of scanning in at least some of the low-vertical/high-horizontal subbands is orthogonal to the direction in the high-vertical/low-horizontal subbands;

performing inverse subband filtering on the decoded frames to produce residual frames; and

forming image frames by motion compensating prior frames stored in memory and adding to each a corresponding residual frame.

27. The method of claim 26 wherein statistical decoding is performed prior to inverse runlength-amplitude representation, using a codebook relating runlength-amplitude pairs to bit strings.

28. The method of claim 27 wherein Huffman coding is used for the statistical coding.

29. The method of clam 27 wherein different codebooks are used for different subbands.

30. The method of claim 29 wherein

the subband filtering comprises multi-scale representation, in which the subband filtering is performed with filter bandwidths that vary with frequency (the bandwidth being generally narrower at lower frequencies), and in which the total number of resulting coefficients in the multi-scale-represented frame is substantially equal to the number of values in the residual frame;

and

different codebooks are used for at least some of the different multi-scale-represented levels.

31. A method of encoding television images for transmission across a channel with a substantially fixed bit capacity, comprising the steps of:

forming a residual frame having a number of values by determing the difference between a current image frame and a predicted version of the current frame;

forming the predicted version of the current frame by motion compensating a prior frame;

quantizing the residual frame (or a subband-filtered or transformed version thereof) at a chosen coarseness of quantization;

runlength-amplitude representing the quantized residual frame;

statistically coding the runlength-amplitude represented frame; and

adaptively choosing the coarseness of quantization on a frame-by-frame basis so that the number of bits produced by the statistical coding substantially equals the number of bits available on the channel for each frame.

32. The method of claim 31 wherein the statistically coded bits are transmitted without the need for a large buffer to accommodate frame-to-frame variation in the number of bits.

33. The method of claim 31 further comprising the step of subband filtering the residual frames prior to quantizing.

34. The method of claim 33 wherein the subband filtering is multi-scale, the subband filtering being performed with filter bandwidths that vary with frequency (the bandwidth being generally narrower at lower frequencies) and the subband filtered frames being subsampled so that the representation is multi-scale with the total number of resulting coefficients in the multi-scale-represented frame being substantially equal to the number of values in the residual frame.

35. The method of claim 31 wherein

there are residual frames for luma and chroma components of the image;

the quantization used for the chroma is relatively coarser than that used for the luma; and

the adaptive variation of quantization coarseness is done while preserving this relative difference in quantization coarseness between the luma and chroma.

36. The method of claim 31 wherein the choice of quantization coarseness is made using an iterative procedure, in which the number of bits produced by statistical coding for a chosen coarseness is repetitively computed for different coarsenesses until a coarseness is found for which the number of bits produced by statistical coding substantially equals the number of bits available on the channel for each frame.

37. A method of decoding television signals, comprising the steps of:

receiving a television signal that has been encoded by a method comprising the steps of

forming a residual frame having a number of values by determining the difference between a current image frame and a predicted version of the current frame;

forming the predicted version of the current frame by motion compensating a prior frame;

quantizing the residual frame (or a subband-filtered or transformed version thereof) at a chosen coarseness of quantization;

runlength-amplitude representing the quantized residual frame;

statistically coding the runlength-amplitude represented frame;

adaptively choosing the coarseness of quantization on a frame-by-frame basis so that the number of bits produced by the statistical coding substantially equals the number of bits available on the channel for each frame;

performing an inverse statistical coding; and

assigning decoded bits to frames without the use of a buffer any larger than a small fraction of a frame, by using as a decoding rule the assumption that each frame of the signal has been encoded using approximately the same number of bits.