Embedded base layer codec for 3D sub-band coding

• US 20060008003A1
• Filed: 05/10/2005
• Published: 01/12/2006
• Est. Priority Date: 07/12/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ji, Xiangyang, Wu, Feng, Xu, Jizheng
• Granted Patent: US 8,340,177 B2
• Time in Patent Office: Days
• US Class Current: 375/240.110
• CPC Class Codes: H04N 19/13 Adaptive entropy coding, e.g. adaptive variable length coding [AVLC] or context adaptive binary arithmetic coding [CABAC]
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/615 using motion compensated temporal filtering [MCTF]
• CPC Class Codes: H04N 19/619 the transform being operated outside the prediction loop
• CPC Class Codes: H04N 19/62 by frequency transforming in three dimensions H04N19/63 takes precedence
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets

Abstract

Techniques and tools are described for scalable video coding and decoding. For example, a 3D sub-band video encoder includes an embedded base layer codec as well as temporal sub-band transforms and spatial sub-band transforms. The placement of the base layer codec among the sub-band transforms and the role of the base layer codec in scalable video coding vary depending on implementation. In general, the base layer codec provides efficient compression at low bit rates and produces a base layer compressed video bit stream compatible with existing decoders. At the same time, the 3D sub-band video encoder provides spatial and temporal scalability options at higher bit rates, refining the base layer video. A corresponding 3D sub-band video decoder includes an embedded base layer decoder.

Claims

1. A method comprising:

encoding video using a 3D sub-band video encoder with a base layer codec embedded therein; and

outputting results of the encoding.

2. The method of claim 1 wherein the base layer codec is embedded among plural temporal sub-band transforms of the 3D sub-band video encoder such that encoding/decoding with the base layer codec follows one or more of the plural sub-band temporal transforms and precedes one or more of the plural temporal sub-band transforms.

3. The method of claim 2 wherein the plural temporal sub-band transforms include motion-compensated temporal filtering.

4. The method of claim 1 wherein the base layer codec is a block-based motion-compensated prediction codec with two-dimensional frequency transforms.

5. The method of claim 1 wherein the encoding comprises:

downsampling input video to the base layer codec;

encoding and decoding the downsampled input video with the base layer codec, producing a base layer bit stream and reconstructed output video; and

upsampling the reconstructed output video.

6. The method of claim 1 wherein the encoding comprises:

performing a spatial sub-band transform, producing a spatial low-pass sub-band and one or more spatial high-pass sub-bands;

encoding and decoding the spatial low-pass sub-band with the base layer codec, producing a base layer bit stream and a reconstructed spatial low-pass sub-band;

computing a residual sub-band as the difference between the spatial low-pass sub-band and the reconstructed spatial low-pass sub-band; and

performing an inverse spatial sub-band transform on the residual sub-band and the one or more spatial high-pass sub-bands.

7. The method of claim 1 wherein the encoding comprises performing one or more temporal sub-band transforms on a residual sub-band, wherein the residual sub-band is based at least in part on the difference between input video to the base layer codec and reconstructed output video from the base layer codec.

8. The method of claim 1 wherein the encoding comprises performing motion-compensated temporal filtering in one or more temporal sub-band transforms following encoding/decoding with the base layer codec, and wherein at least some of the motion-compensated temporal filtering is relative to one or more reference pictures in reconstructed output video from the base layer codec.

9. The method of claim 8 wherein the motion-compensated temporal filtering includes a prediction stage and an update stage, and wherein the motion-compensated temporal filtering comprises, for each of one or more macroblocks:

when a reference picture from a lower level temporal low-pass sub-band is used in the prediction stage, performing the update stage for that reference picture; and

when one of the one or more reference pictures in the reconstructed output video is used in the prediction stage, skipping the update stage for that reference picture.

10. The method of claim 1 wherein the encoding comprises reusing motion information from encoding/decoding with the base layer codec in subsequent encoding with the 3D sub-band video encoder.

11. The method of claim 1 wherein the encoding comprises reusing intra mode information from encoding/decoding with the base layer codec in subsequent encoding with the 3D sub-band video encoder.

12. A method comprising:

decoding video using a 3D sub-band video decoder with a base layer decoder embedded therein; and

outputting results of the decoding.

13. The method of claim 12 wherein the base layer decoder is embedded among plural inverse temporal sub-band transforms of the 3D sub-band video decoder such that decoding with the base layer decoder follows one or more of the plural inverse temporal sub-band transforms and precedes one or more of the plural inverse temporal sub-band transforms.

14. The method of claim 12 wherein the base layer decoder is a block-based motion-compensated prediction decoder with two-dimensional inverse frequency transforms.

15. The method of claim 12 wherein the decoding comprises:

decoding a base layer bit stream with the base layer decoder, producing reconstructed output video; and

upsampling the reconstructed output video.

16. The method of claim 12 wherein the decoding comprises:

performing a spatial sub-band transform, producing a residual sub-band and one or more spatial high-pass sub-bands;

decoding a base layer bit stream with the base layer decoder, producing a reconstructed spatial low-pass sub-band;

combining the residual sub-band and the reconstructed spatial low-pass sub-band, producing a combined low-pass sub-band; and

performing an inverse spatial sub-band transform on the combined low-pass sub-band and the one or more spatial high-pass sub-bands.

17. The method of claim 12 wherein the decoding comprises performing one or more inverse temporal sub-band transforms on a residual sub-band.

18. The method of claim 12 wherein the decoding comprises performing inverse motion-compensated temporal filtering in one or more inverse temporal sub-band transforms following decoding with the base layer decoder, and wherein at least some of the inverse motion-compensated temporal filtering is relative to one or more reference pictures in reconstructed output video from the base layer decoder.

19. The method of claim 18 wherein the inverse motion-compensated temporal filtering includes a prediction stage and an update stage, and wherein the inverse motion-compensated temporal filtering comprises, for each of one or more macroblocks:

when a reference picture from a lower level temporal low-pass sub-band is to be used in the prediction stage, performing the update stage for that reference picture; and

when one of the one or more reference pictures in the reconstructed output video is to be used in the prediction stage, skipping the update stage for that reference picture.

20. A system comprising:

means for processing a scalable compressed video bit stream with a 3D sub-band video encoder or a 3D sub-band video decoder; and

means for processing a base layer compressed video bit stream with a base layer video codec embedded in the 3D sub-band video encoder or a base layer video decoder embedded in the 3D sub-band video decoder.