Spatial scalability in 3D sub-band decoding of SDMCTF-encoded video

• US 8,374,238 B2
• Filed: 05/10/2005
• Issued: 02/12/2013
• Est. Priority Date: 07/13/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Xiong, Ruiqin, Xu, Jizheng, Wu, Feng, Li, Shipeng
• Primary Examiner(s): SENFI, BEHROOZ M
• Application Number:
  US11/127,001
• Publication Number:
  US 20060114993A1
• Time in Patent Office: 2,835 Days
• Field of Search: 348/404, 348/394, 348/409, 348/412, 348/415, 348/416, 375/240.12, 375/240.11, 375/240.18, 375/240.21, 375/240.16, 375/240.08, 375/240.2, 375/240.19, 375/240.27, 375/240.25
• US Class Current: 375/240.11
• CPC Class Codes: H04N 19/13 Adaptive entropy coding, e.g. adaptive variable length coding [AVLC] or context adaptive binary arithmetic coding [CABAC]
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/615 using motion compensated temporal filtering [MCTF]
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets
• CPC Class Codes: H04N 19/635 characterised by filter definition or implementation details

Abstract

Techniques and tools are described for scalable video coding and decoding. For example, a 3D sub-band decoder receives video encoded using spatial-domain motion-compensated temporal filtering at a first spatial resolution. The decoder decodes at least part of the video for output at a second spatial resolution lower than the first spatial resolution. The decoder uses any of several techniques to improve performance by devoting extra computational resources to the decoding, devoting extra buffer resources to storing reference picture information, and/or considering spatial high-pass sub-band information.

Claims

1. In a computing device that implements a video decoder, a method comprising:

receiving, at the computing device that implements the video decoder, video encoded by spatial-domain motion-compensated temporal filtering (“

SDMCTF”

) at a first spatial resolution, the SDMCTF being characterized by encoding with motion-compensated temporal filtering of spatial-domain values in a spatial domain before spatial decomposition into plural sub-band domains including a low-pass sub-band domain at a second spatial resolution lower than the first spatial resolution, wherein the SDMCTF produces a motion-compensated prediction in the spatial domain at the first spatial resolution; and

with the computing device that implements the video decoder, decoding at least part of the video with in-band inverse motion-compensated temporal filtering (“

IBIMCTF”

) for output at the second spatial resolution lower than the first spatial resolution, the IBIMCTF being characterized by inverse motion-compensated temporal filtering of sub-band values, including performing interpolation for one or more reference pictures, wherein the interpolation is not in-band interpolation on buffered spatial low-pass sub-band information, and wherein the IBIMCTF produces a motion-compensated prediction in the low-pass sub-band domain at the second spatial resolution.

2. The method of claim 1 wherein the decoding comprises:

performing an inverse spatial sub-band transform on the buffered spatial low-pass sub-band information;

performing an overcomplete spatial sub-band transform on results of the inverse spatial sub-band transform, wherein the interpolation applies to results of the overcomplete spatial sub-band transform; and

performing the motion-compensated prediction for the IBIMCTF on results of the interpolation.

3. The method of claim 2 wherein the inverse spatial sub-band transform is an inverse discrete wavelet transform (“

IDWT”

), and wherein the overcomplete spatial sub-band transform is an overcomplete discrete wavelet transform (“

DWT”

).

4. The method of claim 1 wherein the decoding comprises:

performing an inverse spatial sub-band transform on the buffered spatial low-pass sub-band information, wherein the interpolation applies to results of the inverse spatial sub-band transform;

performing a downsampling discrete wavelet transform (“

DSDWT”

) on results of the interpolation; and

performing the motion-compensated prediction for the IBIMCTF on results of the DSDWT.

5. The method of claim 4 wherein the inverse spatial sub-band transform is an inverse discrete wavelet transform (“

IDWT”

), and wherein the DSDWT is an m-sampled overcomplete discrete wavelet transform (“

DWT”

).

6. The method of claim 1 wherein the motion-compensated prediction for the IBIMCTF uses motion information from the SDMCTF.

7. The method of claim 1 wherein the decoding further comprises performing an inverse spatial sub-band transform on the buffered spatial low-pass sub-band information but not any spatial high-pass sub-band information, and wherein the interpolation follows the inverse spatial sub-hand transform.

8. The method of claim 1 wherein the first spatial resolution is double the second spatial resolution horizontally and vertically.

9. In a computing device that implements a video decoder, a method comprising:

receiving, at the computing device that implements the video decoder, video encoded by spatial-domain motion-compensated temporal filtering (“

SDMCTF”

) at a first spatial resolution, the SDMCTF being characterized by encoding with motion-compensated temporal filtering of spatial-domain values in a spatial domain before spatial decomposition into plural sub-band domains including a low-pass sub-band domain at a second spatial resolution lower than the first spatial resolution; and

with the computing device that implements the video decoder, decoding at least part of the video with spatial-domain inverse motion-compensated temporal filtering (“

SDIMCTF”

) for output at the second spatial resolution lower than the first spatial resolution, the SDIMCTF being characterized by inverse motion-compensated temporal filtering of spatial-domain values, wherein the SDIMCTF uses one or more reference pictures to produce a motion-compensated prediction in the spatial domain at the first spatial resolution, and wherein the one or more reference pictures for the SDIMCTF are reconstructed from received spatial low-pass sub-band information for the low-pass sub-band domain at the second spatial resolution but not from received spatial high-pass sub-band information associated with resolution enhancement up to the first spatial resolution.

10. The method of claim 9 wherein the decoding comprises:

performing an inverse spatial sub-band transform on the received spatial low-pass sub-band information but not any spatial high-pass sub-band information;

performing spatial-domain interpolation on results of the inverse spatial sub-band transform;

performing the motion-compensated prediction for the SDIMCTF using results of the spatial-domain interpolation as reference picture information; and

performing a spatial sub-band transform on results of the motion-compensated prediction for the SDIMCTF.

11. The method of claim 10 wherein the inverse spatial sub-band transform is an inverse discrete wavelet transform (“

IDWT”

), and wherein the spatial sub-band transform is a discrete wavelet transform (“

DWT”

).

12. The method of claim 10 wherein one or more buffers store the received spatial low-pass sub-band information but not spatial high-pass sub-band information.

13. The method of claim 9 wherein the decoding comprises:

performing spatial-domain interpolation on buffered spatial-domain information; and

performing the motion-compensated prediction for the SDIMCTF using results of the spatial-domain interpolation as reference picture information.

14. The method of claim 13 wherein for an initial level of the SDIMCTF the buffered spatial-domain information is generated from the received spatial low-pass sub-band information but not any spatial high-pass sub-band information, and wherein for one or more subsequent levels of the SDIMCTF the buffered spatial-domain information retains spatial high-pass sub-band information generated as intermediate results of the SDIMCTF.

15. In a computing device that implements a video decoder, a method comprising:

receiving, at the computing device that implements the video decoder, video encoded by spatial-domain motion-compensated temporal filtering (“

SDMCTF”

) at a first spatial resolution, the SDMCTF being characterized by encoding with motion-compensated temporal filtering of spatial-domain values in a spatial domain before spatial decomposition; and

with the computing device that implements the video decoder, decoding at least part of the video with spatial-domain inverse motion-compensated temporal filtering (“

SDIMCTF”

) for output at a second spatial resolution lower than the first spatial resolution, the SDIMCTF being characterized by inverse motion-compensated temporal filtering of spatial-domain values, wherein the SDIMCTF produces a motion-compensated prediction in the spatial domain at the first spatial resolution, and wherein the decoding includes using spatial high-pass sub-band information associated with the first spatial resolution to improve motion compensation performance in the SDIMCTF without using the spatial high-pass sub-band information to enhance spatial resolution up to the first spatial resolution.

16. The method of claim 15 wherein the decoding comprises:

performing spatial-domain interpolation on buffered spatial-domain information; and

performing the motion-compensated prediction for the SDIMCTF using results of the spatial-domain interpolation as reference picture information.

17. The method of claim 16 wherein the spatial high-pass sub-band information is generated as intermediate results of the SDIMCTF and retained as the buffered spatial-domain information.

18. The method of claim 16 wherein the spatial high-pass sub-band information is part of the received video, and wherein the buffered spatial-domain information is generated from received spatial low-pass sub-band information and the received spatial high-pass sub-band information.

19. The method of claim 15 wherein using the spatial high-pass sub-band information helps control drift in the SDIMCTF.

20. The method of claim 15 wherein one or more gain factors associated with the spatial high-pass sub-band information account for how the spatial high-pass sub-band information reduces distortion in the output at the second spatial resolution.