Motion-compensated prediction of inter-layer residuals

• US 8,711,948 B2
• Filed: 03/21/2008
• Issued: 04/29/2014
• Est. Priority Date: 03/21/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Tu, Chengjie, Srinivasan, Sridhar, Regunathan, Shankar, Sun, Shijun, Lin, Chih-Lung
• Primary Examiner(s): Sheikh, Ayaz
• Assistant Examiner(s): Ghafoerkhan, Faiyazkhan
• Application Number:
  US12/077,768
• Publication Number:
  US 20090238279A1
• Time in Patent Office: 2,230 Days
• Field of Search: None
• US Class Current: 375/240.26
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/34 Scalability techniques involving progressive bit-plane based encoding of the enhancement layer, e.g. fine granular scalability [FGS]
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets

Abstract

Techniques and tools are described for scalable video encoding and decoding. In some embodiments, an encoding tool encodes base layer video and outputs encoded base layer video in a base layer bit stream. The encoding tool encodes inter-layer residual video (representing differences between input video and reconstructed base layer video) using motion compensation relative to previously reconstructed inter-layer residual video. For the inter-layer residual video, the encoding tool outputs motion information and motion-compensated prediction residuals in an enhancement layer bit stream. A decoding tool receives the base layer bit stream and enhancement layer bit stream, reconstructs base layer video, reconstructs inter-layer residual video, and combines the reconstructed base layer video and reconstructed inter-layer residual video. Using motion compensation for the inter-layer residual video facilitates the use of separate motion vectors and separate codecs for the base layer video and inter-layer residual video.

Claims

1. A method comprising:

encoding base layer video to produce at least part of a base layer bit stream;

scaling inter-layer residual video that represents differences between the base layer video and an input video, including scaling samples of the inter-layer residual video from a first sample depth to a second sample depth lower than the first sample depth;

encoding the scaled inter-layer residual video at the second sample depth to produce at least part of an enhancement layer bit stream, wherein the encoding the scaled inter-layer residual video includes sample-domain prediction of blocks or macroblocks of a current picture of the scaled inter-layer residual video relative to one or more reference pictures of previously reconstructed scaled inter-layer residual video, and wherein the sample-domain prediction uses motion compensation of the blocks or macroblocks of the current picture of the scaled inter-layer residual video relative to the one or more reference pictures of the previously reconstructed scaled inter-layer residual video; and

signaling the at least part of the base layer bit stream and the at least part of the enhancement layer bit stream, wherein the enhancement layer bit stream includes motion information for the blocks or macroblocks of the current picture of the scaled inter-layer residual video.

2. The method of claim 1 further comprising, on a picture-by-picture basis:

after the encoding the base layer video, determining the inter-layer residual video using reconstructed base layer video and the input video.

3. The method of claim 1 further comprising:

before the encoding the base layer video, scaling the input video to produce the base layer video;

inverse scaling a reconstructed version of the base layer video; and

determining the inter-layer residual video as sample-by-sample differences between the input video and the inverse scaled, reconstructed base layer video.

4. The method of claim 3 wherein the scaling the input video comprises downsampling, and wherein the inverse scaling the reconstructed version of the base layer video comprises upsampling.

10. The method of claim 2 wherein the enhancement layer bit stream includes transform coefficient data for motion compensation residuals for the current picture of the inter-layer residual video, and wherein the decoding the at least part of the enhancement layer bit stream further includes:

decoding the motion compensation residuals for the current picture of the inter-layer residual video; and

combining the motion compensation residuals for the current picture of the inter-layer residual video with the motion-compensated prediction for the current picture of the inter-layer residual video.

5. A method comprising:

decoding at least part of a base layer bit stream to reconstruct base layer video;

decoding at least part of an enhancement layer bit stream to reconstruct inter-layer residual video that represents differences between the base layer video and an input video from encoding, including using motion compensation to predict blocks or macroblocks of a current picture of the inter-layer residual video relative to one or more reference pictures of previously reconstructed inter-layer residual video, wherein the enhancement layer bit stream includes motion information for the blocks or macroblocks of the current picture of the inter-layer residual video, the motion information indicating motion relative to the one or more reference pictures of the previously reconstructed inter-layer residual video, and wherein the motion information for the blocks or macroblocks of the current picture of the inter-layer residual video differs from motion information for blocks or macroblocks of the base layer video;

inverse scaling the reconstructed inter-layer residual video, including inverse scaling samples of the reconstructed inter-layer residual video to a higher sample depth after the reconstruction of the inter-layer residual video at a lower sample depth; and

combining the reconstructed base layer video and the inverse scaled, reconstructed inter-layer residual video to reconstruct output video.

6. The method of claim 5 further comprising:

buffering the reconstructed inter-layer residual video for use as a reference picture in motion compensation to predict one or more subsequent pictures of the inter-layer residual video relative to the buffered, reconstructed inter-layer residual video.

7. The method of claim 5 further comprising, before the combining, inverse scaling the reconstructed base layer video.

8. The method of claim 7 wherein the reconstructed base layer video and the reconstructed inter-layer residual video have different resolutions, and wherein the inverse scaling the reconstructed base layer video comprises upsampling the reconstructed base layer video to a higher resolution.

9. The method of claim 7 wherein the reconstructed output video and the reconstructed inter-layer residual video have different sample depths.

11. The method of claim 5 wherein a first decoding loop includes the decoding the at least part of the base layer bit stream, and wherein a second decoding loop separate from the first decoding loop includes the decoding the at least part of the enhancement layer bit stream.

12. The method of claim 5 further comprising repeating the decoding at least part of the base layer bit stream, the decoding at least part of the enhancement layer bit stream, and the combining on a picture-by-picture basis.

13. A computing system comprising a processor, memory and storage that stores computer-executable instructions for causing the system to implement:

a base layer decoder for decoding base layer video from a base layer bit stream;

an inter-layer prediction residual decoder for decoding inter-layer residual video that represents differences between the base layer video and an input video from encoding, wherein the inter-layer prediction residual decoder includes;

a buffer for storing one or more reference pictures of previously reconstructed inter-layer residual video; and

a motion compensator for predicting blocks or macroblocks of a current picture of the inter-layer residual video relative to the one or more reference pictures of the previously reconstructed inter-layer residual video using motion information for the blocks or macroblocks of the current picture of the inter-layer residual video from an enhancement layer bit stream;

an inverse scaler for inverse scaling the inter-layer residual video, including inverse scaling samples of the inter-layer residual video from a first sample depth to a second sample depth higher than the first sample depth; and

means for combining the base layer video and the inverse scaled inter-layer residual video to reconstruct output video.

14. The computing system of claim 13 wherein the inter-layer prediction residual decoder further includes:

an entropy decoder, inverse quantizer and inverse frequency transformer for decoding motion compensation residuals for the current picture of the inter-layer residual video; and

means for combining the motion compensation residuals for the current picture of the inter-layer residual video with the motion-compensated prediction for the current picture of the inter-layer residual video.

15. The computing system of claim 13 wherein the base layer decoder includes:

a buffer for storing one or more reference pictures of previously reconstructed base layer video; and

a motion compensator for predicting blocks or macroblocks of a current picture of the base layer video relative to the one or more reference pictures of the previously reconstructed base layer video.