Motion-compensated prediction of inter-layer residuals

• US 8,964,854 B2
• Filed: 04/22/2014
• Issued: 02/24/2015
• Est. Priority Date: 03/21/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Tu, Chengjie, Srinivasan, Sridhar, Regunathan, Shankar, Sun, Shijun, Lin, Chih-Lung
• Primary Examiner(s): Sheikh, Ayaz
• Assistant Examiner(s): Ghafoerkhan, Faiyazkhan
• Application Number:
  US14/258,959
• Publication Number:
  US 20140226718A1
• Time in Patent Office: 308 Days
• Field of Search: None
• US Class Current: 375/240.26
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/34 Scalability techniques involving progressive bit-plane based encoding of the enhancement layer, e.g. fine granular scalability [FGS]
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets

Abstract

Techniques and tools are described for scalable video encoding and decoding. In some embodiments, an encoding tool encodes base layer video and outputs encoded base layer video in a base layer bit stream. The encoding tool encodes inter-layer residual video (representing differences between input video and reconstructed base layer video) using motion compensation relative to previously reconstructed inter-layer residual video. For the inter-layer residual video, the encoding tool outputs motion information and motion-compensated prediction residuals in an enhancement layer bit stream. A decoding tool receives the base layer bit stream and enhancement layer bit stream, reconstructs base layer video, reconstructs inter-layer residual video, and combines the reconstructed base layer video and reconstructed inter-layer residual video. Using motion compensation for the inter-layer residual video facilitates the use of separate motion vectors and separate codecs for the base layer video and inter-layer residual video.

Claims

1. A method comprising:

encoding base layer video to produce at least part of a base layer bit stream;

encoding inter-layer residual video to produce at least part of an enhancement layer bit stream, wherein the inter-layer residual video represents differences between the base layer video and input video, and wherein the encoding the inter-layer residual video includes, for a given block of a current picture of the inter-layer residual video;

performing motion compensated prediction of the given block of the current picture of the inter-layer residual video relative to one or more reference pictures of previously reconstructed inter-layer residual video, wherein multiple reference pictures of the previously reconstructed inter-layer residual video are available for use in the motion compensated prediction for the given block of the current picture of the inter-layer residual video;

determining residual values between the given block of the current picture of the inter-layer residual video and the motion compensated prediction for the given block;

performing a frequency transform of the residual values to produce transform coefficients;

quantizing the transform coefficients; and

performing arithmetic coding of the quantized transform coefficients; and

signaling the at least part of the base layer bit stream and the at least part of the enhancement layer bit stream.

2. The method of claim 1 wherein the enhancement layer bit stream includes motion vector information for at least some blocks of the inter-layer residual video.

3. The method of claim 1 further comprising, on a picture-by-picture basis:

after the encoding the base layer video, determining the inter-layer residual video using reconstructed base layer video and the input video.

4. The method of claim 1 further comprising:

before the encoding the base layer video, scaling the input video to produce the base layer video;

inverse scaling a reconstructed version of the base layer video; and

determining the inter-layer residual video as sample-by-sample differences between the input video and the inverse scaled, reconstructed base layer video.

5. The method of claim 4 wherein the scaling comprises one or more of downsampling spatial resolution, downsampling chroma sampling rate and scaling sample depth, and wherein the inverse scaling comprises one or more of upsampling spatial resolution, upsampling chroma sampling rate and scaling sample depth.

6. The method of claim 1 further comprising, before the encoding the inter-layer residual video, scaling the inter-layer residual video.

7. The method of claim 6 wherein the scaling comprises scaling samples of the inter-layer residual video from a first sample depth to a second sample depth smaller than the first sample depth.

8. The method of claim 1 wherein the multiple reference pictures are from the same temporal direction relative to the current picture.

9. One or more computer-readable memory storing instructions for causing a computing device programmed thereby to perform a method comprising:

decoding at least part of a base layer bit stream to reconstruct base layer video;

decoding at least part of an enhancement layer bit stream to reconstruct inter-layer residual video that represents differences between the base layer video and input video from encoding, including, for a given block of a current picture of the inter-layer residual video;

using motion compensation to predict the given block of the current picture of the inter-layer residual video relative to one or more reference pictures of previously reconstructed inter-layer residual video, wherein multiple reference pictures of the previously reconstructed inter-layer residual video are available for use in the motion compensation for the given block of the current picture of the inter-layer residual video;

performing arithmetic decoding of quantized transform coefficients for residual values between the given block of the current picture of the inter-layer residual video and the motion compensated prediction for the given block;

inverse quantizing the quantized transform coefficients;

performing an inverse frequency transform on the transform coefficients to reconstruct the residual values for the given block; and

combining the reconstructed residual values for the given block and the motion compensated prediction for the given block; and

combining the reconstructed base layer video and the reconstructed inter-layer residual video to reconstruct output video.

10. The one or more computer-readable memory of claim 9 wherein the enhancement layer bit stream includes motion vector information for at least some blocks of the current picture of the inter-layer residual video, the motion vector information indicating motion relative to the one of more of the multiple reference pictures of the previously reconstructed inter-layer residual video, and wherein the motion vector information for the at least some blocks of the current picture of the inter-layer residual video differs from motion vector information for blocks of a corresponding picture of the base layer video.

11. The one or more computer-readable memory of claim 9 wherein the method further comprises:

buffering the reconstructed inter-layer residual video for use in motion compensation to predict subsequent inter-layer residual video relative to the buffered, reconstructed inter-layer residual video.

12. The one or more computer-readable memory of claim 9 wherein the method further comprises, before the combining the reconstructed base layer video and the reconstructed inter-layer residual video, inverse scaling the reconstructed base layer video and/or the reconstructed inter-layer residual video.

13. The one or more computer-readable memory of claim 12 wherein the reconstructed base layer video and the reconstructed inter-layer residual video have different spatial resolutions, and wherein the inverse scaling comprises upsampling the reconstructed base layer video to a higher spatial resolution.

14. The one or more computer-readable memory of claim 12 wherein the reconstructed output video and the reconstructed inter-layer residual video have different sample depths, and wherein the inverse scaling comprises scaling samples of the reconstructed inter-layer residual video to a higher sample depth.

15. The one or more computer-readable memory of claim 9 wherein a first decoding loop includes the decoding the at least part of the base layer bit stream, and wherein a second decoding loop separate from the first decoding loop includes the decoding the at least part of the enhancement layer bit stream.

16. The one or more computer-readable memory of claim 9 wherein the method further comprises repeating the decoding at least part of the base layer bit stream, the decoding at least part of the enhancement layer bit stream, and the combining the reconstructed base layer video and the reconstructed inter-layer residual video on a picture-by-picture basis.

17. The one or more computer-readable memory of claim 9 wherein the multiple reference pictures are from the same temporal direction relative to the current picture.

18. A system comprising:

a base layer decoder for decoding base layer video;

an inter-layer prediction residual decoder for decoding inter-layer residual video that represents differences between the base layer video and input video from encoding, wherein the inter-layer prediction residual decoder is configured to;

with buffers, store reference pictures of previously reconstructed inter-layer residual video; and

with a motion compensator, predict the inter-layer residual video relative to one or more of the reference pictures of the previously reconstructed inter-layer residual video;

with an entropy decoder, perform arithmetic decoding of quantized transform coefficients;

with an inverse quantizer, inverse quantize the quantized transform coefficients;

with an inverse frequency transformer, perform an inverse frequency transform on the transform coefficients to reconstruct residual values; and

combine the reconstructed residual values and corresponding motion compensated predictions; and

an inverse scaler for scaling samples of the inter-layer residual video from a first sample depth to a second sample depth higher than the first sample depth;

wherein the system is further configured to combine the base layer video and the inverse scaled inter-layer residual video to reconstruct output video.

19. The system of claim 18 wherein the base layer decoder is configured to:

with a buffer, store previously reconstructed base layer video; and

with a motion compensator, predict the base layer video relative to the previously reconstructed base layer video.

20. The system of claim 18 further comprising:

an inverse scaler for upsampling samples of the base layer video to a higher spatial resolution, upsampling the samples of the base layer video to a higher chroma sampling rate and/or scaling the samples of the base layer video to a higher sample depth.