Resampling and picture resizing operations for multi-resolution video coding and decoding

• US 8,780,272 B2
• Filed: 06/06/2013
• Issued: 07/15/2014
• Est. Priority Date: 01/06/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Sullivan, Gary J.
• Primary Examiner(s): Tran, Trang U
• Application Number:
  US13/911,940
• Publication Number:
  US 20130271651A1
• Time in Patent Office: 404 Days
• Field of Search: 348/581, 348/441, 348/443, 348445-446, 348/448, 348458-459, 348714-721, 348/413, 348415-416, 348/401, 348/402, 348/409, 348/410, 345660-670, 345/555, 345/560, 382/298, 382/307, 382/297, 358/451, 358/426.26, 358/240.26, 358/240.29
• US Class Current: 348/581
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: H04N 19/159 Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
• CPC Class Codes: H04N 19/16 for a given display mode, e.g. for interlaced or progressive display mode
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/593 involving spatial prediction techniques
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/80 Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence
• CPC Class Codes: H04N 19/82 involving filtering within a prediction loop
• CPC Class Codes: H04N 21/234327 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 21/234354 by altering signal-to-noise ratio parameters, e.g. requantization
• CPC Class Codes: H04N 21/234363 by altering the spatial resolution, e.g. for clients with a lower screen resolution
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/64792 Controlling the complexity of the content stream, e.g. by dropping packets intermediate media network packet handling H04L65/765; proxy provisioning conversion or adaptation for reducing the amount or size of exchanged application data H04L67/5651; negotiation of resources in wireless networks H04W28/16
• CPC Class Codes: H04N 7/0102 involving the resampling of the incoming video signal

Abstract

Techniques and tools for high accuracy position calculation for picture resizing in applications such as spatially-scalable video coding and decoding are described. In one aspect, resampling of a video picture is performed according to a resampling scale factor. The resampling comprises computation of a sample value at a position i,j in a resampled array. The computation includes computing a derived horizontal or vertical sub-sample position x or y in a manner that involves approximating a value in part by multiplying a 2ⁿ value by an inverse (approximate or exact) of the upsampling scale factor. The approximating can be a rounding or some other kind of approximating, such as a ceiling or floor function that approximates to a nearby integer. The sample value is interpolated using a filter.

Claims

1. A computer system comprising a processor and memory, wherein the computer system is adapted to perform a method comprising:

buffering a video picture; and

performing upsampling of the video picture according to a horizontal upsampling scale factor and a vertical upsampling scale factor, wherein the upsampling comprises computation of an interpolated sample value at horizontal position i and vertical position j in an upsampled array, and wherein the computation comprises;

computing a derived horizontal sub-sample position x in a manner that is mathematically equivalent in result to the formula x=(i*C+D)>

>

S, wherein C is derived by approximating a value equivalent to 2^S+F multiplied by an inverse of the horizontal upsampling scale factor, and wherein F, C, D, and S are integer values;

computing a derived vertical sub-sample position y in a manner that is mathematically equivalent in result to the formula y=(j*C′

+D′

)>

>

S, wherein C′

is derived by approximating a value equivalent to 2^S+F multiplied by an inverse of the vertical up sampling scale factor, and wherein D′

is an integer value; and

interpolating a sample value at the derived sub-sample position x, y.

2. The computer system of claim 1 wherein the computation further comprises:

selecting a horizontal resampling filter based on F least significant bits of the derived horizontal sub-sample position x; and

selecting lower resolution samples to be filtered based on the remaining more significant bits of the derived horizontal sub-sample position x; and

wherein interpolating a sample value at the derived sub-sample position x, y comprises;

interpolating the sample value based on the selected lower resolution samples and using the selected horizontal resampling filter.

3. The computer system of claim 2 wherein a horizontal resampling filter applied for at least one value of the F least significant bits of the derived horizontal sub-sample position x is a finite impulse response filter with more than two non-zero filter tap values.

4. The computer system of claim 3, wherein a horizontal resampling filter applied for all values other than 0 for the F least significant digits of the derived horizontal sub-sample position x is a finite impulse response filter with more than two non-zero filter tap values.

5. The computer system of claim 1 wherein the computation further comprises:

selecting a vertical resampling filter based on F least significant bits of the derived vertical sub-sample position y; and

selecting lower resolution samples to be filtered based on the remaining more significant bits of the derived vertical sub-sample position y; and

wherein interpolating a sample value at the derived sub-sample position x, y comprises;

interpolating the sample value based on the selected lower resolution samples and using the selected vertical resampling filter.

6. The computer system of claim 5 wherein a vertical resampling filter applied for at least one value of the F least significant bits of the derived vertical sub-sample position y is a finite impulse response filter with more than two non-zero filter tap values.

7. The computer system of claim 6, wherein a vertical resampling filter applied for all values other than 0 for the F least significant digits of the derived vertical sub-sample position y is a finite impulse response filter with more than two non-zero filter tap values.

8. The computer system of claim 1 wherein the interpolating is performed using one or more Mitchell-Netravalli resampling filters.

9. The computer system of claim 1 wherein the interpolating is performed using one or more Catmull-Rom resampling filters.

10. The computer system of claim 1 wherein at least one of F, C, C′

, D, D′

, or S depends at least in part on whether the sample value is a chroma sample value or a luma sample value.

11. The computer system of claim 1 wherein the interpolating is performed using one or more resampling filters having filter tap values controlled by a bandwidth control parameter.

12. The computer system of claim 1 wherein the value of F is equal to 4 and the value of S is equal to 12 .

13. The computer system of claim 1 wherein the video picture is a reference picture for motion-compensated prediction for an enhancement layer.

14. A method of processing reference picture data with a computing device that implements a video encoder or decoder, the method comprising:

with the computing device, buffering reference picture data; and

with the computing device, computing a position in the reference picture data, wherein x indicates a value for the position in the reference picture data, wherein derivation of x includes computation that is mathematically equivalent in result to (j*C+D)>

>

S, and wherein;

j indicates a value for a position in a current picture;

C approximates 2^S+F multiplied by an inverse of a scale factor;

D is an offset;

S is a shift value; and

F is based on a number of bits in a fractional component of x.

15. The method of claim 14 wherein j and D depend in part on whether the reference picture data is for luma or chroma.

16. The method of claim 14 wherein:

S sets dynamic range and precision; and

D depends at least in part on the scale factor and S.

17. The method of claim 14 wherein F is 4 and S is 12 .

18. The method of claim 14 wherein j indicates a horizontal sub-sample component of the position in the current picture, and wherein the scale factor is a horizontal upsampling scale factor.

19. The method of claim 14 wherein j indicates a vertical sub-sample component of the position in the current picture, and wherein the scale factor is a vertical upsampling scale factor.

20. The method of claim 14 wherein j is based on T−

L, T indicating a horizontal offset or vertical offset, and L indicating a left offset or top offset.

21. The method of claim 14 wherein C is derived according to:

((BasePicWidth<

<

16)+(ScaledBaseWidth>

>

1))÷

ScaledBaseWidth,

where BasePicWidth indicates horizontal resolution for the reference picture data and ScaledBaseWidth indicates horizontal resolution after upsampling.

22. The method of claim 14 wherein x further depends on an offset E, the derivation of x including computation that is mathematically equivalent in result to ((j*C+D) >

>

S)+E.

23. The method of claim 14 further comprising:

selecting a filter based on F least significant bits of x and selecting integer positions to be filtered based on remaining bits of x; and

interpolating the reference picture data using the filter at the integer positions.

24. The method of claim 14 further comprising, during video decoding:

receiving encoded data in a bitstream;

decoding the encoded data to reconstruct video pictures, including performing the buffering, performing the computing the position, and decoding an enhancement layer; and

outputting the reconstructed video pictures on the display.

25. The method of claim 14 further comprising, during video encoding:

accepting full-resolution video;

setting resolution for a base layer;

downsampling the full-resolution video to the resolution for the base layer;

encoding the downsampled video for the base layer;

setting resolution for an enhancement layer;

performing the buffering and the computing the position as part of encoding for the enhancement layer, wherein the reference picture data is based on base layer image data; and

outputting a layered bitstream including encoded data for the base layer and encoded data for the enhancement layer.

26. In a computer system that implements a video encoder or decoder, a method comprising:

buffering reference picture data for spatially scalable video;

computing a position in the reference picture data, wherein x indicates a value for the position in the reference picture data, wherein derivation of x includes computation that is mathematically equivalent in result to (j*C+D)>

>

S, and wherein;

j indicates a value for a position in a current picture;

C approximates 2^S+F multiplied by an inverse of a scale factor;

D is an offset;

S is a shift value; and

F is based on a number of bits in a fractional component of x; and

processing the reference picture data for the spatially scalable video at the position using a resampling filter that produces correct relative luma and chroma alignment for plural alignment structures, the spatially scalable video capable of being processed at plural spatial resolutions.

27. The method of claim 26 wherein the processing comprises using one or more bandwidth control parameters.

28. The method of claim 27 wherein the one or more bandwidth control parameters comprise blurriness control parameters.

29. The method of claim 27 further comprising integerizing the one or more bandwidth control parameters.

30. The method of claim 26 wherein the resampling filter includes a Mitchell-Netravali filter or a Catmull-Rom filter.

31. The method of claim 26 wherein the resampling filter has a smooth impulse response and/or a unity DC response.

32. The method of claim 26 wherein the resampling filter is an approximation of a Lanczos-2 design.