RESAMPLING AND PICTURE RESIZING OPERATIONS FOR MULTI-RESOLUTION VIDEO CODING AND DECODING

• US 20110211122A1
• Filed: 05/06/2011
• Published: 09/01/2011
• Est. Priority Date: 01/06/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Sullivan, Gary J.
• Granted Patent: US 8,493,513 B2
• Time in Patent Office: Days
• US Class Current: 348/581
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: H04N 19/159 Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
• CPC Class Codes: H04N 19/16 for a given display mode, e.g. for interlaced or progressive display mode
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/593 involving spatial prediction techniques
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/80 Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence
• CPC Class Codes: H04N 19/82 involving filtering within a prediction loop
• CPC Class Codes: H04N 21/234327 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 21/234354 by altering signal-to-noise ratio parameters, e.g. requantization
• CPC Class Codes: H04N 21/234363 by altering the spatial resolution, e.g. for clients with a lower screen resolution
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/64792 Controlling the complexity of the content stream, e.g. by dropping packets intermediate media network packet handling H04L65/765; proxy provisioning conversion or adaptation for reducing the amount or size of exchanged application data H04L67/5651; negotiation of resources in wireless networks H04W28/16
• CPC Class Codes: H04N 7/0102 involving the resampling of the incoming video signal

Abstract

Techniques and tools for high accuracy position calculation for picture resizing in applications such as spatially-scalable video coding and decoding are described. In one aspect, resampling of a video picture is performed according to a resampling scale factor. The resampling comprises computation of a sample value at a position i, j in a resampled array. The computation includes computing a derived horizontal or vertical sub-sample position x or y in a manner that involves approximating a value in part by multiplying a 2ⁿ value by an inverse (approximate or exact) of the upsampling scale factor. The approximating can be a rounding or some other kind of approximating, such as a ceiling or floor function that approximates to a nearby integer. The sample value is interpolated using a filter.

Claims

1-22. -22. (canceled)

23. A method of performing upsampling of base layer image data with a computing device that implements an image or video encoder or decoder, wherein the computing device includes a processor and memory, the method comprising, for a position in an upsampled array:

with the computing device, computing a position in the base layer image data, wherein y indicates a vertical value for the position in the base layer image data, wherein derivation of y includes computation that is mathematically equivalent in result to (j*C+D)>

>

S, and wherein;

j indicates a vertical value for the position in the upsampled array;

C approximates 2^S+F multiplied by an inverse of a vertical scale factor;

D is an offset;

S is a shift value; and

F is based on a number of bits in a fractional component of y.

24. The method of claim 23 wherein j, C, and D depend in part on whether the base layer image data is for a frame or field, and wherein j and D depend in part on whether the base layer image data is for luma or chroma.

25. The method of claim 23 wherein:

S sets dynamic range and precision; and

D depends on vertical resolution for the base layer image data, the vertical scale factor and S.

26. The method of claim 23 wherein F is 4 and S is 12.

27. The method of claim 23 wherein x indicates a horizontal value for the position in the base layer image data, wherein derivation of x includes computation that is mathematically equivalent in result to (i*C′

+D′

)>

>

S′

, and wherein;

i indicates a horizontal value for the position in the upsampled array;

C′

approximates 2<sup>S′

+F+</sup> multiplied by an inverse of a horizontal scale factor;

D′

is an offset that can be the same or different as D;

S′

is a shift value that can be the same or different as S; and

F′

is based on a number of bits in a fractional component of x.

28. The method of claim 27 wherein i is based on T−

L, T indicating a horizontal offset and L indicating a left offset.

29. The method of claim 27 wherein F′

is 4 and S′

is 12.

30. The method of claim 27 wherein C′

is derived according to;

((BasePicWidth<

<

16)+(ScaledBaseWidth>

>

1))÷

ScaledBaseWidth,

where BasePicWidth indicates horizontal resolution for the base layer image data and ScaledBaseWidth indicates horizontal resolution after the upsampling.

31. The method of claim 27 wherein x further depends on an offset E, the derivation of x including computation that is mathematically equivalent in result to ((i*C′

+D′

)>

>

S′

)+E.

32. The method of claim 27 further comprising:

selecting a vertical filter based on F least significant bits of y and selecting vertical integer positions to be filtered based on remaining bits of y, wherein vertical interpolation of the base layer image data uses the vertical filter at the vertical integer positions; and

selecting a horizontal filter based on F′

least significant bits of x and selecting horizontal integer positions to be filtered based on remaining bits of x, wherein horizontal interpolation of results of the vertical interpolation uses the horizontal filter at the horizontal integer positions.

33. The method of claim 23 further comprising:

interpolating a value at the position in the base layer image data; and

assigning the interpolated value to the position in the upsampled array.

34. The method of claim 23 wherein the base layer image data is sample values or residual data values, the method further comprising:

receiving encoded data in a bitstream over a wireless communication connection;

decoding the encoded data to reconstruct video pictures, including decoding the base layer image data, performing the upsampling, and decoding an enhancement layer; and

outputting the reconstructed video pictures on the display

35. A system comprising a processor, memory and computer-readable storage that stores computer-executable instructions for causing the system to perform a method of upsampling, wherein the method comprises, for a position in an upsampled array:

computing a position in base layer image data, wherein y indicates a vertical value for the position in the base layer image data, wherein derivation of y includes computation that is mathematically equivalent in result to (j*C+D)>

>

S, and wherein;

j indicates a vertical value for the position in the upsampled array;

C approximates 2^S+F multiplied by an inverse of a vertical scale factor;

D is an offset;

S is a shift value; and

F is based on a number of bits in a fractional component of y.

36. The system of claim 35 wherein:

j, C, and D depend in part on whether the base layer image data is for a frame or field;

j and D depend in part on whether the base layer image data is for luma or chroma;

D depends on vertical resolution for the base layer image data, the vertical scale factor and S;

F is 4; and

S is 12.

37. The system of claim 35 wherein x indicates a horizontal value for the position in the base layer image data, wherein derivation of x includes computation that is mathematically equivalent in result to (i*C′

+D′

)>

>

S′

, and wherein;

i indicates a horizontal value for the position in the upsampled array;

C′

approximates 2<sup>S′

+F′</sup>

multiplied by an inverse of a horizontal scale factor;

D′

is an offset that can be the same or different as D;

S′

is a shift value that can be the same or different as S; and

F′

is based on a number of bits in a fractional component of x.

38. The system of claim 37 wherein:

i is based on T−

L, T indicating a horizontal offset and L indicating a left offset;

F′

is 4;

S′

is 12; and

C′

is derived according to;

((BasePicWidth<

<

16)+(ScaledBaseWidth>

>

1))÷

ScaledBaseWidth,

where BasePicWidth indicates horizontal resolution for the base layer image data and ScaledBaseWidth indicates horizontal resolution after the upsampling.

39. The system of claim 37 wherein the method further comprises:

selecting a vertical filter based on F least significant bits of y and selecting vertical integer positions to be filtered based on remaining bits of y, wherein vertical interpolation of the base layer image data uses the vertical filter at the vertical integer positions; and

selecting a horizontal filter based on F′

least significant bits of x and selecting horizontal integer positions to be filtered based on remaining bits of x, wherein horizontal interpolation of results of the vertical interpolation uses the horizontal filter at the horizontal integer positions.

40. The system of claim 39 wherein the computer-readable storage further stores computer-executable instructions for encoding an enhancement layer using interpolated sample values at positions in the upsampled array.

41. The system of claim 39 wherein the computer-readable storage further stores computer-executable instructions for decoding an enhancement layer using interpolated sample values at positions in the upsampled array.

42. A system comprising a processor, memory, a wireless communication connection, a display, and computer-readable storage that stores computer-executable instructions for causing the system to perform a method comprising:

receiving encoded data in a bitstream over the wireless communication connection;

decoding the encoded data to reconstruct video pictures, including;

decoding base layer image data;

upsampling the base layer image data, including for a position in an upsampled array;

computing a vertical value y for a position in the base layer image data, wherein derivation of y includes computation that is mathematically equivalent in result to (j*C+D)>

>

S, and wherein j indicates a vertical value for the position in the upsampled array, C approximates 2^S+F multiplied by an inverse of a vertical scale factor, D is an offset, S is a shift value, and F is based on a number of bits in a fractional component of y;

computing a horizontal value x for the position in the base layer image data, wherein derivation of x includes computation that is mathematically equivalent in result to (i*C′

+D′

)>

>

S′

, and wherein i indicates a horizontal value for the position in the upsampled array, C′

approximates 2<sup>S′

+F′</sup>

multiplied by an inverse of a horizontal scale factor, D′

is an offset that can be the same or different as D, S′

is a shift value that can be the same or different as S, and F′

is based on a number of bits in a fractional component of x;

selecting a vertical filter based on F least significant bits of y and selecting vertical integer positions to be filtered based on remaining bits of y, wherein vertical interpolation of the base layer image data uses the vertical filter at the vertical integer positions; and

selecting a horizontal filter based on F′

least significant bits of x and selecting horizontal integer positions to be filtered based on remaining bits of x, wherein horizontal interpolation of results of the vertical interpolation uses the horizontal filter at the horizontal integer positions; and

decoding an enhancement layer; and

outputting the reconstructed video pictures on the display.