MEMORY MANAGEMENT FOR VIDEO DECODING

• US 20130051478A1
• Filed: 08/31/2011
• Published: 02/28/2013
• Est. Priority Date: 08/31/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wu, Yongjun, Sadhwani, Shyam
• Granted Patent: US 8,731,067 B2
• Time in Patent Office: Days
• US Class Current: 375/240.25
• CPC Class Codes: H04N 19/127 Prioritisation of hardware or computational resources
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 19/159 Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/423 characterised by memory arrangements H04N19/433 takes precedence
• CPC Class Codes: H04N 19/426 using memory downsizing methods
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder

Abstract

Techniques and tools described herein help manage memory efficiently during video decoding, especially when multiple video clips are concurrently decoded. For example, with clip-adaptive memory usage, a decoder determines first memory usage settings expected to be sufficient for decoding of a video clip. The decoder also determines second memory usage settings known to be sufficient for decoding of the clip. During decoding, memory usage is initially set according to the first settings. Memory usage is adaptively increased during decoding, subject to theoretical limits in the second settings. With adaptive early release of side information, the decoder can release side information memory for a picture earlier than the decoder releases image plane memory for the picture. The decoder can also adapt memory usage for decoded transform coefficients depending on whether the coefficients are for intra-coded blocks or inter-coded blocks, and also exploit the relative sparseness of non-zero coefficient values.

Claims

1. In a computing system that implements a video decoder, a method comprising:

receiving at least part of a bitstream for a video clip;

determining first memory usage settings for decoding of the video clip;

determining second memory usage settings different than the first memory usage settings, the second memory usage settings indicating one or more theoretical limits on memory usage according to a standard or format specification for decoding of the video clip; and

during the decoding of the video clip, adapting memory usage based at least in part on the first memory usage settings and the second memory usage settings, wherein the memory usage is initially set according to the first memory usage settings, and wherein the memory usage is increased during the decoding subject to the one or more theoretical limits in the second memory usage settings.

2. The method of claim 1 wherein the first memory usage settings are expected to be sufficient for the decoding of the video clip, and wherein the second memory usage settings are known to be sufficient for decoding of the video clip.

3. The method of claim 1 further comprising:

analyzing one or more syntax elements in the bitstream, wherein the second memory usage settings depend at least in part on the analysis of the one or more syntax elements.

4. The method of claim 1 wherein the adapting memory usage includes:

initializing a coded data buffer size according to the first memory usage settings;

dynamically growing the coded data buffer size during the decoding when the encoded data for a given picture of the video clip is greater than the coded data buffer size, up to a theoretical limit size set according to the second memory usage settings;

allocating a coded data buffer having the coded data buffer size; and

using the coded data buffer to store encoded data for each of one or more pictures of the video clip during the decoding.

5. The method of claim 4 wherein the increasing includes switching between multiple pre-determined coded data buffer sizes.

6. The method of claim 1 wherein the computing system uses multiple threads for the decoding, and wherein the adapting memory usage includes:

allocating one or more default coded data buffers each having a default coded data buffer size according to the first memory usage settings;

allocating a reserve coded data buffer having a reserve coded data buffer size according to a theoretical limit size set according to the second memory usage settings;

using the one or more default coded data buffers and the reserve coded data buffer during the decoding, wherein the reserve coded data buffer stores encoded data for a given picture of the video clip if the encoded data for the given picture is greater than the default coded data buffer size, and one of the one or more default coded data buffers stores the encoded data for the given picture if the encoded data for the given picture is less than the default coded data buffer size.

7. The method of claim 1 wherein the second memory usage settings include maximum bit rate per picture, the maximum bit rate per picture indicating maximum size of network access layer unit for one picture of the video clip.

8. The method of claim 1 wherein the first memory usage settings include, based on spatial resolution of at least part of the video clip, size of memory for side information and size of memory for reconstructed image plane information, and wherein the adapting memory usage includes:

allocating one or more side information buffers according to the size of memory for side information; and

allocating a reconstructed picture buffer according to the size of memory for reconstructed image plane information.

9. The method of claim 8 wherein the size of memory for side information includes size of memory for motion vectors and size of memory for reference index information.

10. The method of claim 8 wherein the adapting memory usage further includes:

if the spatial resolution increases during the decoding of the video clip, increasing the size of memory for side information and the size of memory for reconstructed image plane information; and

if the spatial resolution decreases during the decoding of the video clip, maintaining the size of memory for side information and the size of memory for reconstructed image plane information.

11. The method of claim 1 wherein the first memory usage settings include, based on group of picture structure and picture types for at least part of the video clip, use of memory for motion vectors and use of memory for reference index information.

12. The method of claim 11 wherein the adapting memory usage includes:

for a picture that includes only intra-coded content, using memory for reconstructed image plane information but using no memory for motion vectors and no memory for reference index information;

for a picture that includes at least some uni-directionally predicted content but no bi-directionally predicted content, using memory for reconstructed image plane information, memory for one set of motion vector information per partition and memory for one set of reference index information per partition; and

for a picture that includes at least some bi-directionally predicted content, using memory for reconstructed image plane information, memory for two sets of motion vector information per partition and memory for two sets of reference index information per partition.

13. A computing system comprising a processor, memory, and storage storing computer-executable instructions for causing the computing system to perform a method comprising:

receiving at least part of a bitstream for a video clip; and

during decoding of plural pictures of the video clip, for each of the plural pictures;

allocating first memory for storing side information associated with the picture;

allocating second memory for storing reconstructed image plane information associated with the picture;

upon determination that the picture is no longer to be used for reference in motion compensation during subsequent decoding, releasing the first memory; and

upon deletion of the picture from a decoded picture buffer, releasing the second memory.

14. The computing system of claim 13 wherein the side information includes motion vector information, reference index information, slice identifier information, mode information and partition information, and wherein the first memory includes a buffer for motion vector information, a buffer for reference index information, a buffer for slice identifier information, a buffer for mode information and a buffer for partition information.

15. The computing system of claim 13 wherein the side information includes slice identifier information and mode information, and wherein the first memory includes a buffer for slice identifier information and a buffer for mode information.

16. The computing system of claim 13 wherein, for each of the plural pictures, the first memory is allocated during an entropy decoding task for the picture and the second memory is allocated during a motion compensation task or intra decoding task for the picture.

17. The computing system of claim 13 wherein the computing system uses multiple threads for the decoding, and wherein the method further comprises:

setting priority of tasks whose completion enables release of reconstructed image plane memory higher than priority of tasks in which reconstructed image plane memory is allocated.

18. One of more tangible computer-readable media storing computer-executable instructions for causing a computing system programmed thereby to perform a method comprising:

receiving at least part of a bitstream for a video clip, wherein the video clip includes at least one picture encoded using inter-picture coding;

determining first memory usage settings for decoding intra-coded content of the at least one picture encoded using inter-picture coding;

determining second memory usage settings for decoding inter-coded content of the at least one picture encoded using inter-picture coding; and

during the decoding of the video clip, adapting memory usage based at least in part on the first memory usage settings and the second memory usage settings.

19. The one or more computer-readable media of claim 18 wherein the adapting memory usage includes:

allocating a first buffer according to the first memory usage settings;

in the first buffer, storing entropy decoded transform coefficients for intra-coded blocks of a given picture of the at least one picture encoded using inter-picture coding;

allocating a second buffer according to the second memory usage settings, wherein the first buffer is smaller than the second buffer; and

in the second buffer, storing entropy decoded transform coefficients for inter-coded blocks of the given picture.

20. The one or more computer-readable media of claim 18 wherein the adapting memory usage includes, during the decoding of the video clip, dynamically growing size for the first buffer and/or size for the second buffer.