Signaling for sub-decoded picture buffer (sub-DPB) based DPB operations in video coding

• US 9,819,941 B2
• Filed: 10/08/2014
• Issued: 11/14/2017
• Est. Priority Date: 10/10/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Ramasubramonian, Adarsh Krishnan, Wang, Ye-Kui
• Primary Examiner(s): Vaughn, Jr., William C
• Assistant Examiner(s): UHL, LINDSAY JANE KILE
• Application Number:
  US14/509,797
• Publication Number:
  US 20150103884A1
• Time in Patent Office: 1,133 Days
• CPC Class Codes: H04N 19/156 Availability of hardware or computational resources, e.g. encoding based on power-saving criteria
• CPC Class Codes: H04N 19/177 the unit being a group of pictures [GOP]
• CPC Class Codes: H04N 19/196 being specially adapted for the computation of encoding parameters, e.g. by averaging previously computed encoding parameters processing of motion vectors H04N19/513
• CPC Class Codes: H04N 19/31 in the temporal domain
• CPC Class Codes: H04N 19/423 characterised by memory arrangements H04N19/433 takes precedence
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/463 by compressing encoding parameters before transmission
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards

Abstract

A method of decoding video data including receiving an encoded video bitstream that includes a plurality of pictures and storing the plurality of pictures in one or more sub-DPBs. The method further including receiving a respective set of sub-DPB parameters for each respective operation point of the encoded video bitstream. applying the respective set of sub-DPB parameters to all layers of an output layer set for each respective operation point, and performing a sub-DPB management process on the one or more sub-DPBs in accordance with the received respective single sets of sub-DPB parameters.

Claims

1. A method of decoding video data, the method comprising:

receiving, from an encoded video bitstream, a plurality of output layer sets, wherein each output layer set of the plurality of output layer sets includes a respective plurality of sub-layers, wherein each respective plurality of sub-layers includes a first sub-layer and one or more sub-layers other than the first sub-layer, wherein the first sub-layer of each respective plurality of sub-layers is a base sub-layer;

decoding each respective plurality of sub-layers of the plurality of output layer sets;

storing each respective decoded plurality of sub-layers in one or more sub-decoded picture buffers (DPBs);

receiving, from the encoded video bitstream, a first respective syntax element for each output layer set of the plurality of output layer sets, wherein each first respective syntax element is indicative of whether or not sub-DPB parameters are signaled for each sub-layer of each respective output layer set;

receiving, from the encoded video bitstream, a second respective syntax element for each sub-layer of the one or more sub-layers other than the first sub-layer of a first particular output layer set of the plurality of output layer sets but not for the first sub-layer of the first particular output layer set, wherein each second respective syntax element is indicative of whether or not sub-DPB parameters are signaled for each respective sub-layer of the one or more sub-layers other than the first sub-layer of the first particular output layer set;

receiving, from the encoded video bitstream, a respective set of sub-DPB parameters for the first sub-layer of each respective plurality of sub-layers; and

performing a sub-DPB management process on the one or more sub-DPBs in accordance with each respective set of sub-DPB parameters received.

2. The method of claim 1, wherein each respective set of sub-DPB parameters for the first sub-layer of each respective plurality of sub-layers includes at least one of:

a respective third syntax element indicative of a maximum number of reorder pictures allowed for the one or more sub-DPBs;

or

a respective fourth syntax element indicative of a maximum latency allowed for the one or more sub-DPBs.

3. The method of claim 1, wherein receiving the respective set of sub-DPB parameters comprises receiving the respective set of sub-DPB parameters in a video parameter set (VPS).

4. The method of claim 1, further comprising:

applying, based on the first respective syntax element indicating that sub-DPB parameters are not signaled for each sub-layer of a second particular output layer set of the plurality of output layer sets, the respective set of sub-DPB parameters of the second particular output layer set to all sub-layers of the second particular output layer set.

5. The method of claim 1, further comprising:

receiving each second respective syntax element for each sub-layer of the one or more sub-layers other than the first sub-layer of the first particular output layer set when the first respective syntax element indicating that sub-DPB parameters are signaled for each sub-layer of the first particular output layer set.

6. The method of claim 1, further comprising:

applying, based on the second respective syntax element indicating that sub-DPB parameters are not signaled for a particular sub-layer of the one or more sub-layers other than the first sub-layer of the first particular output layer set, a last received respective set of sub-DPB parameters for the particular sub-layer;

or

receiving, when the second respective syntax element is indicative that sub-DPB parameters are signaled for the particular sub-layer of the one or more sub-layers other than the first sub-layer of the first particular output layer set, a respective set of sub-DPB parameters for the particular sub-layer.

7. The method of claim 1, further comprising:

applying, based on the second respective syntax element indicating that sub-DPB parameters are not signaled for a particular sub-layer with a sub-layer ID X, a last received respective set of sub-DPB parameters for sub-layers of the one or more sub-layers other than the first sub-layer of the first particular output layer set that have a respective sub-layer ID greater than X, wherein the particular sub-layer with the sub-layer ID X is one of the one or more sub-layers other than the first sub-layer of the first particular output layer set.

8. The method of claim 1, wherein the sub-DPB management process includes at least one of marking pictures as unused for reference, marking pictures for output, or removing pictures from the sub-DPB, and wherein the performing the sub-DPB management process comprises performing the sub-DPB management process on an access unit level using the received respective sets of sub-DPB parameters.

9. The method of claim 8, wherein performing the sub-DPB management process on the access unit level comprises:

outputting pictures based on the number of access units in a first sub-DPB of the one or more sub-DPBs that contain at least one picture marked as needed for output;

marking pictures as unused for reference on a per-layer level; and

removing pictures from the first sub-DPB on a per-layer level.

10. An apparatus configured to decode video data, the apparatus comprising:

one or more memory devices comprising one or more sub-decoded picture buffers (DPBs) configured to store video data; and

a video decoder configured to;

receive, from an encoded video bitstream, a plurality of output layer sets, wherein each output layer set of the plurality of output layer sets includes a respective plurality of sub-layers, wherein each respective plurality of sub-layers includes a first sub-layer and one or more sub-layers other than the first sub-layer, wherein the first sub-layer of each respective plurality of sub-layers is a base sub-layer;

decode each respective plurality of sub-layers of the plurality of output layer sets;

store each respective decoded plurality of sub-layers in the one or more sub-DPBs;

receive, from the encoded video bitstream, a first respective syntax element for each output layer set of the plurality of output layer sets, wherein each first respective syntax element is indicative of whether or not sub-DPB parameters are signaled for each sub-layer of each respective output layer set;

receive, from the encoded video bitstream, a second respective syntax element for each sub-layer of the one or more sub-layers other than the first sub-layer of a first particular output layer set of the plurality of output layer sets but not for the first sub-layer of the first particular output layer set, wherein each second respective syntax element is indicative of whether or not sub-DPB parameters are signaled for each respective sub-layer of the one or more sub-layers other than the first sub-layer of the first particular output layer set;

receive, from the encoded video bitstream, a respective set of sub-DPB parameters for the first sub-layer of each respective plurality of sub-layers; and

perform a sub-DPB management process on the one or more sub-DPBs in accordance with each respective set of sub-DPB parameters received.

11. The apparatus of claim 10, wherein each respective set of sub-DPB parameters for the first sub-layer of each respective plurality of sub-layers includes at least one of:

a respective third syntax element indicative of a maximum number of reorder pictures allowed for the one or more sub-DPBs;

or

a respective fourth syntax element indicative of a maximum latency allowed for the one or more sub-DPBs.

12. The apparatus of claim 10, wherein the video decoder is configured to receive the respective set of sub-DPB parameters in a video parameter set (VPS).

13. The apparatus of claim 10, wherein the video decoder is further configured to:

apply, based on the first respective syntax element indicating that sub-DPB parameters are not signaled for each sub-layer of a second particular output layer set of the plurality of output layer sets, the respective set of sub-DPB parameters of the second particular output layer set to all sub-layers of the second particular output layer set.

14. The apparatus of claim 10, wherein the video decoder is further configured to:

receive each second respective syntax element for each sub-layer of the one or more sub-layers other than the first sub-layer of the first particular output layer set when the first respective syntax element indicating that sub-DPB parameters are signaled for each sub-layer of the first particular output layer set.

15. The apparatus of claim 10, wherein the video decoder is further configured to:

apply, based on the second respective syntax element indicating that sub-DPB parameters are not signaled for a particular sub-layer of the one or more sub-layers other than the first sub-layer of the first particular output layer set, a last received respective set of sub-DPB parameters for the particular sub-layer;

or

receive, when the second respective syntax element is indicative that sub-DPB parameters are signaled for the particular sub-layer of the one or more sub-layers other than the first sub-layer of the first particular output layer set, a respective set of sub-DPB parameters for the particular sub-layer.

16. The apparatus of claim 10, wherein the video decoder is further configured to:

apply, based on the second respective syntax element indicating that sub-DPB parameters are not signaled for a particular sub-layer with a sub-layer ID X, a last received respective set of sub-DPB parameters for sub-layers of the one or more sub-layers other than the first sub-layer of the first particular output layer set that have a respective sub-layer ID greater than X, wherein the particular sub-layer with the sub-layer ID X is one of the one or more sub-layers other than the first sub-layer of the first particular output layer set.

17. The apparatus of claim 10, wherein the sub-DPB management process includes at least one of marking pictures as unused for reference, marking pictures for output, or removing pictures from the sub-DPB, and wherein the performing the sub-DPB management process comprises performing the sub-DPB management process on an access unit level using the received respective sets of sub-DPB parameters.

18. The apparatus of claim 17, wherein the video decoder is further configured to:

output pictures based on the number of access units in a first sub-DPB of the one or more sub-DPBs that contain at least one picture marked as needed for output;

mark pictures as unused for reference on a per-layer level; and

remove pictures from the first sub-DPB on a per-layer level.

19. An apparatus configured to decode video data, the apparatus comprising:

means for receiving, from an encoded video bitstream, a plurality of output layer sets, wherein each output layer set of the plurality of output layer sets includes a respective plurality of sub-layers, wherein each respective plurality of sub-layers includes a first sub-layer and one or more sub-layers other than the first sub-layer, wherein the first sub-layer of each respective plurality of sub-layers is a base sub-layer;

means for decoding each respective plurality of sub-layers of the plurality of output layer sets;

means for storing each respective decoded plurality of sub-layers in one or more sub-decoded picture buffers (DPBs);

means for receiving, from the encoded video bitstream, a first respective syntax element for each output layer set of the plurality of output layer sets, wherein each first respective syntax element is indicative of whether or not sub-DPB parameters are signaled for each sub-layer of each respective output layer set;

means for receiving, from the encoded video bitstream, a second respective syntax element for each sub-layer of the one or more sub-layers other than the first sub-layer of a first particular output layer set of the plurality of output layer sets but not for the first sub-layer of the first particular output layer set, wherein each second respective syntax element is indicative of whether or not sub-DPB parameters are signaled for each respective sub-layer of the one or more sub-layers other than the first sub-layer of the first particular output layer set;

means for receiving, from the encoded video bitstream, a respective set of sub-DPB parameters for the first sub-layer of each respective plurality of sub-layers; and

means for performing a sub-DPB management process on the one or more sub-DPBs in accordance with each respective set of sub-DPB parameters received.

20. A non-transitory computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors of a device configured to decode video data to:

receive, from an encoded video bitstream, a plurality of output layer sets, wherein each output layer set of the plurality of output layer sets includes a respective plurality of sub-layers, wherein each respective plurality of sub-layers includes a first sub-layer and one or more sub-layers other than the first sub-layer, wherein the first sub-layer of each respective plurality of sub-layers is a base sub-layer;

decode each respective plurality of sub-layers of the plurality of output layer sets;

store each respective decoded plurality of sub-layers in one or more sub- DPBs;

receive, from the encoded video bitstream, a first respective syntax element for each output layer set of the plurality of output layer sets, wherein each first respective syntax element is indicative of whether or not sub-DPB parameters are signaled for each sub-layer of each respective output layer set;

receive, from the encoded video bitstream, a second respective syntax element for each sub-layer of the one or more sub-layers other than the first sub-layer of a first particular output layer set of the plurality of output layer sets but not for the first sub-layer of the first particular output layer set, wherein each second respective syntax element is indicative of whether or not sub-DPB parameters are signaled for each respective sub-layer of the one or more sub-layers other than the first sub-layer of the first particular output layer set;

receive, from the encoded video bitstream, a respective set of sub-DPB parameters for the first sub-layer of each respective plurality of sub-layers; and

perform a sub-DPB management process on the one or more sub-DPBs in accordance with each respective set of sub-DPB parameters received.

21. A method of encoding video data, the method comprising:

reconstructing a plurality of output layer sets of an encoded video bitstream, wherein each output layer set of the plurality of output layer sets includes a respective plurality of sub-layers, wherein each respective plurality of sub-layers includes a first sub-layer and one or more sub-layers other than the first sub-layer, wherein the first sub-layer of each respective plurality of sub-layers is a base sub-layer, wherein reconstructing the plurality of output layer sets includes reconstructing each respective plurality of sub-layers of the plurality of output layer sets;

storing each respective reconstructed plurality of sub-layers in one or more sub-decoded picture buffers (DPBs);

generating a first respective syntax element for each output layer set of the plurality of output layer sets, wherein each first respective syntax element is indicative of whether or not sub-DPB parameters are signaled for each sub-layer of each respective output layer set;

generating a second respective syntax element for each sub-layer of the one or more sub-layers other than the first sub-layer of a first particular output layer set of the plurality of output layer sets but not for the first sub-layer of the first particular output layer set, wherein each second respective syntax element is indicative of whether or not sub-DPB parameters are signaled for each respective sub-layer of the one or more sub-layers other than the first sub-layer of the first particular output layer set;

generating a respective set of sub-DPB parameters for the first sub-layer of each respective plurality of sub-layers;

performing a sub-DPB management process on the one or more sub-DPBs in accordance with each respective set of sub-DPB parameters received; and

generating each respective set of sub-DPB parameters in the encoded video bitstream.

22. The method of claim 21, wherein each respective set of sub-DPB parameters for the first sub-layer of each respective plurality of sub-layers includes at least one of:

a respective third syntax element indicative of a maximum number of reorder pictures allowed for the one or more sub-DPBs;

or

a respective fourth syntax element indicative of a maximum latency allowed for the one or more sub-DPBs.

23. The method of claim 21, wherein generating the respective set of sub-DPB parameters comprises generating the respective set of sub-DPB parameters in a video parameter set (VPS).

24. The method of claim 21, further comprising:

generating, in the encoded video bitstream, each first respective syntax element for each output layer set of the plurality of output layer sets; and

generating, in the encoded video bitstream, each second respective syntax element for each sub-layer of the one or more sub-layers other than the first sub-layer of a first particular output layer set of the plurality of output layer sets.

25. The method of claim 21, further comprising:

generating each second respective syntax element for each sub-layer of the one or more sub-layers other than the first sub-layer of the first particular output layer set based on the first respective syntax element indicating that sub-DPB parameters are signaled for each sub-layer of the first particular output layer set.

26. The method of claim 21, further comprising:

generating, based on the second respective syntax element indicating that sub-DPB parameters are signaled for a particular sub-layer of the one or more sub-layers other than the first sub-layer of the first particular output layer set, a respective set of sub-DPB parameters for the particular sub-layer.

27. The method of claim 21, wherein the sub-DPB management process includes at least one of marking pictures as unused for reference, marking pictures for output, or removing pictures from the sub-DPB, and wherein the performing the sub-DPB management process comprises performing the sub-DPB management process on an access unit level using the received respective sets of sub-DPB parameters.

28. The method of claim 27, wherein performing the sub-DPB management process on the access unit level comprises:

outputting pictures based on the number of access units in a first sub-DPB of the one or more sub-DPBs that contain at least one picture marked as needed for output;

marking pictures as unused for reference on a per-layer level; and

removing pictures from the first sub-DPB on a per-layer level.

29. An apparatus configured to encode video data, the apparatus comprising:

one or more memory devices comprising one or more sub-decoded picture buffers (DPBs) configured to store video data; and

a video encoder configured to;

reconstruct a plurality of output layer sets of an encoded video bitstream, wherein each output layer set of the plurality of output layer sets includes a respective plurality of sub-layers, wherein each respective plurality of sub-layers includes a first sub-layer and one or more sub-layers other than the first sub-layer, wherein the first sub-layer of each respective plurality of sub-layers is a base sub-layer, wherein reconstructing the plurality of output layer sets includes reconstructing each respective plurality of sub-layers of the plurality of output layer sets;

store each respective reconstructed plurality of sub-layers in the one or more sub-DPBs;

generate a first respective syntax element for each output layer set of the plurality of output layer sets, wherein each first respective syntax element is indicative of whether or not sub-DPB parameters are signaled for each sub-layer of each respective output layer set;

generate a second respective syntax element for each sub-layer of the one or more sub-layers other than the first sub-layer of a first particular output layer set of the plurality of output layer sets but not for the first sub-layer of the first particular output layer set, wherein each second respective syntax element is indicative of whether or not sub-DPB parameters are signaled for each respective sub-layer of the one or more sub-layers other than the first sub-layer of the first particular output layer set;

generate a respective set of sub-DPB parameters for the first sub-layer of each respective plurality of sub-layers;

perform a sub-DPB management process on the one or more sub-DPBs in accordance with each respective set of sub-DPB parameters received; and

generate each respective set of sub-DPB parameters in the encoded video bitstream.

30. The apparatus of claim 29, wherein each respective set of sub-DPB parameters for the first sub-layer of each respective plurality of sub-layers includes at least one of:

a respective third syntax element indicative of a maximum number of reorder pictures allowed for the one or more sub-DPBs;

or

a respective fourth syntax element indicative of a maximum latency allowed for the one or more sub-DPBs.

31. The apparatus of claim 29, wherein the video encoder is configured to generate the respective set of sub-DPB parameters in a video parameter set (VPS).

32. The apparatus of claim 29, wherein the video encoder is further configured to:

generate, in the encoded video bitstream, each first respective syntax element for each output layer set of the plurality of output layer sets; and

generate, in the encoded video bitstream, each second respective syntax element for each sub-layer of the one or more sub-layers other than the first sub-layer of a first particular output layer set of the plurality of output layer sets.

33. The apparatus of claim 29, wherein the video encoder is further configured to:

generate each second respective syntax element for each sub-layer of the one or more sub-layers other than the first sub-layer of the first particular output layer set based on the first respective syntax element indicating that sub-DPB parameters are signaled for each sub-layer of the first particular output layer set.

34. The apparatus of claim 29, wherein the video encoder is further configured to:

generate, based on the second respective syntax element indicating that sub-DPB parameters are signaled for a particular sub-layer of the one or more sub-layers other than the first sub-layer of the first particular output layer set, a respective set of sub-DPB parameters for the particular sub-layer.

35. The apparatus of claim 29, wherein the sub-DPB management process includes at least one of marking pictures as unused for reference, marking pictures for output, or removing pictures from the sub-DPB, and wherein the performing the sub-DPB management process comprises performing the sub-DPB management process on an access unit level using the received respective sets of sub-DPB parameters.

36. The apparatus of claim 35, wherein the video encoder is further configured to:

output pictures based on the number of access units in a first sub-DPB of the one or more sub-DPBs that contain at least one picture marked as needed for output;

mark pictures as unused for reference on a per-layer level; and

remove pictures from the first sub-DPB on a per-layer level.

37. An apparatus configured to encode video data, the apparatus comprising:

means for reconstructing a plurality of output layer sets of an encoded video bitstream, wherein each output layer set of the plurality of output layer sets includes a respective plurality of sub-layers, wherein each respective plurality of sub-layers includes a first sub-layer and one or more sub-layers other than the first sub-layer, wherein the first sub-layer of each respective plurality of sub-layers is a base sub-layer, wherein reconstructing the plurality of output layer sets includes reconstructing each respective plurality of sub-layers of the plurality of output layer sets;

means for storing each respective reconstructed plurality of sub-layers in one or more sub-decoded picture buffers (DPBs);

means for generating a first respective syntax element for each output layer set of the plurality of output layer sets, wherein each first respective syntax element is indicative of whether or not sub-DPB parameters are signaled for each sub-layer of each respective output layer set;

means for generating a second respective syntax element for each sub-layer of the one or more sub-layers other than the first sub-layer of a first particular output layer set of the plurality of output layer sets but not for the first sub-layer of the first particular output layer set, wherein each second respective syntax element is indicative of whether or not sub-DPB parameters are signaled for each respective sub-layer of the one or more sub-layers other than the first sub-layer of the first particular output layer set;

means for generating a respective set of sub-DPB parameters for the first sub-layer of each respective plurality of sub-layers;

means for performing a sub-DPB management process on the one or more sub-DPBs in accordance with each respective set of sub-DPB parameters received; and

means for generating each respective set of sub-DPB parameters in the encoded video bitstream.

38. A non-transitory computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors of a device configured to encode video data to:

reconstruct a plurality of output layer sets of an encoded video bitstream, wherein each output layer set of the plurality of output layer sets includes a respective plurality of sub-layers, wherein each respective plurality of sub-layers includes a first sub-layer and one or more sub-layers other than the first sub-layer, wherein the first sub-layer of each respective plurality of sub-layers is a base sub-layer, wherein reconstructing the plurality of output layer sets includes reconstructing each respective plurality of sub-layers of the plurality of output layer sets;

store each respective reconstructed plurality of sub-layers in one or more sub-decoded picture buffers (DPBs);

generate a first respective syntax element for each output layer set of the plurality of output layer sets, wherein each first respective syntax element is indicative of whether or not sub-DPB parameters are signaled for each sub-layer of each respective output layer set;

generate a second respective syntax element for each sub-layer of the one or more sub-layers other than the first sub-layer of a first particular output layer set of the plurality of output layer sets but not for the first sub-layer of the first particular output layer set, wherein each second respective syntax element is indicative of whether or not sub-DPB parameters are signaled for each respective sub-layer of the one or more sub-layers other than the first sub-layer of the first particular output layer set;

generate a respective set of sub-DPB parameters for the first sub-layer of each respective plurality of sub-layers;

perform a sub-DPB management process on the one or more sub-DPBs in accordance with each respective set of sub-DPB parameters received; and

generate each respective set of sub-DPB parameters in the encoded video bitstream.