Carriage of HEVC extension bitstreams and buffer model with MPEG-2 systems

• US 10,567,804 B2
• Filed: 01/07/2015
• Issued: 02/18/2020
• Est. Priority Date: 01/08/2014
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Chen, Ying, Wang, Ye-Kui
• Primary Examiner(s): Atala, Jamie J
• Assistant Examiner(s): Demosky, Patrick E
• Application Number:
  US14/591,649
• Publication Number:
  US 20150195578A1
• Time in Patent Office: 1,868 Days
• Field of Search: 37524026
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/423 characterised by memory arrangements H04N19/433 takes precedence
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 21/234327 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 21/236 Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream multiplexing of data packets for data networks, e.g. RTP/UDP H04L65/00
• CPC Class Codes: H04N 21/2401 Monitoring of the client buffer
• CPC Class Codes: H04N 21/44004 involving video buffer management, e.g. video decoder buffer or video display buffer
• CPC Class Codes: H04N 21/8451 using Advanced Video Coding [AVC]

Abstract

A video decoder assembles, in a buffer model, an access unit from a plurality of elementary streams of a video data stream. The video data stream may be a transport stream or a program stream. The same buffer model is used regardless of whether the elementary streams contain Scalable High Efficiency Video Coding (SHVC), Multi-View HEVC (MV-HEVC), or 3D-HEVC bitstreams. Furthermore, the video decoder decodes the access unit.

Claims

1. A method of decoding video data, the method comprising:

receiving a video data stream comprising a plurality of elementary streams and a program map table (PMT) separate from the plurality of elementary streams, the PMT comprising information about which of the elementary streams comprises a program, the PMT including a High Efficiency Video Coding (HEVC) extension descriptor, and the PMT including a plurality of hierarchy extension descriptors, wherein;

the HEVC extension descriptor signals a current operation point that corresponds to an output layer set, the HEVC extension descriptor comprising a maximum temporal identifier syntax element and a set of output layer flags, the maximum temporal identifier syntax element indicating a highest temporal identifier of Network Abstraction Layer (NAL) units in the current operation point, each output layer flag in the set of output layer flags indicating whether a different corresponding layer is in the output layer set of the current operation point, and the set of output layer flags including at least one output layer flag indicating the corresponding layer is not in the output layer set of the current operation point,

each respective hierarchy extension descriptor of the plurality of hierarchy extension descriptors corresponds to a respective elementary stream in the plurality of elementary streams, each respective elementary stream of the plurality of elementary streams being a HEVC extension video stream in an MPEG-2 system, the hierarchy extension descriptors comprising respective sets of values,

for each respective hierarchy extension descriptor of the plurality of hierarchy extension descriptors, the respective hierarchy extension descriptor includes a respective number of embedded layers element specifying a number of direct dependent program elements that need to be accessed and be present in decoding order before decoding of the elementary stream corresponding to the respective hierarchy extension descriptor, a total number of values in the respective set of values is equal to the number specified by the respective number of embedded layers element, each value in the respective set of values defines a hierarchy layer index of a different program element that needs to be accessed and be present in decoding order before decoding the elementary stream corresponding to the respective hierarchy extension descriptor, and

for at least a particular hierarchy extension descriptor of the plurality of hierarchy extension descriptors, the respective number of embedded layers element of the particular hierarchy extension descriptor indicates that there are two or more direct dependent program elements that need to be accessed and be present in decoding order before decoding of the elementary stream corresponding to the respective hierarchy extension descriptor;

assembling, in a buffer model, HEVC layer pictures within an access unit from the plurality of elementary streams of the video data stream, wherein;

the buffer model is a transport stream system target decoder model or a program stream system target decoder model,

the video data stream is a transport stream or a program stream, and

the same buffer model is used to assemble the HEVC layer pictures within the access unit regardless of whether the elementary streams in the plurality of elementary streams contain any of a plurality of different types of multi-layer coded bitstreams, and

assembling the HEVC layer pictures within the access unit comprises identifying, based on the sets of values in the hierarchy extension descriptors, a plurality of reference layers required for decoding the output layer set of the current operation point; and

decoding the access unit.

2. The method of claim 1, wherein the plurality of different types of multi-layer coded bitstreams include Scalable High Efficiency Video Coding (SHVC), Multi-View HEVC (MV-HEVC), and 3D-HEVC bitstreams.

3. The method of claim 1, further comprising:

assembling HEVC layer pictures within access units using separate instances of the buffer model for each respective HEVC layered video stream of the video data stream, wherein;

each respective HEVC layered video stream comprises a plurality of HEVC video layer sub-bitstreams, and

each respective HEVC video layer sub-bitstream of the plurality of HEVC video layer sub-bitstreams comprises Video Coding Layer (VCL) Network Abstraction Layer (NAL) units with a same layer identifier value.

4. The method of claim 1, wherein:

for each respective elementary stream associated with the program;

the buffer model comprises a buffer for the respective elementary stream,

the access unit comprises a respective HEVC layer picture subset for the respective elementary stream,

the respective HEVC layer picture subset for the respective elementary stream comprises HEVC layer pictures within the access unit that are associated with a respective layer identifier set,

each of the HEVC layer pictures within the access unit is a coded picture as defined in Rec. ITU-T H.265|ISO/IEC 23008-2 Annex F, and

assembling the HEVC layer pictures within the access unit comprises, for each respective elementary stream associated with the program;

removing the respective HEVC layer picture subset for the respective elementary stream from the buffer for the respective elementary stream; and

including the respective HEVC layer picture subset for the respective elementary stream in the access unit.

5. The method of claim 4, wherein:

the video data stream is a transport stream,

for each respective elementary stream associated with the program;

the buffer for the respective elementary stream is a first buffer for the respective elementary stream,

the buffer model comprises a second buffer for the respective elementary stream; and

the method further comprises, for each respective packetized elementary stream (PES) packet of the transport stream belonging to the respective elementary stream, storing the respective PES packet in the second buffer for the respective elementary stream.

6. The method of claim 5, wherein:

for each respective elementary stream associated with the program;

the buffer model comprises a third buffer for the respective elementary stream; and

the method further comprises;

removing PES packets from the second buffer for the respective elementary stream;

storing, in the third buffer for the respective elementary stream, the PES packets removed from the second buffer for the respective elementary stream;

removing bytes from the third buffer for the respective elementary stream; and

storing, in the first buffer for the respective elementary stream, the bytes removed from the third buffer for the respective elementary stream.

7. The method of claim 1, wherein:

the method further comprises, responsive to determining that there is a set of HEVC layers in the program and that there is at least one HEVC layered video sub-bitstream in the plurality of elementary streams that is an HEVC extension video stream conforming to one or more profiles as defined in Annex G or Annex H of ITU-T Rec. H.265|ISO/IEC 23008-2, selecting the buffer model to use in assembling the HEVC layer pictures within the access unit.

8. A video decoding device comprising:

a memory configured to store video data; and

one or more processors configured to;

receive a video data stream comprising a plurality of elementary streams and a program map table (PMT) separate from the plurality of elementary streams, the PMT comprising information about which of the elementary streams comprises a program, the PMT including a High Efficiency Video Coding (HEVC) extension descriptor, and the PMT including a plurality of hierarchy extension descriptors, wherein;

the HEVC extension descriptor signals a current operation point that corresponds to an output layer set, the HEVC extension descriptor comprising a maximum temporal identifier syntax element and a set of output layer flags, the maximum temporal identifier syntax element indicating a highest temporal identifier of Network Abstraction Layer (NAL) units in the current operation point, each output layer flag in the set of output layer flags indicating whether a different corresponding layer is in the output layer set of the current operation point, and the set of output layer flags including at least one output layer flag indicating the corresponding layer is not in the output layer set of the current operation point,

each respective hierarchy extension descriptor of the plurality of hierarchy extension descriptors corresponds to a respective elementary stream in the plurality of elementary streams, each respective elementary stream of the plurality of elementary streams being a HEVC extension video stream in an MPEG-2 system, the hierarchy extension descriptors comprising respective sets of values, for each respective hierarchy extension descriptor of the plurality of hierarchy extension descriptors, the respective hierarchy extension descriptor includes a respective number of embedded layers element specifying a number of direct dependent program elements that need to be accessed and be present in decoding order before decoding of the elementary stream corresponding to the respective hierarchy extension descriptor, a total number of values in the respective set of values is equal to the number specified by the respective number of embedded layers element, each value in the respective set of values defines a hierarchy layer index of a different program element that needs to be accessed and be present in decoding order before decoding the elementary stream corresponding to the respective hierarchy extension descriptor, and

for at least a particular hierarchy extension descriptor of the plurality of hierarchy extension descriptors, the respective number of embedded layers element of the particular hierarchy extension descriptor indicates that there are two or more direct dependent program elements that need to be accessed and be present in decoding order before decoding of the elementary stream corresponding to the respective hierarchy extension descriptor;

assemble, in a buffer model, HEVC layer pictures within an access unit from the plurality of elementary streams of the video data stream, wherein;

the buffer model is a transport stream system target decoder model or a program stream system target decoder model,

the video data stream is a transport stream or a program stream, and

the same buffer model is used to assemble the HEVC layer pictures within the access unit regardless of whether the elementary streams in the plurality of elementary streams contain any of a plurality of different types of multi-layer coded bitstreams, and

assembling the HEVC layer pictures within the access unit comprises identifying, based on the sets of values in the hierarchy extension descriptors, a plurality of reference layers required for decoding the output layer set of the current operation point; and

decode the access unit.

9. The video decoding device of claim 8, wherein the plurality of different types of multi-layer coded bitstreams include Scalable High Efficiency Video Coding (SHVC), Multi-View HEVC (MV-HEVC), and 3D-HEVC bitstreams.

10. The video decoding device of claim 8, wherein the one or more processors are configured to assemble HEVC layer pictures within access units using separate instances of the buffer model for each respective HEVC layered video stream of the video data stream, wherein:

each respective HEVC layered video stream comprises a plurality of HEVC video layer sub-bitstreams, and

each respective HEVC video layer sub-bitstream of the plurality of HEVC video layer sub-bitstreams comprises Video Coding Layer (VCL) Network Abstraction Layer (NAL) units with a same layer identifier value.

11. The video decoding device of claim 8, wherein:

for each respective elementary stream associated with the program;

the buffer model comprises a buffer for the respective elementary stream,

the access unit comprises a respective HEVC layer picture subset for the respective elementary stream,

the respective HEVC layer picture subset for the respective elementary stream comprises HEVC layer pictures within the access unit that are associated with a respective layer identifier set,

each of the HEVC layer pictures within the access unit is a coded picture as defined in Rec. ITU-T H.265|ISO/IEC 23008-2 Annex F, and

as part of assembling the HEVC layer pictures within the access unit, the one or more processors, for each respective elementary stream associated with the program;

remove the respective HEVC layer picture subset for the respective elementary stream from the buffer for the respective elementary stream; and

include the respective HEVC layer picture subset for the respective elementary stream in the access unit.

12. The video decoding device of claim 11, wherein:

the video data stream is a transport stream,

for each respective elementary stream associated with the program;

the buffer for the respective elementary stream is a first buffer for the respective elementary stream,

the buffer model comprises a second buffer for the respective elementary stream; and

the one or more processors are configured to, for each respective packetized elementary stream (PES) packet of the transport stream belonging to the respective elementary stream, store the respective PES packet in the second buffer for the respective elementary stream.

13. The video decoding device of claim 12, wherein:

for each respective elementary stream associated with the program;

the buffer model comprises a third buffer for the respective elementary stream; and

the one or more processors are configured to;

remove PES packets from the second buffer for the respective elementary stream;

store, in the third buffer for the respective elementary stream, the PES packets removed from the second buffer for the respective elementary stream;

remove bytes from the third buffer for the respective elementary stream; and

store, in the first buffer for the respective elementary stream, the bytes removed from the third buffer for the respective elementary stream.

14. The video decoding device of claim 8, wherein:

the one or more processors are further configured to, responsive to determining that there is a set of HEVC layers in the program and that there is at least one HEVC layered video sub-bitstream in the plurality of elementary streams that is an HEVC extension video stream conforming to one or more profiles as defined in Annex G or Annex H of ITU-T Rec. H.265|ISO/IEC 23008-2, select the buffer model to use in assembling the HEVC layer pictures within the access unit.

15. A video decoding device comprising:

means for receiving a video data stream comprising a plurality of elementary streams and a program map table (PMT) separate from the plurality of elementary streams, the PMT comprising information about which of the elementary streams comprises a program, the PMT including a High Efficiency Video Coding (HEVC) extension descriptor, and the PMT including a plurality of hierarchy extension descriptors, wherein;

the HEVC extension descriptor signals a current operation point that corresponds to an output layer set, the HEVC extension descriptor comprising a maximum temporal identifier syntax element and a set of output layer flags, the maximum temporal identifier syntax element indicating a highest temporal identifier of Network Abstraction Layer (NAL) units in the current operation point, each output layer flag in the set of output layer flags indicating whether a different corresponding layer is in the output layer set of the current operation point, and the set of output layer flags including at least one output layer flag indicating the corresponding layer is not in the output layer set of the current operation point,

each respective hierarchy extension descriptor of the plurality of hierarchy extension descriptors corresponds to a respective elementary stream in the plurality of elementary streams, each respective elementary stream of the plurality of elementary streams being a HEVC extension video stream in an MPEG-2 system, the hierarchy extension descriptors comprising respective sets of values,

for each respective hierarchy extension descriptor of the plurality of hierarchy extension descriptors, the respective hierarchy extension descriptor includes a respective number of embedded layers element specifying a number of direct dependent program elements that need to be accessed and be present in decoding order before decoding of the elementary stream corresponding to the respective hierarchy extension descriptor, a total number of values in the respective set of values is equal to the number specified by the respective number of embedded layers element, each value in the respective set of values defines a hierarchy layer index of a different program element that needs to be accessed and be present in decoding order before decoding the elementary stream corresponding to the respective hierarchy extension descriptor, and

for at least a particular hierarchy extension descriptor of the plurality of hierarchy extension descriptors, the respective number of embedded layers element of the particular hierarchy extension descriptor indicates that there are two or more direct dependent program elements that need to be accessed and be present in decoding order before decoding of the elementary stream corresponding to the respective hierarchy extension descriptor;

means for assembling, in a buffer model, HEVC layer pictures within an access unit from the plurality of elementary streams of the video data stream, wherein;

the buffer model is a transport stream system target decoder model or a program stream system target decoder model,

the video data stream is a transport stream or a program stream, and

the same buffer model is used to assemble the HEVC layer pictures within the access unit regardless of whether the elementary streams in the plurality of elementary streams contain any of a plurality of different types of multi-layer coded bitstreams, and

assembling the HEVC layer pictures within the access unit comprises identifying, based on the sets of values in the hierarchy extension descriptors, a plurality of reference layers required for decoding the output layer set of the current operation point; and

means for decoding the access unit, the access unit comprising one or more pictures of the video data.

16. The video decoding device of claim 15, wherein the plurality of different types of multi-layer coded bitstreams include Scalable High Efficiency Video Coding (SHVC), Multi-View HEVC (MV-HEVC), and 3D-HEVC bitstreams.

17. The video decoding device of claim 15, further comprising:

means for assembling HEVC layer pictures within access units using separate instances of the buffer model for each respective HEVC layered video stream of the video data stream, wherein;

each respective HEVC layered video stream comprises a plurality of HEVC video layer sub-bitstreams, and

each respective HEVC video layer sub-bitstream of the plurality of HEVC video layer sub-bitstreams comprises Video Coding Layer (VCL) Network Abstraction Layer (NAL) units with a same layer identifier value.

18. The video decoding device of claim 15, wherein:

for each respective elementary stream associated with the program;

the buffer model comprises a buffer for the respective elementary stream,

the access unit comprises a respective HEVC layer picture subset for the respective elementary stream,

the respective HEVC layer picture subset for the respective elementary stream comprises HEVC layer pictures within the access unit that are associated with a respective layer identifier set,

each of the HEVC layer pictures within the access unit is a coded picture as defined in Rec. ITU-T H.265|ISO/IEC 23008-2 Annex F, and

the means for assembling the HEVC layer pictures within the access unit comprises, means for, for each respective elementary stream associated with the program;

removing the respective HEVC layer picture subset for the respective elementary stream from the buffer for the respective elementary stream; and

including the respective HEVC layer picture subset for the respective elementary stream in the access unit.

19. The video decoding device of claim 18, wherein:

the video data stream is a transport stream,

for each respective elementary stream associated with the program;

the buffer for the respective elementary stream is a first buffer for the respective elementary stream,

the buffer model comprises a second buffer for the respective elementary stream; and

the video decoding device further comprises means for storing, for each respective packetized elementary stream (PES) packet of the transport stream belonging to the respective elementary stream, the respective PES packet in the second buffer for the respective elementary stream.

20. The video decoding device of claim 19, wherein:

for each respective elementary stream associated with the program;

the buffer model comprises a third buffer for the respective elementary stream; and

the video decoding device further comprises;

means for removing PES packets from the second buffer for the respective elementary stream;

means for storing, in the third buffer for the respective elementary stream, the PES packets removed from the second buffer for the respective elementary stream;

means for removing bytes from the third buffer for the respective elementary stream; and

means for storing, in the first buffer for the respective elementary stream, the bytes removed from the third buffer for the respective elementary stream.

21. The video decoding device of claim 15, wherein:

the video decoding device further comprises, means for selecting, responsive to determining that there is a set of HEVC layers in the program and that there is at least one HEVC layered video sub-bitstream in the plurality of elementary streams that is an HEVC extension video stream conforming to one or more profiles as defined in Annex G or Annex H of ITU-T Rec. H.265|ISO/IEC 23008-2, the buffer model to use in assembling the HEVC layer pictures within the access unit.

22. A computer-readable data storage medium having instructions stored thereon that, when executed, cause a video decoding device to:

receive a video data stream comprising a plurality of elementary streams and a program map table (PMT) separate from the plurality of elementary streams, the PMT comprising information about which of the elementary streams comprises a program, the PMT including a High Efficiency Video Coding (HEVC) extension descriptor, and the PMT including a plurality of hierarchy extension descriptors, wherein;

the HEVC extension descriptor signals a current operation point that corresponds to an output layer set, the HEVC extension descriptor comprising a maximum temporal identifier syntax element and a set of output layer flags, the maximum temporal identifier syntax element indicating a highest temporal identifier of Network Abstraction Layer (NAL) units in the current operation point, each output layer flag in the set of output layer flags indicating whether a different corresponding layer is in the output layer set of the current operation point, and the set of output layer flags including at least one output layer flag indicating the corresponding layer is not in the output layer set of the current operation point,

each respective hierarchy extension descriptor of the plurality of hierarchy extension descriptors corresponds to a respective elementary stream in the plurality of elementary streams, each respective elementary stream of the plurality of elementary streams being a HEVC extension video stream in an MPEG-2 system, the hierarchy extension descriptors comprising respective sets of values,

for each respective hierarchy extension descriptor of the plurality of hierarchy extension descriptors, the respective hierarchy extension descriptor includes a respective number of embedded layers element specifying a number of direct dependent program elements that need to be accessed and be present in decoding order before decoding of the elementary stream corresponding to the respective hierarchy extension descriptor, a total number of values in the respective set of values is equal to the number specified by the respective number of embedded layers element, each value in the respective set of values defines a hierarchy layer index of a different program element that needs to be accessed and be present in decoding order before decoding the elementary stream corresponding to the respective hierarchy extension descriptor, and

for at least a particular hierarchy extension descriptor of the plurality of hierarchy extension descriptors, the respective number of embedded layers element of the particular hierarchy extension descriptor indicates that there are two or more direct dependent program elements that need to be accessed and be present in decoding order before decoding of the elementary stream corresponding to the respective hierarchy extension descriptor;

assemble, in a buffer model, HEVC layer pictures within an access unit from the plurality of elementary streams of the video data stream, wherein;

the buffer model is a transport stream system target decoder model or a program stream system target decoder model,

the video data stream is a transport stream or a program stream, and

the same buffer model is used to assemble the HEVC layer pictures within the access unit regardless of whether the elementary streams in the plurality of elementary streams contain any of a plurality of different types of multi-layer coded bitstreams, and

assembling the HEVC layer pictures within the access unit comprises identifying, based on the sets of values in the hierarchy extension descriptors, a plurality of reference layers required for decoding the output layer set of the current operation point; and

decode the access unit.

23. The computer-readable data storage medium of claim 22, wherein the plurality of different types of multi-layer coded bitstreams include Scalable High Efficiency Video Coding (SHVC), Multi-View HEVC (MV-HEVC), and 3D-HEVC bitstreams.

24. The computer-readable data storage medium of claim 22, the instructions further causing the video decoding device to:

assemble HEVC layer pictures within access units using separate instances of the buffer model for each respective HEVC layered video stream of the video data stream, wherein;

each respective HEVC layered video stream comprises a plurality of HEVC video layer sub-bitstreams, and

each respective HEVC video layer sub-bitstream of the plurality of HEVC video layer sub-bitstreams comprises Video Coding Layer (VCL) Network Abstraction Layer (NAL) units with a same layer identifier value.

25. The computer-readable data storage medium of claim 22, wherein:

for each respective elementary stream associated with the program;

the buffer model comprises a buffer for the respective elementary stream,

the access unit comprises a respective HEVC layer picture subset for the respective elementary stream,

the respective HEVC layer picture subset for the respective elementary stream comprises HEVC layer pictures within the access unit that are associated with a respective layer identifier set,

each of the HEVC layer pictures within the access unit is a coded picture as defined in Rec. ITU-T H.265|ISO/IEC 23008-2 Annex F, and

as part of assembling the HEVC layer pictures within the access unit, for each respective elementary stream associated with the program, the instructions cause the video decoding device to;

remove the respective HEVC layer picture subset for the respective elementary stream from the buffer for the respective elementary stream; and

include the respective HEVC layer picture subset for the respective elementary stream in the access unit.

26. The computer-readable data storage medium of claim 25, wherein:

the video data stream is a transport stream,

for each respective elementary stream associated with the program;

the buffer for the respective elementary stream is a first buffer for the respective elementary stream,

the buffer model comprises a second buffer for the respective elementary stream; and

the instructions further cause the video decoding device to store, for each respective packetized elementary stream (PES) packet of the transport stream belonging to the respective elementary stream, the respective PES packet in the second buffer for the respective elementary stream.

27. The computer-readable data storage medium of claim 26, wherein:

for each respective elementary stream associated with the program;

the buffer model comprises a third buffer for the respective elementary stream; and

the instructions further cause the video decoding device to;

remove PES packets from the second buffer for the respective elementary stream;

store, in the third buffer for the respective elementary stream, the PES packets removed from the second buffer for the respective elementary stream;

remove bytes from the third buffer for the respective elementary stream; and

store, in the first buffer for the respective elementary stream, the bytes removed from the third buffer for the respective elementary stream.

28. The computer-readable data storage medium of claim 22, wherein:

the instructions further cause the video decoding device to select, responsive to determining that there is a set of HEVC layers in the program and that there is at least one HEVC layered video sub-bitstream in the plurality of elementary streams that is an HEVC extension video stream conforming to one or more profiles as defined in Annex G or Annex H of ITU-T Rec. H.265|ISO/IEC 23008-2, the buffer model to use in assembling the HEVC layer pictures within the access unit.