Multi-layer encoding and decoding

• US 9,769,485 B2
• Filed: 01/20/2017
• Issued: 09/19/2017
• Est. Priority Date: 09/16/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Lu, Mei-Hsuan, Lee, Ming-Chieh
• Primary Examiner(s): Patel, Jay
• Assistant Examiner(s): Habib, Irfan
• Application Number:
  US15/411,832
• Publication Number:
  US 20170134737A1
• Time in Patent Office: 242 Days
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/39 involving multiple description coding [MDC], i.e. with separate layers being structured as independently decodable descriptions of input picture data

Abstract

Innovations described herein provide a generic encoding and decoding framework that includes some features of simulcast and some features of scalable video coding. For example, a bitstream multiplexer multiplexes component bitstreams into a multi-layer encoding (MLE) bitstream that provides temporal scalability, spatial resolution scalability and/or signal to noise ratio scalability. Each of the component bitstreams provides an alternative version of input video, and a given component bitstream can be a non-scalable bitstream or scalable bitstream. The multiplexer follows composition rules for the MLE bitstream and may rewrite values of certain syntax elements of component bitstreams using an approach that avoids bit shifting operations. A corresponding demultiplexer receives an MLE bitstream that includes component bitstreams and demultiplexes at least part of at least one of the component bitstreams from the MLE bitstream, following decomposition rules for the demultiplexing.

Claims

1. A computing system comprising:

memory configured to store multiple component bitstreams; and

a bitstream multiplexer configured to perform operations comprising;

multiplexing the multiple component bitstreams into a multi-layer encoding (MLE) bitstream, wherein each of the multiple component bitstreams provides an alternative version of input video, and wherein the multiplexing includes inserting multiple marker data units in the MLE bitstream, each of the multiple marker data units including a priority identifier and one or more quality layer identifiers, the priority identifier associating a video data unit with a given layer of one of the multiple component bitstreams, the multiple component bitstreams including at least some parameter set identifiers having values set to avoid conflicting uses of the values between the multiple component bitstreams in the MLE bitstream, such that each of the values is unique across the multiple component bitstreams; and

outputting the MLE bitstream.

2. The computing system of claim 1 further comprising:

a video encoder configured to perform operations comprising;

encoding the input video to produce the multiple component bitstreams; and

during the encoding, setting the values of the at least some parameter set identifiers in the multiple component bitstreams to avoid conflicting uses of the values in the MLE bitstream.

3. The computing system of claim 1 wherein the operations of the bitstream multiplexer further comprise:

receiving the multiple component bitstreams; and

rewriting the values of the at least some parameter set identifiers in the multiple component bitstreams to avoid conflicting uses of the values in the MLE bitstream.

4. The computing system of claim 3 wherein the operations of the bitstream multiplexer further comprise, as part of the rewriting, for a given value of the values of the at least some parameter set identifiers in the multiple component bitstreams:

identifying a set of candidate values appropriate for the given value based on number of bits in the given value;

selecting a new value from the set of candidate values; and

replacing the given value with the new value.

5. The computing system of claim 4 wherein the set of candidate values includes one or more candidate values having the number of bits in the given value and multiple candidate values having one or more bytes plus the number of bits in the given value, whereby the rewriting can be performed without bit shifting operations.

6. The computing system of claim 1 wherein at least one of the multiple component bitstreams is a scalable bitstream that includes a base layer and an enhancement layer, and wherein:

the base layer represents a first temporal resolution of the input video and the enhancement layer represents a second temporal resolution higher than the first temporal resolution; and

/or

the base layer represents a first signal to noise ratio (SNR) level of the input video and the enhancement layer represents a second SNR level higher than the first SNR level.

7. The computing system of claim 6 wherein the alternative versions of the input video are alternative simulcast versions of the input video, such that the multiple component bitstreams provide multiple spatial resolution options between the multiple component bitstreams.

8. The computing system of claim 1 wherein each of the multiple component bitstreams differs from the other bitstreams of the multiple component bitstreams in terms of one or more of:

number of layers of temporal, spatial and/or signal to noise ratio (SNR) scalability, if any, supported in that component bitstream;

format of decoding of the component bitstream; and

profile and/or level for decoding of the component bitstream.

9. The computing system of claim 1 wherein the multiplexing enforces multiple composition rules, the multiple composition rules including:

a first composition rule according to which priority identifiers identify layers of the multiple component bitstreams, wherein the priority identifiers are included in the marker data units for at least some slice data units and are directly included as part of other slice data units;

a second composition rule according to which each of the multiple component bitstreams is assigned a non-overlapping range of the priority identifiers within which a base layer for the component bitstream has the lowest priority identifier in the range;

a third composition rule according to which picture parameter set identifiers are unique across the multiple component bitstreams; and

a fourth composition rule according to which sequence parameter set identifiers are unique across the multiple component bitstreams.

10. The computing system of claim 1 wherein the multiple marker data units are a type of network abstraction layer unit.

11. A method comprising:

receiving a multi-layer encoding (MLE) bitstream that includes multiple component bitstreams, wherein each of the multiple component bitstreams provides an alternative version of input video, the multiple component bitstreams including at least some parameter set identifiers having values set to avoid conflicting uses of the values between the multiple component bitstreams in the MLE bitstream, such that each of the values is unique across the multiple component bitstreams;

demultiplexing at least part of at least one of the multiple component bitstreams from the MLE bitstream, wherein the demultiplexing is based at least in part on multiple marker data units in the MLE bitstream, wherein each of the multiple marker data units includes a priority identifier and one or more quality layer identifiers, the priority identifier associating a video data unit with a given layer of one of the multiple component bitstreams; and

outputting the at least part of at least one of the multiple component bitstreams.

12. The method of claim 11 wherein at least one of the multiple component bitstreams is a scalable bitstream that includes a base layer and an enhancement layer, and wherein:

the base layer represents a first temporal resolution of the input video and the enhancement layer represents a second temporal resolution higher than the first temporal resolution; and

/or

the base layer represents a first signal to noise ratio (SNR) level of the input video and the enhancement layer represents a second SNR level higher than the first SNR level.

13. The method of claim 12, wherein the alternative versions of the input video are alternative simulcast versions of the input video, such that the multiple component bitstreams provide multiple spatial resolution options between the multiple component bitstreams.

14. The method of claim 11 wherein each of the multiple component bitstreams differs from the other bitstreams of the multiple component bitstreams in terms of one or more of:

number of layers of temporal, spatial and/or signal to noise ratio (SNR) scalability, if any, supported in that component bitstream;

format of decoding of the component bitstream; and

profile and/or level for decoding of the component bitstream.

15. The method of claim 11 wherein the demultiplexing follows multiple decomposition rules, the multiple decomposition rules including:

a first decomposition rule according to which a base layer for each of the multiple component bitstreams has a priority identifier and one or more zero-value quality layer identifiers;

a second decomposition rule according to which a given non-base layer of any one of the multiple component bitstreams has a priority identifier that, relative to the priority identifiers of the base layers, indicates one of the multiple component bitstreams for the given non-base layer;

a third decomposition rule according to which picture parameter set identifiers in slice data units identify corresponding picture parameter set data units; and

a fourth decomposition rule according to which sequence parameter set identifiers in the corresponding picture parameter sets identify corresponding sequence parameter set data units.

16. One or more computer-readable media storing a multi-layer encoding (MLE) bitstream that includes multiple component bitstreams, wherein the one or more computer-readable media are selected from the group consisting volatile memory, non-volatile memory, magnetic disk, CD-ROM, and DVD, each of the multiple component bitstreams providing an alternative version of input video, the multiple component bitstreams including at least some parameter set identifiers having values set to avoid conflicting uses of the values between the multiple component bitstreams in the MLE bitstream, such that each of the values is unique across the multiple component bitstreams, wherein the MLE bitstream includes multiple marker data units, wherein each of the multiple marker data units includes a priority identifier and one or more quality layer identifiers, the priority identifier associating a video data unit with a given layer of one of the multiple component bitstreams, and wherein the MLE bitstream is organized to facilitate processing by operations that include:

receiving the MLE bitstream;

demultiplexing at least part of at least one of the multiple component bitstreams from the MLE bitstream, wherein the demultiplexing is based at least in part on the multiple marker data units in the MLE bitstream; and

outputting the at least part of at least one of the multiple component bitstreams.

17. The one or more computer-readable media of claim 16

wherein at least one of the multiple component bitstreams is a scalable bitstream that includes a base layer and an at least one enhancement layer, and wherein:

the base layer represents a first temporal resolution of the input video and the enhancement layer represents a second temporal resolution higher than the first temporal resolution; and

/or

the base layer represents a first signal to noise ratio (SNR) level of the input video and the enhancement layer represents a second SNR level higher than the first SNR level.

18. The one or more computer-readable media of claim 17 wherein the alternative versions of the input video are alternative simulcast versions of the input video, such that the multiple component bitstreams provide multiple spatial resolution options between the multiple component bitstreams.

19. The one or more computer-readable media of claim 16 wherein each of the multiple component bitstreams differs from the other bitstreams of the multiple component bitstreams in terms of one or more of:

number of layers of temporal, spatial and/or signal to noise ratio (SNR) scalability, if any, supported in that component bitstream;

format of decoding of the component bitstream; and

profile and/or level for decoding of the component bitstream.

20. The one or more computer-readable media of claim 16 wherein the demultiplexing follows multiple decomposition rules, the multiple decomposition rules including:

a first decomposition rule according to which a base layer for each of the multiple component bitstreams has a priority identifier and one or more zero-value quality layer identifiers;

a second decomposition rule according to which a given non-base layer of any one of the multiple component bitstreams has a priority identifier that, relative to the priority identifiers of the base layers, indicates one of the multiple component bitstreams for the given non-base layer;

a third decomposition rule according to which picture parameter set identifiers in slice data units identify corresponding picture parameter set data units; and

a fourth decomposition rule according to which sequence parameter set identifiers in the corresponding picture parameter sets identify corresponding sequence parameter set data units.

21. One or more computer-readable media having stored thereon computer-executable instructions for causing a computing system, when programmed thereby, to perform operations, wherein the one or more computer-readable media are selected from the group consisting volatile memory, non-volatile memory, magnetic disk, CD-ROM, and DVD, the operations comprising:

receiving multiple component bitstreams;

with a bitstream multiplexer, multiplexing the multiple component bitstreams into a multi-layer encoding (MLE) bitstream, wherein each of the multiple component bitstreams provides an alternative version of input video, and wherein the multiplexing includes inserting multiple marker data units in the MLE bitstream, each of the multiple marker data units including a priority identifier and one or more quality layer identifiers, the priority identifier associating a video data unit with a given layer of one of the multiple component bitstreams, the multiple component bitstreams including at least some parameter set identifiers having values set to avoid conflicting uses of the values in the MLE bitstream, such that each of the values is unique across the multiple component bitstreams; and

outputting the MLE bitstream.

22. The one or more computer-readable media of claim 21, wherein the operations further comprise, with a video encoder:

encoding the input video to produce the multiple component bitstreams; and

during the encoding, setting the values of the at least some parameter set identifiers in the multiple component bitstreams to avoid conflicting uses of the values in the MLE bitstream.

23. The one or more computer-readable media of claim 21, wherein the operations further comprise, with the bitstream multiplexer:

rewriting the values of the at least some parameter set identifiers in the multiple component bitstreams to avoid conflicting uses of the values in the MLE bitstream.

24. The one or more computer-readable media of claim 23, wherein the operations further comprise, as part of the rewriting, for a given value of the values of the at least some parameter set identifiers in the multiple component bitstreams:

identifying a set of candidate values appropriate for the given value based on number of bits in the given value;

selecting a new value from the set of candidate values; and

replacing the given value with the new value.

25. The one or more computer-readable media of claim 21, wherein at least one of the multiple component bitstreams is a scalable bitstream that includes a base layer and an enhancement layer, and wherein:

the base layer represents a first temporal resolution of the input video and the enhancement layer represents a second temporal resolution higher than the first temporal resolution; and

/or

the base layer represents a first signal to noise ratio (SNR) level of the input video and the enhancement layer represents a second SNR level higher than the first SNR level.

26. The one or more computer-readable media of claim 25, wherein the alternative versions of the input video are alternative simulcast versions of the input video, such that the multiple component bitstreams provide multiple spatial resolution options between the multiple component bitstreams.

27. The one or more computer-readable media of claim 21, wherein the multiplexing enforces multiple composition rules, the multiple composition rules including:

a first composition rule according to which priority identifiers identify layers of the multiple component bitstreams, wherein the priority identifiers are included in the marker data units for at least some slice data units and are directly included as part of other slice data units;

a second composition rule according to which each of the multiple component bitstreams is assigned a non-overlapping range of the priority identifiers within which a base layer for the component bitstream has the lowest priority identifier in the range;

a third composition rule according to which picture parameter set identifiers are unique across the multiple component bitstreams; and

a fourth composition rule according to which sequence parameter set identifiers are unique across the multiple component bitstreams.

28. The one or more computer-readable media of claim 21, wherein the multiple marker data units are a type of network abstraction layer unit.

29. One or more computer-readable media having stored thereon computer-executable instructions for causing a computing system, when programmed thereby, to perform operations, wherein the one or more computer-readable media are selected from the group consisting volatile memory, non-volatile memory, magnetic disk, CD-ROM, and DVD, the operations comprising:

receiving a multi-layer encoding (MLE) bitstream that includes multiple component bitstreams, wherein each of the multiple component bitstreams provides an alternative version of input video, the multiple component bitstreams including at least some parameter set identifiers having values set to avoid conflicting uses of the values between the multiple component bitstreams in the MLE bitstream, such that each of the values is unique across the multiple component bitstreams;

demultiplexing at least part of at least one of the multiple component bitstreams from the MLE bitstream, wherein the demultiplexing is based at least in part on multiple marker data units in the MLE bitstream, wherein each of the multiple marker data units includes a priority identifier and one or more quality layer identifiers, the priority identifier associating a video data unit with a given layer of one of the multiple component bitstreams; and

outputting the at least part of at least one of the multiple component bitstreams.

30. The one or more computer-readable media of claim 29, wherein at least one of the multiple component bitstreams is a scalable bitstream that includes a base layer and an enhancement layer, and wherein:

the base layer represents a first temporal resolution of the input video and the enhancement layer represents a second temporal resolution higher than the first temporal resolution; and

/or

the base layer represents a first signal to noise ratio (SNR) level of the input video and the enhancement layer represents a second SNR level higher than the first SNR level.

31. The one or more computer-readable media of claim 30, wherein the alternative versions of the input video are alternative simulcast versions of the input video, such that the multiple component bitstreams provide multiple spatial resolution options between the multiple component bitstreams.

32. The one or more computer-readable media of claim 29, wherein the demultiplexing follows multiple decomposition rules, the multiple decomposition rules including:

a first decomposition rule according to which a base layer for each of the multiple component bitstreams has a priority identifier and one or more zero-value quality layer identifiers;

a second decomposition rule according to which a given non-base layer of any one of the multiple component bitstreams has a priority identifier that, relative to the priority identifiers of the base layers, indicates one of the multiple component bitstreams for the given non-base layer;

a third decomposition rule according to which picture parameter set identifiers in slice data units identify corresponding picture parameter set data units; and

a fourth decomposition rule according to which sequence parameter set identifiers in the corresponding picture parameter sets identify corresponding sequence parameter set data units.