Content delivery edge storage optimized media delivery to adaptive bitrate (ABR) streaming clients

• US 10,390,071 B2
• Filed: 04/13/2017
• Issued: 08/20/2019
• Est. Priority Date: 04/16/2016
• Status: Active Grant

• Current Assignee: Ittiam Systems Pte Ltd
• Original Assignee: Ittiam Systems Pte Ltd
• Inventors: Sethuraman, Sriram, Muthukrishnan, Murali Babu
• Primary Examiner(s): Huynh, An Son P
• Application Number:
  US15/486,327
• Publication Number:
  US 20170302981A1
• Time in Patent Office: 859 Days
• Field of Search: None
• CPC Class Codes: H04N 19/146 Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/40 using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 21/234354 by altering signal-to-noise ratio parameters, e.g. requantization
• CPC Class Codes: H04N 21/235 Processing of additional data, e.g. scrambling of additional data or processing content descriptors
• CPC Class Codes: H04N 21/236 Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream multiplexing of data packets for data networks, e.g. RTP/UDP H04L65/00
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/6587 Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments

Abstract

A system includes an interface unit to receive a request, from a media client, to obtain a segment of media content at an ABR representation, and a transcoding unit. The transcoding unit includes a scalable decoder, hint stream decoder, and re-encoder. The scalable decoder receives and decodes a set of two or more layers of a scalable coded representation corresponding to the requested segment of media content at the requested ABR representation to provide a set of media frames and scalable stream information for the segment. The hint stream decoder receives and decodes a hint stream corresponding to the requested segment of media content at the requested ABR representation and the scalable stream information to provide transcoding hints for the segment. The re-encoder re-encodes the set of media frames using the transcoding hints to provide an encoded bit stream corresponding to the requested segment of media content.

Claims

1. A system comprising:

a media content delivery unit comprising;

an interface unit to receive a request, from a media client, to obtain a segment of media content at an adaptive bitrate (ABR) representation of a plurality of ABR representations; and

a transcoding unit comprising;

a scalable decoder that receives a set of two or more layers of a scalable coded representation corresponding to the requested segment of media content at the requested ABR representation and decodes the set of two or more layers to provide a set of media frames and scalable stream information for the segment, wherein the scalable stream information comprises coding tree block level information for a plurality of coding tree blocks in a scalable stream for the segment, wherein the coding tree block level information comprises partitioning information for prediction, a specific prediction mode within a type of prediction for each prediction partition, and motion vectors with respect to specific reference pictures for each predicted partition;

a hint stream decoder coupled to the scalable decoder, wherein the hint stream decoder receives a hint stream corresponding to the requested segment of media content at the requested ABR representation as a first input and the scalable stream information for the segment as a second input, and uses at least one of;

a) the partitioning information for prediction, b) the specific prediction mode within the type of prediction for each prediction partition, and c) the motion vectors with respect to specific reference pictures for each predicted partition as predictors in a hint stream coding representation to decode the hint stream for providing transcoding hints for the segment as an output; and

a re-encoder coupled to the scalable decoder and the hint stream decoder to receive the set of media frames and the transcoding hints, respectively, as inputs, and re-encode the set of media frames using the transcoding hints to provide an encoded bit stream corresponding to the requested segment of media content at the requested ABR representation for packaging and eventual delivery to the media client.

2. The system of claim 1, wherein the transcoding hints for the segment comprises information to substantially reduce the complexity of re-encoding the set of media frames within a bit-rate indicated for the requested segment of media content at the requested ABR representation.

3. The system of claim 1, wherein the media content delivery unit comprises:

a QA hint based selection unit to;

receive a quality assurance (QA) hint corresponding to the requested segment of media content at the requested ABR representation of the segment; and

conditionally schedule the transcoding unit to generate the encoded bit stream of the segment for packaging and eventual delivery to the media client when the QA hint indicates that transcoding of the segment results in a quality greater than or equal to a reference quality.

4. The system of claim 3, wherein the QA hint based selection unit conditionally obtains a pre-packaged or unpackaged simulcast ABR stream corresponding to the requested segment of media content at the requested ABR representation for further packaging and eventual delivery to the media client, when the QA hint indicates that transcoding of the segment results in a quality less than the reference quality.

5. The system of claim 4, further comprising:

a media client request agnostic streams preparation unit communicatively coupled with the media content delivery unit, that generates a set of layers of the scalable coded representation, hint streams, and QA hints using a subset of which any one of the plurality of ABR representations can be generated.

6. The system of claim 5, wherein the media client request agnostic streams preparation unit comprises:

a high-complexity ABR encoder to generate, from a mezzanine input stream, a set of simulcast ABR streams, reconstructed pixel values for the set of simulcast ABR streams, and ABR stream information for the set of simulcast ABR streams according to an ABR recipe, wherein the set of simulcast ABR streams is generated at a first computational complexity level.

7. The system of claim 6, wherein the set of layers of the scalable coded representation, the hint streams, and the QA hint streams are generated such that a total storage requirements or first mile bandwidth requirements of the set of layers of the scalable coded representation, the hint streams, and the QA hint streams are substantially less than the storage requirements or first mile bandwidth requirements for the simulcast ABR streams.

8. The system of claim 6, wherein the ABR stream information comprises picture level information and coding tree block level information, wherein the picture level information comprises information selected from a group consisting of a picture type, reference pictures, scaling matrices for intra/inter and luma/chroma, and deblocking filter offset parameters, and wherein the coding tree block level information comprises information selected from a group consisting of the partitioning information for prediction, a type of prediction information for each prediction partition, a specific prediction mode within the type of prediction for each prediction partition, motion vectors with respect to specific reference pictures for each predicted partition, partitioning information for transform coding, and scalar or vector quantizer information at the transform partition level.

9. The system of claim 6, wherein the ABR recipe comprises at least one parameter selected from a group consisting of a coding format, a bit-rate switchable segment duration, an intra picture period, a number of temporal layers, a distance between two consecutive temporal base layer pictures, a spatial resolution, a frame-rate, a coded picture buffer size, a maximum instantaneous bit-rate, a bit-rate, and a ratio of bit-rates with respect to a bit-rate of a reference stream for each simulcast ABR stream in the set of simulcast ABR streams.

10. The system of claim 6, wherein the media client request agnostic streams preparation unit comprises:

an ABR stream information analyzer to determine one or more groups of bit-rates that can be encoded with dependency through scalable encoding based on the ABR stream information from the high-complexity ABR encoder and the ABR recipe, wherein a lowest bit-rate in each group of bit-rates is a base bit-rate for the scalable coded representation.

11. The system of claim 10, wherein the media client request agnostic streams preparation unit comprises:

a scalable encoder to;

generate a set of layers of a scalable coded representation for each group of bit-rates by performing spatial scalable and SNR scalable coding of the mezzanine input at different resolutions or the reconstructed pixel values of the set of simulcast ABR streams corresponding to each group of bit-rates, wherein the scalable decoder receives and decodes the set of two or more layers of the segment from the set of layers corresponding to each group of bit-rates; and

export reconstructed pixel values and scalable stream information corresponding to the set of layers of the scalable coded representation for each group of bit-rates.

12. The system of claim 11, wherein the set of layers of the scalable coded representation for each group of bit-rates is stored in a centralized storage unit for accessing by one or more transcoding units.

13. The system of claim 11, wherein the set of layers comprises at least one enhancement layer of the scalable coded representation and a base layer of the scalable coded representation, and wherein the base layer of the scalable coded representation for a group of bit-rates is a simulcast ABR stream within the set of simulcast ABR streams that corresponds to the lowest bit-rate within the group of bit-rates.

14. The system of claim 11, wherein the media client request agnostic streams preparation unit comprises:

reconstructed pixel values and hint stream (RPVHS) generation unit to generate reconstructed pixel values for each ABR representation and hint stream for each ABR representation based on the reconstructed pixel values and the scalable stream information from the scalable encoder, and the ABR stream information from the high-complexity ABR encoder.

15. The system of claim 14, wherein the RPVHS generation unit comprises:

a medium-complexity ABR encoder to generate a set of ABR streams at a second computational complexity level, simulcast stream information for the set of ABR streams, and reconstructed pixel values for each ABR representation using the reconstructed pixel values and the scalable stream information of the set of layers from the scalable encoder and the ABR stream information from the high-complexity ABR encoder, wherein the second computational complexity level is significantly less than the first computational complexity level;

a hint generator to generate transcoding hints corresponding to each ABR representation based on the simulcast stream information received from the medium-complexity ABR encoder; and

a hint encoder to encode the transcoding hints corresponding to each ABR representation to provide the hint stream corresponding to each ABR representation, wherein the hint encoder uses the scalable stream information from the scalable encoder to encode the transcoding hints corresponding to each ABR representation.

16. The system of claim 14, wherein the RPVHS generation unit comprises:

a hint generator to generate transcoding hints corresponding to each ABR representation based on the ABR stream information received from the high-complexity ABR encoder;

a JIT transcoder simulator instance to receive, for each ABR representation, the corresponding generated transcoding hints, encode the reconstructed pixels values of the corresponding set of layers from the scalable encoder using the generated transcoding hints, and generate the reconstructed pixel values; and

a hint encoder to encode the transcoding hints corresponding to each ABR representation to provide the hint stream corresponding to each ABR representation, wherein the hint encoder uses the scalable stream information from the scalable encoder to encode the transcoding hints corresponding to each ABR representation.

17. The system of claim 14, wherein the media client request agnostic streams preparation unit comprises:

a segment-level QA based hint generation unit to generate QA hints for a plurality of segments of the media content at each ABR representation by comparing the reconstructed pixel values from the high-complexity ABR encoder and the reconstructed pixel values from the RPVHS generation unit.

18. The system of claim 1, wherein the ABR representation for providing the segment of the media content comprises one or more resolutions, one or more different data rates and/or one or more different formats.

19. A method for providing adaptive bit-rate (ABR) streaming to a media client, comprising:

receiving, via an interface unit of a media content delivery unit, a request, from the media client, to obtain a segment of media content at an adaptive bitrate (ABR) representation of a plurality of ABR representations;

receiving, by a scalable decoder of a transcoding unit, a set of two or more layers of a scalable coded representation corresponding to the requested segment of media content at the requested ABR representation and decoding the set of two or more layers to provide a set of media frames and scalable stream information for the segment, wherein the scalable stream information comprises coding tree block level information for a plurality of coding tree blocks in a scalable stream for the segment, wherein the coding tree block level information comprises partitioning information for prediction, a specific prediction mode within a type of prediction for each prediction partition, and motion vectors with respect to specific reference pictures for each predicted partition;

receiving, by a hint stream decoder coupled to the scalable decoder of the transcoding unit, a hint stream corresponding to the requested segment of media content at the requested ABR representation as a first input and the scalable stream information for the segment as a second input, and using at least one of;

a) the partitioning information for prediction, b) the specific prediction mode within the type of prediction for each prediction partition, and c) the motion vectors with respect to specific reference pictures for each predicted partition as predictors in a hint stream coding representation to decode for providing transcoding hints for the segment as an output; and

receiving, by a re-encoder coupled to the scalable decoder and the hint stream decoder, the set of media frames and the transcoding hints as inputs, and re-encoding the set of media frames using the transcoding hints to provide an encoded bit stream, corresponding to the requested segment of media content at the requested ABR representation by the transcoding unit for packaging and eventual delivery to the media client.

20. The method of claim 19, wherein the transcoding hints for the segment comprises information to substantially reduce the complexity of re-encoding the set of media frames within a bit-rate indicated for the requested segment of media content at the requested ABR representation.

21. The method of claim 19, further comprising:

receiving a quality assurance (QA) hint corresponding to the requested segment of media content at the requested ABR representation of the segment; and

conditionally scheduling the transcoding unit to generate the encoded bit stream of the segment for packaging and eventual delivery to the media client when the QA hint indicates that transcoding of the segment results in a quality greater than or equal to a reference quality.

22. The method of claim 21, further comprising:

when the QA hint indicates that transcoding of the segment results in the quality less than the reference quality, conditionally obtaining a pre-packaged or unpackaged simulcast ABR stream corresponding to the requested segment of media content at the requested ABR representation for further packaging and eventual delivery to the media client.

23. The method of claim 22, wherein the set of simulcast ABR streams, reconstructed pixel values for the set of simulcast ABR streams, and ABR stream information for a set of simulcast ABR streams are generated, from a mezzanine input stream, by a high-complexity ABR encoder according to an ABR recipe, wherein the set of simulcast ABR streams is generated at a first computational complexity level.

24. The method of claim 23, comprising:

determining one or more groups of bit-rates that can be encoded with dependency through scalable encoding by an ABR stream information analyzer based on the ABR stream information from the high-complexity ABR encoder and the ABR recipe, wherein a lowest bit-rate in each group of bit-rates is a base bit-rate for the scalable coded representation.

25. The method of claim 24, further comprising:

generating a set of layers of a scalable coded representation for each group of bit-rates based on the mezzanine input at different resolutions or the reconstructed pixel values of the simulcast ABR streams corresponding to each group of bit-rates by a scalable encoder, wherein the set of two or more layers of the scalable coded representation corresponding to the requested segment of media content at the requested ABR representation of the segment is received from the set of layers corresponding to each group of bit-rates; and

exporting reconstructed pixel values and scalable stream information corresponding to the set of layers of the scalable coded representation for each group of bit-rates by the scalable encoder.

26. The method of claim 25, further comprising:

storing the set of layers corresponding to each group of bit-rates in a centralized storage unit for accessing by one or more transcoding units.

27. The method of claim 25, wherein the set of layers comprises at least one enhancement layer of the scalable coded representation and a base layer of the scalable coded representation, and wherein the base layer of the scalable coded representation for a group of bit-rates is a simulcast ABR stream within the set of simulcast ABR streams that corresponds to the lowest bit-rate within the group of bit-rates.

28. The method of claim 25, further comprising:

generating reconstructed pixel values for each ABR representation and hint stream for each ABR representation by a reconstructed pixel values and hint stream (RPVHS) generation unit based on the reconstructed pixel values and the scalable stream information from the scalable encoder, and the ABR stream information from the high-complexity ABR encoder.

29. The method of claim 28, wherein generating the reconstructed pixel values and the hint stream for each ABR representation comprises:

generating a set of ABR streams at a second computational complexity level, simulcast stream information for the set of ABR streams, and reconstructed pixel values for each ABR representation by a medium-complexity ABR encoder using the reconstructed pixel values and the scalable stream information of the set of layers from the scalable encoder and the ABR stream information from the high-complexity ABR encoder, wherein the second computational complexity level is significantly less than the first computational complexity level; and

generating transcoding hints corresponding to each ABR representation by a hint generator based on the simulcast stream information received from the medium-complexity ABR encoder; and

encoding the transcoding hints corresponding to each ABR representation to provide the hint stream corresponding to each ABR representation by a hint encoder, wherein the transcoding hints corresponding to each ABR representation are encoded using the scalable stream information from the scalable encoder.

30. The method of claim 28, wherein generating the reconstructed pixel values and the hint stream for each ABR representation comprises:

generating transcoding hints corresponding to each ABR representation by a hint generator based on the ABR stream information received from the high-complexity ABR encoder;

receiving, for each ABR representation, the corresponding generated transcoding hints, encode the reconstructed pixels values of the corresponding set of layers from the scalable encoder using the generated transcoding hints, and generate the reconstructed pixel values by a JIT transcoder simulator instance; and

encoding the transcoding hints corresponding to each ABR representation to provide the hint stream corresponding to each ABR representation by a hint encoder, wherein the hint encoder uses the scalable stream information from the scalable encoder to encode the transcoding hints corresponding to each ABR representation.

31. The method of claim 28, further comprising:

generating QA hints for a plurality of segments of the media content at each ABR representation, using a segment-level QA based hint generation unit, by comparing the reconstructed pixel values from the high-complexity ABR encoder and the reconstructed pixel values from the RPVHS generation unit.

32. A non-transitory computer-readable storage medium having instructions that, when executed by a computing device, cause the computing device to:

receive, via an interface unit of a media content delivery unit, a request, from a media client, to obtain a segment of media content at an adaptive bitrate (ABR) representation of a plurality of ABR representations;

receive, by a scalable decoder of a transcoding unit, a set of two or more layers of a scalable coded representation corresponding to the requested segment of media content at the requested ABR representation and decode the set of two or more layers to provide a set of media frames and scalable stream information for the segment, wherein the scalable stream information comprises coding tree block level information for a plurality of coding tree blocks in a scalable stream for the segment, wherein the coding tree block level information comprises partitioning information for prediction, a specific prediction mode within a type of prediction for each prediction partition, and motion vectors with respect to specific reference pictures for each predicted partition;

receive, by a hint stream decoder coupled to the scalable decoder of the transcoding unit, a hint stream corresponding to the requested segment of media content at the requested ABR representation as a first input and the scalable stream information for the segment as a second input, and use at least one of;

a) the partitioning information for prediction, b) the specific prediction mode within the type of prediction for each prediction partition, and c) the motion vectors with respect to specific reference pictures for each predicted partition as predictors in a hint stream coding representation to decode the hint stream for providing transcoding hints for the segment as an output; and

receive, by a re-encoder coupled to the scalable decoder and the hint stream decoder, the set of media frames and the transcoding hints as inputs, and re-encode the set of media frames using the transcoding hints to provide an encoded bit stream, corresponding to the requested segment of media content at the requested ABR representation by the transcoding unit for packaging and eventual delivery to the media client.

33. The non-transitory computer-readable storage medium of claim 32, wherein the transcoding hints for the segment comprises information to substantially reduce the complexity of re-encoding the set of media frames within a bit-rate indicated for the requested segment of media content at the requested ABR representation.

34. The non-transitory computer-readable storage medium of claim 32, comprising instructions to:

receive a quality assurance (QA) hint corresponding to the requested segment of media content at the requested ABR representation of the segment; and

conditionally schedule the transcoding unit to generate the encoded bit stream of the segment for packaging and eventual delivery to the media client when the QA hint indicates that transcoding of the segment results in a quality greater than or equal to a reference quality.

35. The non-transitory computer-readable storage medium of claim 34, comprising instructions to:

when the QA hint indicates that transcoding of the segment results in the quality less than the reference quality, conditionally obtaining a pre-packaged or unpackaged simulcast ABR stream corresponding to the requested segment of media content at the requested ABR representation for further packaging and eventual delivery to the media client.

36. The non-transitory computer-readable storage medium of claim 35, wherein a simulcast ABR stream and reconstructed pixel values of the simulcast ABR stream are generated, from a mezzanine input stream, by a high-complexity ABR encoder according to an ABR recipe.

37. The non-transitory computer-readable storage medium of claim 36, wherein the set of layers of a scalable coded representation is generated by a scalable encoder based on the mezzanine input at different resolutions or the reconstructed pixel values of the simulcast ABR stream.

38. The non-transitory computer-readable storage medium of claim 37, wherein the hint stream and reconstructed pixel values corresponding to the requested segment of media content at the requested ABR representation are generated by a reconstructed pixel values and hint stream (RPVHS) generation unit based on reconstructed pixel values and scalable stream information from the scalable encoder, and ABR stream information from the high-complexity ABR encoder.

39. The non-transitory computer-readable storage medium of claim 38, wherein the QA hint for the segment of the media content is generated by a segment-level QA based hint generation unit based on comparing the reconstructed pixel values from the high-complexity ABR encoder and the reconstructed pixel values from the RPVHS generation unit.