Adaptive streaming transcoder synchronization

• US 9,414,100 B2
• Filed: 12/23/2014
• Issued: 08/09/2016
• Est. Priority Date: 03/31/2014
• Status: Active Grant

• Current Assignee: ARRIS Enterprises LLC (CommScope Holding Company, Inc.)
• Original Assignee: ARRIS Enterprises LLC (CommScope Holding Company, Inc.)
• Inventors: Van Veldhuisen, Brenda
• Primary Examiner(s): Flynn, Nathan
• Assistant Examiner(s): KIM, WILLIAM JW
• Application Number:
  US14/581,108
• Publication Number:
  US 20150281752A1
• Time in Patent Office: 595 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: H04N 19/68 involving the insertion of resynchronisation markers into the bitstream
• CPC Class Codes: H04N 21/231 Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/2335 involving reformatting operations of audio signals, e.g. by converting from one coding standard to another details of audio signal transcoding G10L19/173
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/234309 by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4 or from Quicktime to Realvideo conversion of standards in analog television systems H04N7/01
• CPC Class Codes: H04N 21/234381 by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping
• CPC Class Codes: H04N 21/23439 for generating different versions
• CPC Class Codes: H04N 21/23602 Multiplexing isochronously with the video sync, e.g. according to bit-parallel or bit-serial interface formats, as SDI
• CPC Class Codes: H04N 21/23605 Creation or processing of packetized elementary streams [PES]
• CPC Class Codes: H04N 21/23655 Statistical multiplexing, e.g. by controlling the encoder to alter its bitrate to optimize the bandwidth utilization
• CPC Class Codes: H04N 21/2368 Multiplexing of audio and video streams
• CPC Class Codes: H04N 21/242 Synchronization processes, e.g. processing of PCR [Program Clock References] arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18
• CPC Class Codes: H04N 21/64322 IP
• CPC Class Codes: H04N 21/647 Control signaling between network components and server or clients; Network processes for video distribution between server and clients, e.g. controlling the quality of the video stream, by dropping packets, protecting content from unauthorised alteration within the network, monitoring of network load, bridging between two different networks, e.g. between IP and wireless real-time session protocols H04L65/1101
• CPC Class Codes: H04N 21/845 Structuring of content, e.g. decomposing content into time segments
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 21/8547 involving timestamps for synchronizing content
• CPC Class Codes: H04N 5/067 Arrangements or circuits at the transmitter end

Abstract

Techniques for synchronizing streams in an adaptive streaming environment include cross-chassis and intra-chassis synchronization of video and/or audio. In an adaptive streaming system that streams content in segments or chunks at varying levels of quality corresponding to different coding rates, implementations enable seamless switching between output streams. For example, synchronizing multiple output streams for switching between coded adaptive bit rate streams across multiple chassis involve enforcing decisions of a master encoder for inverse telecine, frame rate conversion, and segment boundary to slave encoders associated with the same group of output stream representations.

Claims

1. An adaptive transcoder for synchronizing multiple stream representations in an output group, the transcoder comprising:

at least one input for receiving an input stream;

a switch coupling the at least one input to one or more transcoding modules;

each of the one or more transcoding modules hosting a plurality of transcoding processors, each transcoding processor comprising;

a decoder for receiving an input to the transcoding processor and decoding the stream;

a pre-processor for receiving the decoded stream from the decoder, detecting frame content, and outputting a stream with frames identified for encoding;

a plurality of encoders for outputting stream representations associated with the same input stream, wherein at least one of the plurality of encoders is designated as a master encoder based on an output frame rate and resolution of stream representations output from the plurality of encoders, the output group comprising of at least one of;

a plurality of stream representations generated from a single stream input to said transcoder, or a plurality of stream representations generated from corresponding input streams input to a plurality of transcoders including said transcoder,

wherein one or more of the remaining encoders associated with the output group are designated as slave encoders;

the master encoder determining where to fragment a received stream;

the master encoder identifying fragment boundary points for synchronizing stream representations in the output group with the master encoder output representation stream; and

the master encoder providing the identified fragment boundary points and corresponding presentation time stamps (PTS) to each slave encoder associated with the output group for generating a slave encoder output stream representation that is synchronized with the master encoder output stream representation.

2. The transcoder of claim 1, wherein in response to the fragment boundary point information received from the master encoder, the slave encoders fragment each slave encoder stream representation using the received fragment boundary points and the presentation time stamps.

3. The transcoder of claim 1, the master encoder further providing results of an inverse telecine determination, determined internal to the transcoding processor hosting the master encoder, to each slave encoder associated with the output group.

4. The transcoder of claim 3, wherein in response to results of the inverse telecine determination received from the master encoder, each slave encoder chooses the same repeat fields for performing inverse telecine at each respective slave encoder as chosen by the master encoder.

5. The transcoder of claim 3, wherein the master encoder providing results of an inverse telecine determination comprises providing frame statistics to the slave encoders for processing by each slave encoder of the same 24p image statistics.

6. The transcoder of claim 1, the master encoder further providing a selection of one or more frames for frame rate conversion, determined internal to the transcoding processor hosting the master encoder, to each slave encoder associated with the output group.

7. The transcoder of claim 6, wherein in response to the selection of the one or more frame rates for frame rate conversion by the master encoder, the slave encoders choose the same frames for frame rate conversion for each respective slave encoder output stream representation.

8. The transcoder of claim 6, the master encoder converting from 30 frames per second interlaced content (i.e., 30i) to 30 frames per second progressive output by extracting only a top field from the 30i content.

9. The transcoder of claim 8, wherein the master encoder provides the manner of frame rate conversion to the slave encoders for extraction by the slave encoders of a same image during conversion from 30i to 30p.

10. The transcoder of claim 6, the master encoder converting from 60 frames per second interlaced content (i.e., 60p) to 30 frames per second progressive output by using a frame in which a difference in PTS from a previous frame'"'"'s PTS is 1501 ticks of a clock.

11. The transcoder of claim 6, the master encoder converting from 25i (i.e., 50i) to 25p by extracting only a top field from the 25i content.

12. The transcoder of claim 6, the master encoder converting from 50p to 25p by initiating a selection of one 50p frame and then selecting every other subsequent 50p frame for the 25p output.

13. The transcoder of claim 12, wherein the master encoder signals to the slave encoders which frames were selected for use by the slave encoders or for the slave encoders to skip a 50p frame to select the corresponding frame selected by the master encoder.

14. The transcoder of claim 6, wherein the master encoder determines a frame in the master encoder output representation stream in which a scene change is located that will be skipped when converting the output frame rate of the stream from a first output frame rate to a second output frame rate, said second output frame rate causing an increase in a number of frames in the output stream representation to achieve the first output frame rate, sets a PTS at a fragment boundary on the output stream representation at the first output frame rate to match a PTS at a fragment boundary of the output stream representation at the second output frame rate, and signals a single frame instead of the increased number of frames on a previous frame.

15. The transcoder of claim 14, wherein the second output rate causes the output stream representation to be frame doubled or frame tripled to achieve the first output frame rate, wherein the single frame on the previous frame is signaled instead of signaling frame doubling or frame tripling, respectively.

16. The transcoder of claim 14, wherein the first output rate is 30p and the second output rate is 60p.

17. The transcoder of claim 6, wherein the master encoder determines a frame in a stream in which a scene change is located that is skipped when converting the output frame rate of the stream from a first output rate to a second output rate, and moves a PTS on the stream at the second output frame rate to match an instantaneous decode refresh (IDR) on the stream at the first output frame rate.

18. The transcoder of claim 17, wherein the master encoder moves a fragment as far away from a scene change as allowed by a fragment range.

19. The transcoder of claim 1, wherein the transcoder is a first transcoder in communication with a second transcoder, the second transcoder comprising only slave encoders that receive information from the master encoder in the first transcoder and synchronize the second transcoder'"'"'s stream representations to the first encoder'"'"'s master encoder stream representation.

20. The transcoder of claim 1, wherein the master encoder collects frame statistics to determine where to locate fragment boundaries in the stream, the frame statistics including at least one of a scene change, film mode, fade, or flash.

21. The transcoder of claim 1, wherein enforcing decisions of the master encoder at each slave encoder maintains identical fragment boundaries in each of the stream representations output in the output group.

22. The transcoder of claim 1, wherein each fragment boundary point identified by the master encoder is represented by an instantaneous decode refresh (IDR) frame and the IDR frame'"'"'s corresponding time stamp.

23. The transcoder of claim 22, wherein each slave encoder receives fragment boundary point information from the master encoder, marks the same fragment boundary points, places an instantaneous decode refresh (IDR) on a first frame after each fragment boundary point, and matches a PTS in each first frame in the output stream to the PTS in the corresponding frame from the master output channel.

24. The transcoder of claim 22, wherein the master encoder determines an optimal video quality location to place each instantaneous decode refresh (IDR) frame for a same placement by each slave encoder.

25. The transcoder of claim 22, wherein a fragment boundary point is moved to a location of a scene change, at an end of a fade, or at an end of a pan, thereby separating an instantaneous decode refresh (IDR) frame from an intracoded frame.

26. The transcoder of claim 22, wherein an instantaneous decode refresh (IDR) fragment is inserted at a MaxFragmentSize location, where MaxFragmentSize is a maximum number of frames since the previous fragment allowed that meets a defined maximum fragment interval.

27. The transcoder of claim 22, wherein the master encoder determines whether at least one of a scene change, an end of a pan, or an end of a fade is upcoming within a frame range, and designates a frame corresponding to the at least one of a scene change, the end of a pan, or the end of a fade as a new instantaneous decode refresh (IDR) fragment boundary point.

28. The transcoder of claim 27, wherein a boundary is not at a MaxFragmentSize location, where MaxFragmentSize is a maximum number of frames since the previous fragment allowed that meets a defined maximum fragment interval.

29. The transcoder of claim 1, wherein segments are set at every n fragment boundary point.

30. The transcoder of claim 1, wherein the transcoder determines for each fragment identified whether the fragment is a complete segment.

31. The transcoder of claim 30, wherein if a segment picture count (SegmentPicCnt) added to a maximum fragment size (MaxFragment Size) is greater than a segment size (SegmentSize), then the fragment is a segment, else if the segment picture count (SegmentPicCnt) is a number of frames since a last segment boundary, a MaxFragmentSize is set as a longest fragment size defined by a user and a SegmentSize is set as a longest SegmentSize defined by the user.

32. The transcoder of claim 31, wherein when a group of pictures (GOP) selection code sets a current frame as an instantaneous decode refresh (IDR), the current frame is reset as a fragment boundary so a next frame is not forced as the instantaneous decode refresh (IDR).

33. The transcoder of claim 1, wherein the input to each of the plurality of encoders has an input resolution and frame rate of the single input stream input to the transcoder, and the stream representations output from the plurality of encoders in the output group vary from each other by both resolution and frame rate but maintain identical segment boundaries between the stream representations.

34. The transcoder of claim 1, wherein the master encoder is selected as the encoder in the plurality of encoders that produces a stream representation with a highest resolution in the output group.

35. The transcoder of claim 1, wherein the slave encoders structure their respective output stream representations based on a group of pictures (GOP) structure of the master encoder stream representation provided to each of the slave encoders.

36. The transcoder of claim 1, wherein no active synchronization to the master encoder is required for frame rate conversion by the slave encoders.

37. The transcoder of claim 1, wherein more than one frame rate conversion is actively synchronized between the master encoder and slave encoders by:

each slave encoder compares a PTS value of a respective slave channel with a PTS of a master output channel of the master encoder; and

if a PTS mismatch is detected by a slave encoder, the slave channel drops an extra frame rate in a frame rate conversion to bring the slave channel back in synch with the master output channel,

wherein the extra frames are dropped multiple times as needed to bring the slave channel in synch with the master output channel.

38. The transcoder of claim 1, wherein the output group includes stream representations that span at least one of:

multiple transcoding processors, multiple transcoders, or multiple chassis.

39. The transcoder of claim 1, wherein a segment comprises a single fragment and a fragment boundary point is the segment boundary point.

40. A master encoder designated from a plurality of encoders for synchronizing stream representations in an output group, the master encoder comprising:

at least one input for receiving a video stream pre-processed with frames within the input stream identified;

a processor configured to perform;

determining at which frames within the input stream to segment the stream;

identifying segment boundary points for providing to slave encoders for synchronizing the plurality of output stream representations in the output group with the master encoder output representation stream,

wherein the output group comprises at least one of a plurality of stream representations generated from a single stream input to a transcoder, or a plurality of stream representations generated from corresponding input streams input to a plurality of transcoders including said transcoder, and

wherein one or more of the remaining encoders associated with the output group are designated as slave encoders, wherein the master encoder is designated based on an output frame rate and resolution of stream representations output from the plurality of encoders,

providing results of an inverse telecine determination and a selection of one or more frames for frame rate conversion, both determined internal to a transcoding processor, to each slave encoder associated with the output group;

providing the identified segment boundary points and corresponding presentation time stamps (PTS) to each slave encoder associated with the output group to enable generation of an output representation that synchronizes with the master encoder output stream representation; and

wherein the information is provided to the slave encoders within the transcoding processor or within the plurality of encoders associated with the output group for synchronizing the outputs of the slave encoder to the outputs of the master encoder.

41. An adaptive transcoder for synchronizing multiple stream representations in an output group, the transcoder comprising:

at least one input for receiving an input stream;

a switch coupling the at least one input to one or more transcoding modules;

each of the one or more transcoding modules hosting a plurality of transcoding processors, each transcoding processor comprising;

a decoder for receiving an input to the transcoding processor and decoding the stream;

a pre-processor for receiving the decoded stream from the decoder, detecting frame content, and outputting a stream with frames identified for encoding;

a plurality of encoders for outputting stream representations associated with the same input stream, wherein at least one of the plurality of encoders is designated as a master encoder, the output group comprising of at least one of;

a plurality of stream representations generated from a single stream input to said transcoder, or a plurality of stream representations generated from corresponding input streams input to a plurality of transcoders including said transcoder,

wherein one or more of the remaining encoders associated with the output group are designated as slave encoders;

the master encoder determining where to fragment a received stream;

the master encoder identifying fragment boundary points for synchronizing stream representations in the output group with the master encoder output representation stream; and

the master encoder providing the identified fragment boundary points and corresponding presentation time stamps (PTS) to each slave encoder associated with the output group for generating a slave encoder output stream representation that is synchronized with the master encoder output stream representation,

wherein the transcoder determines for each fragment identified whether the fragment is a complete segment, and if a segment picture count (SegmentPicCnt) added to a maximum fragment size (MaxFragment Size) is greater than a segment size (SegmentSize), then the fragment is a segment, else if the segment picture count (SegmentPicCnt) is a number of frames since a last segment boundary, a MaxFragmentSize is set as a longest fragment size defined by a user and a SegmentSize is set as a longest SegmentSize defined by the user.

42. An adaptive transcoder for synchronizing multiple stream representations in an output group, the transcoder comprising:

at least one input for receiving an input stream;

a switch coupling the at least one input to one or more transcoding modules;

each of the one or more transcoding modules hosting a plurality of transcoding processors, each transcoding processor comprising;

a decoder for receiving an input to the transcoding processor and decoding the stream;

a pre-processor for receiving the decoded stream from the decoder, detecting frame content, and outputting a stream with frames identified for encoding;

a plurality of encoders for outputting stream representations associated with the same input stream, wherein at least one of the plurality of encoders is designated as a master encoder, the output group comprising of at least one of;

a plurality of stream representations generated from a single stream input to said transcoder, or a plurality of stream representations generated from corresponding input streams input to a plurality of transcoders including said transcoder,

wherein one or more of the remaining encoders associated with the output group are designated as slave encoders;

the master encoder determining where to fragment a received stream;

the master encoder identifying fragment boundary points for synchronizing stream representations in the output group with the master encoder output representation stream, wherein when a group of pictures (GOP) selection code sets a current frame as an instantaneous decode refresh (IDR), the current frame is reset as a fragment boundary so a next frame is not forced as the instantaneous decode refresh (IDR); and

the master encoder providing the identified fragment boundary points and corresponding presentation time stamps (PTS) to each slave encoder associated with the output group for generating a slave encoder output stream representation that is synchronized with the master encoder output stream representation.

43. An adaptive transcoder for synchronizing multiple stream representations in an output group, the transcoder comprising:

at least one input for receiving an input stream;

a switch coupling the at least one input to one or more transcoding modules;

each of the one or more transcoding modules hosting a plurality of transcoding processors, each transcoding processor comprising;

a decoder for receiving an input to the transcoding processor and decoding the stream;

a pre-processor for receiving the decoded stream from the decoder, detecting frame content, and outputting a stream with frames identified for encoding;

a plurality of encoders for outputting stream representations associated with the same input stream, wherein at least one of the plurality of encoders is designated as a master encoder based on the encoder that produces a stream representation with a highest resolution in the output group, the output group comprising of at least one of;

a plurality of stream representations generated from a single stream input to said transcoder, or a plurality of stream representations generated from corresponding input streams input to a plurality of transcoders including said transcoder,

wherein one or more of the remaining encoders associated with the output group are designated as slave encoders;

the master encoder determining where to fragment a received stream;

the master encoder identifying fragment boundary points for synchronizing stream representations in the output group with the master encoder output representation stream; and

the master encoder providing the identified fragment boundary points and corresponding presentation time stamps (PTS) to each slave encoder associated with the output group for generating a slave encoder output stream representation that is synchronized with the master encoder output stream representation.