Seamless splicing of encoded MPEG video and audio

• US 6,678,332 B1
• Filed: 03/31/2000
• Issued: 01/13/2004
• Est. Priority Date: 01/04/2000
• Status: Expired due to Term

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Bixby, Peter, Forecast, John, Rochette, Jean Louis, Faibish, Sorin, Duso, Wayne W., Gardere, Daniel, Keller, Sebastian, Noury, Michel, Oguz, Seyfullah H.
• Primary Examiner(s): Britton, Howard
• Assistant Examiner(s): WONG, ALLEN C
• Application Number:
  US09/540,347
• Time in Patent Office: 1,383 Days
• Field of Search: 375/240.26, 375/240.25, 348/705, 370/487, 725/32
• US Class Current: 375/240.26
• CPC Class Codes: H04N 21/23424 involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement
• CPC Class Codes: H04N 21/23608 Remultiplexing multiplex streams, e.g. involving modifying time stamps or remapping the packet identifiers
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/44016 involving splicing one content stream with another content stream, e.g. for substituting a video clip

Abstract

Predictive analysis is performed upon encoded digital motion video (such as an MPEG Transport Stream) to facilitate real-time splicing. The predictive analysis includes estimation of upper and lower bounds of the data levels in a decoder'"'"'s video and audio buffers for splicing in such a way as to prevent buffer overflow and underflow. This enables buffer management techniques including padding or stuffing, micro-restamping, freeze or repeat of frames, skip or drop of frames, alignment of audio with video. The predictive analysis also includes analysis of the structure of the encoded audio including audio access units (AAUs) and compression windows (AFs), prediction in the compressed domain of initial conditions of the decoder buffer levels for every single Elementary Stream (ES) component of a Transport Stream (TS), and identification of valid splicing In Points and Out Points based on the predicted buffer levels without any special encoder. This permits splicing of different compressed audio types without consideration of the details of the compression mechanism. The predictive analysis may also include recommendations or marking. The analysis is performed on ingest of the data as a pre-processing in preparation for splicing of a new file on early queuing in a data storage system before streaming.

Claims

1. A method of seamless splicing of a first transport stream to a second transport stream to produce a spliced transport stream, the first transport stream including video access units encoding video presentation units representing video frames and audio access units encoding audio presentation units representing segments of a first audio signal, the second transport stream including video access units encoding video presentation units representing video frames and audio access units encoding audio presentation units representing segments of a second audio signal, the first transport stream having a last video frame to be included in the spliced transport stream, and the second transport stream having a first video frame to be included in the spliced transport stream, said method comprising:

(a) finding, in the first transport stream, an audio access unit that is best aligned with the last video frame from the first transport stream to be included in the spliced transport stream, and removing audio access units from the first transport stream that are subsequent to the audio access unit that is best aligned with the last video frame from the first transport stream;

(b) finding, in the second transport stream, an audio access unit that is best aligned with the first video frame from the second transport stream to be included in the spliced transport stream; and

removing audio access units from the second transport stream that are prior to the audio access unit that is best aligned with the first video frame from the second transport stream; and

(c) concatenating a portion of the first transport stream up to and including the last video frame to a portion of the second transport stream including and subsequent to the first video frame.

2. The method as claimed in claim 1, which includes finding, in the first transport stream, the audio access unit that is best aligned with the last video frame from the first transport stream to be included in the spliced transport stream by computing a difference between presentation time at the end of an audio presentation unit encoded in the first transport stream and a presentation time at the end of the last video frame from the first transport stream to be included in the spliced transport stream, and finding that the difference has a magnitude less than half of a duration of the audio presentation unit, wherein the audio access unit that is best aligned with the last video frame from the first transport stream to be included in the spliced transport stream encodes the audio presentation unit having an ending presentation time that is best aligned with the ending presentation time of the last video frame from the first transport stream to be included in the spliced transport stream.

3. The method as claimed in claim 1, which includes finding, in the second transport stream, the audio access unit that is best aligned with the first video frame from the second transport stream to be included in the spliced transport stream by computing a difference between presentation time at the beginning of an audio presentation unit encoded in the second transport stream and a presentation time at the beginning of the first video frame from the second transport stream to be included in the spliced transport stream, and finding that the difference has a magnitude less that a half of a duration of the audio presentation unit, wherein the audio access unit that is best aligned with the first video frame from the second transport stream to be included in the spliced transport stream encodes the audio presentation unit having a beginning presentation time that is best aligned with the beginning presentation time of the first video frame from the second transport stream to be included in the spliced transport stream.

4. The method as claimed in claim 1, which includes deciding that decoding of the second transport stream in the absence of splicing would cause an audio decoder buffer to have a mean level substantially more than half full, and upon deciding that decoding of the second transport stream in the absence of splicing would cause the audio decoder buffer to have a mean level substantially more than half full and also concluding that a backward audio skew would not result in the second transport stream after splicing if both of the audio access unit that is best aligned with the first video frame and the audio access unit that is best aligned with the second video frame, then excluding, from the spliced transport stream, only one of either the audio access unit that is best aligned with the first video frame or the audio access unit that is best aligned with the second video frame in order to reduce the mean level of the audio decoder buffer for decoding of the second transport stream.

5. The method as claimed in claim 1, which includes deciding that decoding of the second transport stream in the absence of splicing would cause an audio decoder buffer to have a mean level substantially less than half full and also concluding that a backward audio skew, and upon deciding that decoding of the second transport stream in the absence of splicing would cause the audio decoder buffer to have a mean level substantially less than half full and also concluding that a forward audio skew in the second transport stream after splicing would not result without inclusion of an additional audio access unit in the spliced transport stream, then including, in the spliced transport stream, only one additional audio access unit either after the best aligned audio access unit in the first transport stream or before the best aligned audio access unit in the second transport stream in order to increase the mean level of the audio decoder buffer for decoding of the second transport stream.

6. The method as claimed in claim 1, wherein audio access units to be included in the spliced transport stream from the first transport stream and the second transport stream are selected by decoding one of sixteen possible cases determined from a presentation time offset (delta 1) of alignment between audio and video for the audio access unit that is best aligned with the last video frame from the first transport stream to be included in the spliced transport stream, a presentation time offset (delta 2) of alignment between audio and video for the audio access unit that is best aligned with the first video frame from the second transport stream to be included in the spliced transport stream;

and a mean level that an audio decoding buffer would have upon decoding of the second transport stream in the absence of splicing.

7. The method as claimed in claim 6, wherein the decoding includes inspecting the sign of the presentation time offset delta 1, inspecting the sign of the presentation time offset delta 2, comparing a difference between the presentation time offset delta 1 and the presentation time offset delta 2 to a numerical range, and comparing at least two threshold to a mean level that the audio decoding buffer would have upon decoding of the second transport stream in the absence of splicing.

8. The method as claimed in claim 1, wherein the first transport stream, the second transport stream, and the spliced transport stream are MPEG-2 compliant.

9. The method as claimed in claim 1, which includes skewing, in the spliced transport stream, presentation times for audio access units from the second transport stream with respect to presentation times for associated video access units from the second transport stream in order to adjust a mean audio buffer level that would be substantially different from half-full when decoding the second transport stream in the absence of splicing, toward a half-full audio buffer level when decoding the spliced transport stream.

10. The method as claimed in claim 9, wherein audio/visual skew is not introduced in a cumulative fashion for multiple consecutive splices.

11. The method as claimed in claim 9, which includes estimating the mean audio buffer level that would result when decoding the second transport stream, in the absence of splicing, by filtering a sequence of differences between presentation times and corresponding extrapolated program clock reference times for the audio access units of the second transport stream.

12. The method as claimed in claim 11, which further includes estimating a variance of the audio buffer level that would result when decoding the second stream, in the absence of splicing, to ensure with a level of confidence that the audio buffer for the spliced transport stream will not overflow or underflow.

13. The method as claimed in claim 1, which includes modifying the portion of the first transport stream contained in the spliced transport stream in order to eliminate any substantial video presentation discontinuity, the modification being performed prior to finding, in the first transport stream, an audio access unit that is best aligned with the last video frame from the first transport stream to be included in the spliced transport stream.

14. A method of seamless splicing of a first transport stream to a second transport stream to produce a spliced transport stream, the first transport stream including video access units encoding video presentation units representing video frames and audio access units encoding audio presentation units representing segments of a first audio signal, the second transport stream including video access units encoding video presentation units representing video frames and audio access units encoding audio presentation units representing segments of a second audio signal, the first transport stream having a last video frame to be included in the spliced transport stream, and the second transport stream having a first video frame to be included in the spliced transport stream, said method comprising:

(a) computing differences between presentation times and corresponding extrapolated program clock reference times for the audio access units of the second transport stream in order to estimate the mean audio buffer level that would result when decoding the second transport stream; and

(b) concatenating a portion of the first transport stream up to and including the last video frame to a portion of the second transport stream including and subsequent to the first video frame, wherein presentation times for audio access units from the second transport stream are skewed in the spliced transport stream with respect to presentation times for video access units from the second transport stream in order to adjust the estimated mean audio buffer level toward a half-full audio buffer level when decoding the spliced transport stream.

15. The method as claimed in claim 14, which further includes passing the differences through a digital filter to compute an estimate of the mean audio buffer level that would result when decoding the second transport stream.

16. The method as claimed in claim 14, which further includes computing an estimate of a variance of the audio buffer level of the second stream, in the absence of splicing, to ensure with a level of confidence that the audio buffer for the spliced transport stream will not overflow or underflow.

17. A method of seamless splicing of a first transport stream to a second transport stream to produce a spliced transport stream, the first transport stream including video access units encoding video presentation units representing video frames, the video access units of the first transport stream encoding the video presentation units using a data compression technique and containing a variable amount of compressed video data, the second transport stream including video access units encoding video presentation units representing video frames, the video access units of the second transport stream encoding the video presentation units using a data compression technique and containing a variable amount of compressed video data, the first transport stream having a last video frame to be included in the spliced transport stream, and the second transport stream having a first video frame to be included in the spliced transport stream, each of the video access units having a time at which each video access unit is to be received in a video decoder buffer and a time at which said each video access unit is to be removed from the video decoder buffer, said method comprising:

(a) setting the time at which the video access unit for the first video frame of the second transport stream is to be removed from the video decoder buffer to a time following in a decoding sequence next after the time at which the last video access unit for the last frame of the first transport stream is to be removed from the video decoder buffer;

(b) adjusting content of the first transport stream so that the beginning of the video access unit for first video frame of the second transport stream will be received in the video decoder buffer immediately after the end of the video access unit for the last video frame of the first transport stream is received in the video decoder buffer; and

(c) concatenating a portion of the first transport stream up to and including the last video frame to a portion of the second transport stream including and subsequent to the first video frame.

18. The method as claimed in claim 17, wherein the video access unit for the last video frame of the first transport stream has a decode time stamp (DTS_L1) specifying when the video access unit for the last video frame of the first transport stream is to be removed from the video decoder buffer, the video access unit for the first video frame of the second transport stream has a decode time stamp (DTS_F2) specifying when the video access unit for the first video frame of the second transport stream is to be removed from the video decoder buffer, and the method includes setting the decode time stamp of the video access unit for the first video frame of the second video stream substantially equal to the decode time stamp of the video access unit for the last video frame of the first video stream plus one video frame interval.

19. The method as claimed in claim 18, wherein the first transport stream has at least one program clock reference time stamp from which can be extrapolated a time (T_e) at which the end of the video access unit for the last video frame of the first transport stream will be received in the video decoder buffer, and the second transport stream has at least one program clock reference time stamp from which can be extrapolated a time (PCR_e2) at which the beginning of the video access unit for the first video frame of the second transport stream will be received in the video decoder buffer, and the method includes adjusting the content of the first video stream so that PCR_e2 will immediately follow T_e and the original difference (DTS_F2−

PCR_e2) for the second stream will be substantially equal to the difference (DTS_F2−

PCR_e2) between corresponding locations in the spliced transport stream.

20. The method as claimed in claim 19, wherein PCR_e2 in the spliced transport stream is substantially equal to T_e plus the quotient of eight divided by a bit transmission rate for the spliced transport stream.

21. The method as claimed in claim 17, which includes computing a time (T_S) at which a first byte for the first video frame of the second transport stream should be received in the video decoder buffer where T_S=DTS_F1−

DTS_F2+PCR_e2, and computing a difference between T_e and T_S to compute an amount of data that is removed from at least one video access unit in the first transport stream that is included in the spliced transport stream.

22. The method as claimed in claim 17, wherein the content of the first transport stream is adjusted by replacing at least one video access unit in the first transport stream with a video access unit encoding a freeze frame having a size selected so that the beginning of the video access unit for the first video frame of the second transport stream will be received in the video decoder buffer immediately after the end of the video access unit of the last video frame of the first transport stream is received in the video decoder buffer.

23. The method as claimed in claim 22, which includes selecting the size of the freeze frame by selecting the size of at least one slice in the freeze frame.

24. The method as claimed in claim 17, wherein the second transport stream has a higher bit transmission rate than the first transport stream, and the second transport stream is spliced in substantially seamless fashion to the first transport stream.

25. The method as claimed in claim 17, wherein the first transport stream, the second transport stream, and the spliced transport stream are MPEG-2 compliant.

26. The method as claimed in claim 17, wherein the first transport stream includes audio packets, the second transport stream includes audio packets, a plurality of (j) non-obsolete audio packets in the first transport stream follow the beginning of the video access unit of the last video frame in the first transport stream and are included in the spliced transport stream, a total of (k) null packets and obsolete audio packets in the second transport stream follow the video access unit of the first video frame of the second transport stream, and the method includes, for j>

k, replacing the null packets and obsolete audio packets with (k) of the non-obsolete audio packets, inserting the remaining (j−

k) audio packets in the spliced transport stream before the video access unit of the first video frame from the second transport stream, and taking into consideration the amount of data in the remaining (j−

k) audio packets when ensuring that the time at which the video access unit for the first video frame of the second transport stream is to be removed from the video decoder buffer follows in a decoding sequence next after the time at which the last video access unit for the last frame of the first transport stream is to be removed from the video decoder buffer, and that the beginning of the video access unit for the first video frame of the second transport stream will be received in the video decoder buffer immediately after the end of the video access unit for the last video frame of the first transport stream is received in the video decoder buffer and the original difference (DTS_F2−

PCR_e2) for the second stream will be substantially equal to the difference (DTS_F2−

PCR_e2) between the corresponding locations in the spliced transport stream.

27. A method of seamless splicing a first transport stream to a second transport stream to produce a spliced transport stream, the first transport stream including video access units encoding video presentation units representing video frames and audio packets including data of audio access units encoding audio presentation units representing segments of a first audio signal, the second transport stream including video access units encoding video presentation units representing video frames and audio packets including data of audio access units encoding audio presentation units representing segments of a second audio signal, the first transport stream having a last video frame to be included in the spliced transport stream, and the second transport stream having a first video frame to be included in the spliced transport stream, said method comprising:

finding a plurality of (j) non-obsolete audio packets in the first transport stream following the video access unit of the last video frame in the first transport stream and to be included in the spliced transport stream,

finding a total of (k) null packets and obsolete audio packets in the second transport stream follow the video access unit of the first video frame of the second transport stream, where j>

k;

replacing the null packets and obsolete audio packets with (k) of the non-obsolete audio packets; and

concatenating a portion of the first transport stream up to and including the last video frame to a portion of the second transport stream including and subsequent to the first video frame to form the spliced transport stream, wherein the remaining (j−

k) audio packets are inserted in the spliced transport stream before the video access unit of the first video frame from the second transport stream.

28. The method as claimed in claim 27, wherein content of the portion of the first transport stream up to and including the last video frame is adjusted to reduce a video decoding discontinuity, and the amount of data in the remaining (j−

k) audio packets is taken into consideration when adjusting the content of the portion of the first transport stream up to and including the last video frame.