Real time processing and streaming of spliced encoded MPEG video and associated audio

• US 6,792,047 B1
• Filed: 03/31/2000
• Issued: 09/14/2004
• Est. Priority Date: 01/04/2000
• Status: Expired due to Term

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Faibish, Sorin, Bixby, Peter, Forecast, John, Duso, Wayne W., Gardere, Daniel
• Primary Examiner(s): Kelley, Chris
• Assistant Examiner(s): BUGG, GEORGE A
• Application Number:
  US09/539,747
• Time in Patent Office: 1,628 Days
• Field of Search: 375/240.28, 375/240.26
• US Class Current: 375/240.26
• CPC Class Codes: H04N 21/23424 involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement
• CPC Class Codes: H04N 21/23608 Remultiplexing multiplex streams, e.g. involving modifying time stamps or remapping the packet identifiers
• CPC Class Codes: H04N 21/23611 Insertion of stuffing data into a multiplex stream, e.g. to obtain a constant bitrate synchronisation arrangements in time-division multiplex systems using bit stuffing for systems with different or fluctuating information rates H04J3/073
• CPC Class Codes: H04N 21/44016 involving splicing one content stream with another content stream, e.g. for substituting a video clip

Abstract

Time stamps in an encoded digital motion video data stream (such as an MPEG Transport Stream) are re-stamped for seamless splicing. The re-stamping includes attainment of Presentation Time Stamp (PTS) agreement in the video stream, alignment of audio stream and video stream Presentation Time Stamps, and re-stamping of the Program Clock Reference (PCR) to eliminate jitter and to comply with MPEG standards. The MPEG compliance is achieved by padding, stuffing and insertion of freeze B and P frames and pre-encoded black I frames. Freeze frame insertion can be also achieved by replacing B and P frames by, smaller in size, freeze B and P frames respectively. No frames are discarded such that both spatial and temporal frame accuracy (no extra frame insertion assumed) is achieved. Moreover, padding is inserted into the data stream to compensate for irregularities in the decoder buffer level, and the audio and video buffer levels are managed dynamically to recover from data miss-delivery to minimize secondary effects, including prediction of errors, inducing freeze or skip, and insertion of padding. The start of the play of the second MPEG stream spliced to the first one is frame accurate and starts as scheduled based on the use of an external clock signal in a stream server computer.

Claims

1. A method of real-time seamless splicing of a first transport stream to a second transport stream to produce a spliced transport stream, the first transport stream including video access units encoding video presentation units representing video frames, the video access units of the first transport stream encoding the video presentation units using a data compression technique and containing a variable amount of compressed video data, the second transport stream including video access units encoding video presentation units representing video frames, the video access units of the second transport stream encoding the video presentation units using a data compression technique and containing a variable amount of compressed video data, the first transport stream having a last video frame to be included in the spliced transport stream, and the second transport stream having a first video frame to be included in the spliced transport stream, each of the video access units having a time at which each video access unit is to be received in a video decoder buffer and a time at which said each video access unit is to be removed from the video decoder buffer, said method comprising:

(a) setting the time at which the video access unit for the first video frame of the second transport stream is to be removed from the video decoder buffer to a time following in a decoding sequence next after the time at which the last video access unit for the last frame of the first transport stream is to be removed from the video decoder buffer;

(b) accessing pre-computed metadata for the second transport stream including metadata about a decode time stamp (DTS_F2) at which the beginning of the video access unit for the first video frame of the second transport stream is removed from the video decoder buffer and an extrapolated program clock reference (PCR_e2) time at which the beginning of the video access unit for the first video frame of the second transport stream will be received in the video decoder buffer, and using the pre-computed metadata to adjust content of the first transport stream so that the beginning of the video access unit for the first video frame of the second transport stream will be received in the video decoder buffer immediately after the end of the video access unit for the last video frame of the first transport stream is received in the video decoder buffer while maintaining the difference (DTS_F2−

PCR_e2) in the spliced transport stream; and

(c) concatenating a portion of the first transport stream up to and including the last video frame to a portion of the second transport stream including and subsequent to the first video frame.

2. A method of real-time seamless splicing of a first transport stream to a second transport stream to produce a spliced transport stream, the first transport stream including video access units encoding video presentation units representing video frames, the video access units of the first transport stream encoding the video presentation units using a data compression technique and containing a variable amount of compressed video data, the second transport stream including video access units encoding video presentation units representing video frames, the video access units of the second transport stream encoding the video presentation units using a data compression technique and containing a variable amount of compressed video data, the first transport stream having a last video frame to be included in the spliced transport stream, and the second transport stream having a first video frame to be included in the spliced transport stream, each of the video access units having a time at which each video access unit is to be received in a video decoder buffer and a time at which said each video access unit is to be removed from the video decoder buffer, said method comprising:

(a) setting the time at which the video access unit for the first video frame of the second transport stream is to be removed from the video decoder buffer to a time following in a decoding sequence next after the time at which the last video access unit for the last frame of the first transport stream is to be removed from the video decoder buffer;

(b) accessing pre-computed metadata for the second transport stream including metadata about a decode time stamp (DTS_F2) at which the beginning of the video access unit for the first video frame of the second transport stream is removed from the video decoder buffer and an extrapolated program clock reference (PCR_e2) time at which the beginning of the video access unit for the first video frame of the second transport stream will be received in the video decoder buffer, and using the pre-computed metadata to adjust content of the first transport stream so that the beginning of the video access unit for the first video frame of the second transport stream will be received in the video decoder buffer immediately after the end of the video access unit for the last video frame of the first transport stream is received in the video decoder buffer while maintaining the difference (DTS_F2−

PCR_e2) in the spliced transport stream; and

(c) concatenating a portion of the first transport stream up to and including the last video frame to a portion of the second transport stream including and subsequent to the first video frame;

wherein the content of the first transport stream is adjusted by replacing at least one video access unit in the first transport stream with a video access unit encoding a freeze frame having a size selected so that the beginning of the video access unit for the first video frame of the second transport stream will be received in the video decoder buffer immediately after the end of the video access unit of the last video frame of the first transport stream is received in the video decoder buffer.

3. The method as claimed in claim 2, which includes selecting the size of the freeze frame by selecting the size of at least one slice in the freeze frame.

4. A method of real-time seamless splicing of a first transport stream to a second transport stream to produce a spliced transport stream, the first transport stream including video access units encoding video presentation units representing video frames, the video access units of the first transport stream encoding the video presentation units using a data compression technique and containing a variable amount of compressed video data, the second transport stream including video access units encoding video presentation units representing video frames, the video access units of the second transport stream encoding the video presentation units using a data compression technique and containing a variable amount of compressed video data, the first transport stream having a last video frame to be included in the spliced transport stream, and the second transport stream having a first video frame to be included in the spliced transport stream, each of the video access units having a time at which each video access unit is to be received in a video decoder buffer and a time at which said each video access unit is to be removed from the video decoder buffer, said method comprising:

(a) setting the time at which the video access unit for the first video frame of the second transport stream is to be removed from the video decoder buffer to a time following in a decoding sequence next after the time at which the last video access unit for the last frame of the first transport stream is to be removed from the video decoder buffer;

(b) accessing pre-computed metadata for the second transport stream including metadata about a decode time stamp (DTS_F2) at which the beginning of the video access unit for the first video frame of the second transport stream is removed from the video decoder buffer and an extrapolated program clock reference (PCR_e2) time at which the beginning of the video access unit for the first video frame of the second transport stream will be received in the video decoder buffer, and using the pre-computed metadata to adjust content of the first transport stream so that the beginning of the video access unit for the first video frame of the second transport stream will be received in the video decoder buffer immediately after the end of the video access unit for the last video frame of the first transport stream is received in the video decoder buffer while maintaining the difference (DTS_F2−

PCR_e2) in the spliced transport stream; and

(c) concatenating a portion of the first transport stream up to and including the last video frame to a portion of the second transport stream including and subsequent to the first video frame;

wherein the second transport stream has a higher bit transmission rate than the first transport stream.

5. A method of real-time seamless splicing of a first transport stream to a second transport stream to produce a spliced transport stream, the first transport stream including video access units encoding video presentation units representing video frames, the video access units of the first transport stream encoding the video presentation units using a data compression technique and containing a variable amount of compressed video data, the second transport stream including video access units encoding video presentation units representing video frames, the video access units of the second transport stream encoding the video presentation units using a data compression technique and containing a variable amount of compressed video data, the first transport stream having a last video frame to be included in the spliced transport stream, and the second transport stream having a first video frame to be included in the spliced transport stream, each of the video access units having a time at which each video access unit is to be received in a video decoder buffer and a time at which said each video access unit is to be removed from the video decoder buffer, said method comprising:

(a) setting the time at which the video access unit for the first video frame of the second transport stream is to be removed from the video decoder buffer to a time following in a decoding sequence next after the time at which the last video access unit for the last frame of the first transport stream is to be removed from the video decoder buffer;

(b) accessing pre-computed metadata for the second transport stream including metadata about a decode time stamp (DTS_F2) at which the beginning of the video access unit for the first video frame of the second transport stream is removed from the video decoder buffer and an extrapolated program clock reference (PCR_e2) time at which the beginning of the video access unit for the first video frame of the second transport stream will be received in the video decoder buffer, and using the pre-computed metadata to adjust content of the first transport stream so that the beginning of the video access unit for the first video frame of the second transport stream will be received in the video decoder buffer immediately after the end of the video access unit for the last video frame of the first transport stream is received in the video decoder buffer while maintaining the difference (DTS_F2−

PCR_e2) in the spliced transport stream; and

(c) concatenating a portion of the first transport stream up to and including the last video frame to a portion of the second transport stream including and subsequent to the first video frame;

wherein the second transport stream has a lower bit transmission rate than the first transport stream.

6. A method of real-time seamless splicing of a first transport stream to a second transport stream to produce a spliced transport stream, the first transport stream including video access units encoding video presentation units representing video frames, the video access units of the first transport stream encoding the video presentation units using a data compression technique and containing a variable amount of compressed video data, the second transport stream including video access units encoding video presentation units representing video frames, the video access units of the second transport stream encoding the video presentation units using a data compression technique and containing a variable amount of compressed video data, the first transport stream having a last video frame to be included in the spliced transport stream, and the second transport stream having a first video frame to be included in the spliced transport stream, each of the video access units having a time at which each video access unit is to be received in a video decoder buffer and a time at which said each video access unit is to be removed from the video decoder buffer, said method comprising:

(a) setting the time at which the video access unit for the first video frame of the second transport stream is to be removed from the video decoder buffer to a time following in a decoding sequence next after the time at which the last video access unit for the last frame of the first transport stream is to be removed from the video decoder buffer;

(b) accessing pre-computed metadata for the second transport stream including metadata about a decode time stamp (DTS_F2) at which the beginning of the video access unit for the first video frame of the second transport stream is removed from the video decoder buffer and an extrapolated program clock reference (PCR_e2) time at which the beginning of the video access unit for the first video frame of the second transport stream will be received in the video decoder buffer, and using the pre-computed metadata to adjust content of the first transport stream so that the beginning of the video access unit for the first video frame of the second transport stream will be received in the video decoder buffer immediately after the end of the video access unit for the last video frame of the first transport stream is received in the video decoder buffer while maintaining the difference (DTS_F2−

PCR_e2) in the spliced transport stream; and

(c) concatenating a portion of the first transport stream up to and including the last video frame to a portion of the second transport stream including and subsequent to the first video frame;

wherein the first transport stream, the second transport stream, and the spliced transport stream are MPEG-2 compliant.

7. A method of real-time seamless splicing of a first transport stream to a second transport stream to produce a spliced transport stream, the first transport stream including video access units encoding video presentation units representing video frames, the video access units of the first transport stream encoding the video presentation units using a data compression technique and containing a variable amount of compressed video data, the second transport stream including video access units encoding video presentation units representing video frames, the video access units of the second transport stream encoding the video presentation units using a data compression technique and containing a variable amount of compressed video data, the first transport stream having a last video frame to be included in the spliced transport stream, and the second transport stream having a first video frame to be included in the spliced transport stream, each of the video access units having a time at which each video access unit is to be received in a video decoder buffer and a time at which said each video access unit is to be removed from the video decoder buffer, said method comprising:

(a) setting the time at which the video access unit for the first video frame of the second transport stream is to be removed from the video decoder buffer to a time following in a decoding sequence next after the time at which the last video access unit for the last frame of the first transport stream is to be removed from the video decoder buffer;

(b) accessing pre-computed metadata for the second transport stream including metadata about a decode time stamp (DTS_F2) at which the beginning of the video access unit for the first video frame of the second transport stream is removed from the video decoder buffer and an extrapolated program clock reference (PCR_e2) time at which the beginning of the video access unit for the first video frame of the second transport stream will be received in the video decoder buffer, and using the pre-computed metadata to adjust content of the first transport stream so that the beginning of the video access unit for the first video frame of the second transport stream will be received in the video decoder buffer immediately after the end of the video access unit for the last video frame of the first transport stream is received in the video decoder buffer while maintaining the difference (DTS_F2−

PCR_e2) in the spliced transport stream; and

(c) concatenating a portion of the first transport stream up to and including the last video frame to a portion of the second transport stream including and subsequent to the first video frame;

which includes parsing the first transport stream and the second transport streams in real time for audio PES packet headers, parsing the audio PES packets in, real time for audio access units (AAUs), identifying in real time non-obsolete AAUs in the first transport stream following the last video frame in the first transport stream and identifying in real time obsolete AAUs in the second transport stream following the first video frame in the second transport stream, reformatting the non-obsolete AAUs in the first transport stream following the last video frame in the first transport stream in real time, eliminating the obsolete AAUs in the second transport stream following the first video frame in the second transport stream in real time, and computing time stamp offsets in real time and re-stamping, in real time, time stamps and continuity counters in the spliced transport stream following the first video frame from the second transport stream.

8. A method of real-time seamless splicing of a first transport stream to a second transport stream to produce a spliced transport stream, the first transport stream including video access units encoding video presentation units representing video frames, the video access units of the first transport stream encoding the video presentation units using a data compression technique and containing a variable amount of compressed video data, the second transport stream including video access units encoding video presentation units representing video frames, the video access units of the second transport stream encoding the video presentation units using a data compression technique and containing a variable amount of compressed video data, the first transport stream having a last video frame to be included in the spliced transport stream, and the second transport stream having a first video frame to be included in the spliced transport stream, each of the video access units having a time at which each video access unit is to be received in a video decoder buffer and a time at which said each video access unit is to be removed from the video decoder buffer, said method comprising:

(a) setting the time at which the video access unit for the first video frame of the second transport stream is to be removed from the video decoder buffer to a time following in a decoding sequence next after the time at which the last video access unit for the last frame of the first transport stream is to be removed from the video decoder buffer;

(b) accessing pre-computed metadata for the second transport stream including metadata about a decode time stamp (DTS_F2) at which the beginning of the video access unit for the first video frame of the second transport stream is removed from the video decoder buffer and an extrapolated program clock reference (PCR_e2) time at which the beginning of the video access unit for the first video frame of the second transport stream will be received in the video decoder buffer, and using the pre-computed metadata to adjust content of the first transport stream so that the beginning of the video access unit for the first video frame of the second transport stream will be received in the video decoder buffer immediately after the end of the video access unit for the last video frame of the first transport stream is received in the video decoder buffer while maintaining the difference (DTS_F2−

PCR_e2) in the spliced transport stream; and

(c) concatenating a portion of the first transport stream up to and including the last video frame to a portion of the second transport stream including and subsequent to the first video frame;

wherein the real-time seamless splicing is performed by a server when reading the first transport stream and the second transport stream from file storage and streaming the spliced transport stream to an application.

9. The method as claimed in claim 8, which includes the server reading data of the first transport stream and the second transport stream from file storage and storing the data of the first transport stream and the second transport stream in a carousel of buffers, accessing the carousel of buffers to splice the first transport stream to the second transport stream and store data of the spliced transport stream in the carousel of buffers, and then accessing the carousel of buffers to transfer data of the spliced transport stream from the carousel of buffers for streaming to the application.

10. The method as claimed in claim 9, wherein the data of the first transport stream are stored in a first series of alternate buffers in sequence around the carousel, and data of the second transport stream are stored in a second series of alternate buffers in sequence around the carousel.

11. The method as claimed in claim 9, wherein the server includes a stream server computer and a cached disk storage subsystem, the carousel of buffers are allocated in random access memory of the stream server computer, and the stream server computer is programmed with a file system that maps the file storage to disk storage in the cached disk storage subsystem and that reads the data of the first transport stream and the second transport stream from disk storage of the cached disk storage subsystem for storing of the data of the first transport stream and the second transport stream in the carousel of buffers.

12. The method as claimed in claim 9, wherein the server includes a network interface board that reads the data of the spliced transport stream from the carousel of buffers for streaming to the application using a direct memory access protocol.

13. The method as claimed in claim 12, wherein the network interface board communicates with the application using a technology selected from group consisting of DVB, ATM, Ethernet, Fiber Channel, and Gigabit Ethernet.

14. A method of real-time seamless splicing of a first transport stream to a second transport stream to produce a spliced transport stream, the first transport stream including video access units encoding video presentation units representing video frames, the video access units of the first transport stream encoding the video presentation units using a data compression technique and containing a variable amount of compressed video data, the second transport stream including video access units encoding video presentation units representing video frames, the video access units of the second transport stream encoding the video presentation units using a data compression technique and containing a variable amount of compressed video data, the first transport stream having a last video frame to be included in the spliced transport stream, and the second transport stream having a first video frame to be included in the spliced transport stream, each of the video access units having a time at which each video access unit is to be received in a video decoder buffer and a time at which said each video access unit is to be removed from the video decoder buffer, said method comprising:

parsing the first transport stream and the second transport streams in real time for audio PES packet headers, parsing the audio PES packets in real time for audio access units (AAUs), identifying in real time non-obsolete AAUs in the first transport stream following the last video frame in the first transport stream and identifying in real time obsolete AAUs in the second transport stream following the first video frame in the second transport stream, reformatting the non-obsolete AAUs in the first transport stream following the last video frame in the first transport stream in real time, eliminating the obsolete AAUs in the second transport stream following the first video frame in the second transport stream in real time, concatenating a portion of the first transport stream up to and including the last video frame to a portion of the second transport stream including and subsequent to the first video frame, and computing time stamp offsets in real time and re-stamping, in real time, time stamps and continuity counters in the spliced transport stream following the first video frame from the second transport stream.

15. The method as claimed in claim 14, which includes adjusting content of the first transport stream by replacing at least one video access unit in the first transport stream with a video access unit encoding a freeze frame having a size selected so that the beginning of the video access unit for the first video frame of the second transport stream will be received in the video decoder buffer immediately after the end of the video access unit of the last video frame of the first transport stream is received in the video decoder buffer.

16. The method as claimed in claim 15, which includes selecting the size of the freeze frame by selecting the size of at least one slice in the freeze frame.

17. The method as claimed in claim 14, wherein the second transport stream has a higher bit transmission rate than the first transport stream.

18. The method as claimed in claim 14, wherein the first transport stream, the second transport stream, and the spliced transport stream are MPEG-2 compliant.

19. The method as claimed in claim 14, wherein the real-time seamless splicing is performed by a server when reading the first transport stream and the second transport stream from file storage and streaming the spliced transport stream to an application.

20. The method as claimed in claim 19, wherein the server streams the spliced transport stream to the application using Internet Protocol (IP).

21. The method as claimed in claim 19, which includes the server reading data of the first transport stream and the second transport stream from file storage and storing the data of the first transport stream and the second transport stream in a carousel of buffers, accessing the carousel of buffers to splice the first transport stream to the second transport stream and store data of the spliced transport stream in the carousel of buffers, and then accessing the carousel of buffers to transfer data of the spliced transport stream from the carousel of buffers for streaming to the application.

22. The method as claimed in claim 21, wherein the data of the first transport stream are stored in a first series of alternate buffers in sequence around the carousel, and data of the second transport stream are stored in a second series of alternate buffers in sequence around the carousel.

23. The method as claimed in claim 21, wherein the server includes a stream server computer and a cached disk storage subsystem, the carousel of buffers are allocated in random access memory of the stream server computer, and the stream server computer is programmed with a file system that maps the file storage to disk storage in the cached disk storage subsystem and that reads the data of the first transport stream and the second transport stream from disk storage of the cached disk storage subsystem for storing of the data of the first transport stream and the second transport stream in the carousel of buffers.

24. The method as claimed in claim 21, wherein the server includes a network interface board that reads the data of the spliced transport stream from the carousel of buffers for streaming to the application using a direct memory access protocol.