Synchronization of audio/video information

• US 5,396,497 A
• Filed: 03/16/1993
• Issued: 03/07/1995
• Est. Priority Date: 02/26/1993
• Status: Expired due to Term

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Veltman, Markus H.
• Primary Examiner(s): Olms, Douglas W.
• Assistant Examiner(s): Kizou, Hassan
• Application Number:
  US08/032,341
• Time in Patent Office: 721 Days
• Field of Search: 370/60, 370/60.1, 370/77, 370/94.1, 370/94.2, 370/94.3, 370/100.1, 370/105.1, 370/105.4, 370/110.1, 370/112, 375/106, 375/107, 375/110, 375/111, 375/114, 375/118, 375/119, 375/120, 358/12, 358/14, 358/15, 358/141, 358/142, 358/143, 358/146, 358/148, 358/149, 358/198, 348/423, 348/465
• US Class Current: 375/240.25
• CPC Class Codes: H04N 19/152 by measuring the fullness of the transmission buffer
• CPC Class Codes: H04N 19/50 using predictive coding H04N19/61 takes precedence
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 21/4305 Synchronising client clock from received content stream, e.g. locking decoder clock with encoder clock, extraction of the PCR packets arrangements for synchronising receiver with transmitter by comparing receiver clock with transmitter clock H04L7/0012; arrangements for synchronising receiver with transmitter wherein the receiver takes measures against momentary loss of synchronisation H04L7/0083
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4341 Demultiplexing of audio and video streams
• CPC Class Codes: H04N 21/4348 Demultiplexing of additional data and video streams
• CPC Class Codes: H04N 21/8455 involving pointers to the content, e.g. pointers to the I-frames of the video stream
• CPC Class Codes: H04N 5/9206 the additional signal being a character code signal
• CPC Class Codes: H04N 5/9264 using transform coding
• CPC Class Codes: H04N 5/9265 with processing of the sound signal
• CPC Class Codes: H04N 7/56 Synchronising systems therefor

Abstract

A compressed video signal and a compressed audio signal decoding apparatus which including a local access unit counter intended for digital AV applications where access unit headers (e.g. frame headers, audio unit headers or other kind of data unit headers) may be lost or generated by storage media or transmission errors. In order to prevent a resulting AV sync error a value called "access unit count" is included in each elementary stream. With this value each decoder can detect lost or false access unit headers, and correct the AV synchronization. In order to support editing, an extra bit can be added to indicate discontinuity in the access unit count.

Claims

1. A demultiplexer for a multiplexed audio video signal, the demultiplexer comprising:

a clock producing a time reference;

a demultiplexing switch for separating a composite data stream into audio sample batches, audio time stamps, video frames, and video time stamps, wherein the audio time stamps are fewer or equal in number than the audio sample batches and wherein the video time stamps are fewer or equal in number than the video frames;

an audio phase-locked loop, receiving the audio time stamps and the time reference, for comparing the time reference with the audio time stamps to produce an audio timing signal and to associate one of the audio sample batches with each of the audio time stamps;

an audio decoder receiving the audio sample batches and the audio timing signal, and for decoding the audio sample batches in synchronization with the audio timing signal;

a video phase-locked loop, receiving the video time stamps and the time reference, for comparing the time reference with the video time stamps to produce a video timing signal and to associate one of the video frames with each of the video time stamps; and

a video decoder receiving the video frames and the video timing signal, and for decoding the video frames in synchronization with the video timing signal.

2. The demultiplexer of claim 1, further comprising a video time stamp buffer receiving the video time stamps from the demultiplexing switch.

3. The demultiplexer of claim 1, further comprising an audio time stamp buffer receiving the audio time stamps from the demultiplexing switch.

4. The demultiplexer of claim 1, further comprising a video buffer receiving the video frames from the demultiplexing switch.

5. The demultiplexer of claim 1, further comprising an audio buffer receiving the audio sample batches from the demultiplexing switch.

6. The demultiplexer of claim 1, wherein the time reference is indicative of an absolute time elapsed since a reference time.

7. The demultiplexer of claim 1, wherein the clock operates at 90 kHz.

8. The demultiplexer of claim 1, wherein the multiplexed audio video signal is an MPEG standard signal.

9. The demultiplexer of claim 6, wherein the one of the audio sample batches associated by the audio phase-locked loop with each of the audio time stamps is the one of the audio sample batches that, absent the audio time stamp, would be decoded closest in time, as measured by the clock, to the value of the audio time stamp.

10. The demultiplexer of claim 6, wherein the one of the video frames associated by the video phase-locked loop each of the video time stamps is the one of the video frames that, absent the video time stamp, would be decoded closest in time, as measured by the clock, to the value of the video time stamp.

11. A demultiplexer for an MPEG standard multiplexed audio video signal, the demultiplexer comprising:

a 90 kHz clock producing a time reference indicative of an absolute time elapsed since a reference time;

a demultiplexing switch for separating a composite data stream into audio sample batches, audio time stamps, video frames, and video time stamps, wherein the audio time stamps are fewer or equal in number than the audio sample batches and the video time stamps are fewer or equal in number than the video frames;

a video timestamp buffer receiving the video timestamps from the demultiplexing switch;

an audio timestamp buffer receiving the audio time stamps from the demultiplexing switch;

a video buffer receiving the video frames from the demultiplexing switch;

an audio buffer receiving the audio sample batches from the demultiplexing switch;

an audio phase-locked loop, receiving the audio time stamps from the audio timestamp buffer and receiving the time reference, for comparing the time reference with the audio time stamps to produce an audio timing signal and to associate one of the audio sample batches with each of the audio time stamps, the one of the audio sample batches associated by the audio phase-locked loop with each of the audio time stamps being the one of the audio sample batches that, absent the audio time stamp, would be decoded closest in time, as measured by the clock, to the value of the audio time stamp;

an audio decoder receiving the audio sample batches and the audio timing signal, and for decoding the audio sample batches in synchronization with the audio timing signal;

a video phase-locked loop, receiving the video time stamps from the video time stamp buffer and receiving the time reference, for comparing the time reference with the video time stamps to produce a video timing signal and to associate one of the video frames with each of the video time stamps, the one of the video frames associated by the video phase-locked loop with each of the video time stamps being the one of the video frames that, absent the video time stamp would be decoded closest in time, as measured by the clock, to the value of the video time stamp; and

a video decoder receiving the video frames and the video timing signal, and for decoding the video frames in synchronization with the video timing signal.

12. A decoder, comprising:

a clock producing a time reference;

means for separating a composite data stream into data units and data time stamps, wherein the data time stamps are fewer or equal in number than the data units;

a phase-locked loop, receiving the time stamps and the time reference, for comparing the time reference with the time stamps to produce a timing signal and to associate one of the data units with each of the data time stamps, the one of the data units associated by the phase-locked loop with each of the data time stamps being the one of the data units that, absent the data time stamp, would be decoded closest in time, as measured by the clock, to the value of the data time stamp; and

a data decoder receiving the data units and the timing signal, and for decoding the data units in synchronization with the timing signal.

13. A demultiplexer for a multiplexed audio video signal, the demultiplexer comprising:

a demultiplexing switch for separating a composite data stream into audio sample batches, audio time stamps, video frames, and video time stamps;

an audio phase-locked loop, receiving the audio time stamps, for producing an audio timing signal in synchronization with the audio time stamps;

an audio decoder receiving the audio sample batches and the audio timing signal, and for decoding the audio sample batches in synchronization with the audio timing signal;

the audio decoder including means for extracting an access unit count from the audio sample batches and comparing the access unit count with a stored access unit count to verify that the audio decoder is operating in proper synchronization;

a video phase-locked loop, receiving the video time stamps, for producing a video timing signal in synchronization with the video time stamps;

a video decoder receiving the video frames and the video timing signal, and for decoding the video frames in synchronization with the video timing signal; and

the video decoder including means for extracting an access unit count from the video frames and comparing the access unit count with a stored access unit count to verify that the video decoder is operating in proper synchronization.

14. The demultiplexer of claim 13, further comprising a video timestamp buffer receiving the video timestamps from the demultiplexing switch.

15. The demultiplexer of claim 13, further comprising an audio timestamp buffer receiving the audio timestamps from the demultiplexing switch.

16. The demultiplexer of claim 13, further comprising a video buffer receiving the video frames from the demultiplexing switch.

17. The demultiplexer of claim 13, further comprising an audio buffer receiving the audio sample batches from the demultiplexing switch.

18. A decoder, comprising:

means for separating a composite data stream into data units and data time stamps;

timing means receiving the data time stamps, for producing a timing signal in synchronization with the data time stamps; and

a data decoder receiving the data units and the timing signal, for decoding the data units in synchronization with the timing signal;

the data decoder including means for extracting an access unit count from each of the data units and comparing the access unit count with a stored access unit count to verify that the data decoder is operating in proper synchronization.

19. The decoder of claim 18, wherein the timing means includes a phase-locked loop.

20. A method for demultiplexing a multiplexed audio video signal, the method comprising the steps of:

separating a composite data stream into audio sample batches, audio time stamps, video frames, and video time stamps;

producing an audio timing signal in synchronization with the audio time stamps;

decoding the audio sample batches in synchronization with the audio timing signal;

extracting an audio access unit count from the audio sample batches and comparing the audio access unit count with a stored audio access unit count to verify audio synchronization;

producing a video timing signal in synchronization with the video time stamps;

decoding the video frames in synchronization with the video timing signal;

extracting a video access unit count from the video frames and comparing the video access unit count with a stored video access unit count to verify video synchronization; and

incrementing the stored audio access unit count each time an audio sample batch is decoded, and incrementing the stored video access unit each time a video frame is decoded.

21. The method of claim 20, further comprising the step of duplicating a video frame if the video access unit count for the video frame is larger than the stored video access unit count.

22. The method of claim 20, further comprising the step of omitting a video frame if the video access unit count for the video frame is smaller than the stored video access unit count.

23. 3The method of claim 20, further comprising the step of omitting an audio sample batch if the audio access unit count for the audio sample batch is smaller than the stored audio access unit count.

24. The method of claim 20, further comprising the step of duplicating an audio sample batch if the audio access unit count for the audio sample batch is larger than the stored audio access unit count.

25. A method for processing a video and an associated audio signal, the method comprising the steps of:

providing a signal including audio sample batches and video frames;

decoding the audio sample batches;

extracting an audio access unit count from the audio sample batches and comparing the audio access unit count with a stored audio access unit count to verify audio synchronization;

decoding the video frames;

extracting a video access unit count from the video frames and comparing the video access unit count with a stored video access unit count to verify video synchronization; and

incrementing the stored audio access unit count each time an audio sample batch is decoded, and incrementing the stored video access unit count each time a video frame is decoded.

26. The method of claim 25, further comprising the step of duplicating a video frame if the video access unit count for the video frame is larger than the stored video access unit count.

27. The method of claim 25, further comprising the step of omitting a video frame if the video access unit count for the video frame is smaller than the stored video access unit count.

28. The method of claim 25, further comprising the step of omitting an audio sample batch if the audio access unit count for the audio sample batch is smaller than the stored audio access unit count.

29. The method of claim 25, further comprising the step of duplicating an audio sample batch if the audio access unit count for the audio sample batch is larger than the stored audio access unit count.

30. The method of claim 25, wherein the audio video signal includes additional audio and video time stamps respectively indicating presentation times for ones of the audio sample batches and video frames.

31. A method for processing a stream of data bits divided into access units, the method comprising the steps of:

receiving a first access unit;

initializing a register with an access unit count corresponding to the first access unit;

receiving and decoding one or more subsequent access units following the first access unit;

incrementing the access unit count stored in the register for each of the subsequent access units and comparing the access unit count stored in the register with an access unit count extracted from each of the subsequent access units;

determining if the access unit count extracted from each of the subsequent access units is the same as the access unit count stored in the register to verify synchronization;

omitting one of the subsequent access units if the access unit count extracted from said one of the subsequent access units is smaller than the access unit count stored in the register;

detecting a discontinuity bit in the stream of data bits; and

reinitializing the register upon detection of the discontinuity bit.

32. The method of claim 31, wherein the register is reinitialized with an access unit count extracted from an access unit following the discontinuity bit.