Apparatus and method for synchronizing audio and video frames in an MPEG presentation system

• US 5,949,410 A
• Filed: 10/18/1996
• Issued: 09/07/1999
• Est. Priority Date: 10/18/1996
• Status: Expired due to Term

• Current Assignee: Samsung Electronics Co. Ltd., Samsung Semiconductor Incorporated (Samsung Electronics Co. Ltd.)
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Fung, Hei Tao
• Primary Examiner(s): Feild, Joseph H.
• Assistant Examiner(s): Kindred, Alford W.
• Application Number:
  US08/733,437
• Time in Patent Office: 1,054 Days
• Field of Search: 345/302, 345/328, 345/330, 345/213, 348/12, 348/13, 348/15, 348/8, 348/4, 379/88.13, 379/158, 707/516
• US Class Current: 715/203
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4341 Demultiplexing of audio and video streams
• CPC Class Codes: H04N 21/440281 by altering the temporal resolution, e.g. by frame skipping television signal recording using magnetic recording on tape for reproducing at a rate different from the recording rate H04N5/783

Abstract

Synchronization of MPEG audio and video presentations is attained by detecting a lack of synchrony between the data presentations and modifying the audio data stream based on the detected lack of synchrony. Synchrony of the MPEG audio and video presentations is monitored by tracking the amount of data transferred to audio and video interfaces over time. Synchronization of MPEG audio and video presentations is achieved in a system in which audio and video interfaces are controlled by separate and independent timing signal generators by tracking the amount of audio and video data output by the respective audio and video interfaces. The amount of audio and video data output by the interfaces is compared and, as a result of the comparison, the audio data stream is modified to restore synchrony. Alternatively, presentation of the video data stream is modified to achieve synchrony.

Claims

1. An apparatus for synchronizing audio and video presentation signals in a multiprocessor multimedia system comprising:

a data source for supplying a video data bitstream and an audio data bitstream;

a video signal interface coupled to the data source for receiving the video data bitstream, converting the video data bitstream into video performance signals, and transmitting the video performance signals to a video display, the video signal interface having a first timing signal source;

an audio signal interface coupled to the data source for receiving the audio data bitstream, converting the audio data bitstream into audio performance signals, and transmitting the audio performance signals to an audio speaker, the audio signal interface having a second timing signal source having a timing independent from the first timing signal source;

a controller coupled to the video signal interface, the audio signal interface, and the data source, the controller having a program code including;

an audio/video synchrony monitoring routine for determining whether the audio performance signals are asynchronous to the video performance signals; and

an audio/video synchrony restoring routine operative when the audio performance signals are asynchronous to the video performance signals.

2. An apparatus according to claim 1 wherein:

the audio signal interface generates an audio end of process (aEOP) signal upon transmitting the audio performance signals;

the video signal interface generates a video end of process (vEOP) signal upon transmitting the video performance signals;

the audio/video synchrony monitoring routine includes a routine for counting the relative number of aEOP and vEOP signals to detect when the audio performance signals are asynchronous to the video performance signals.

3. An apparatus according to claim 2 wherein:

the audio/video synchrony monitoring routine includes an asynchrony detection routine for comparing the count of aEOP signals received to an expected number of aEOP signals after receipt of a predefined number of vEOP signals are received.

4. An apparatus according to claim 2 wherein:

the audio/video synchrony monitoring routine includes a routine for determining whether the audio performance signal leads the video performance signal or the audio performance signal lags behind the video performance signal; and

the audio/video synchrony restoring routine includes;

a routine for discarding samples from the audio performance signal when the audio performance signal lags behind the video performance signal; and

a routine for adding samples to the audio performance signal when the audio performance signal leads the video performance signal.

5. An apparatus according to claim 4 wherein:

the routine for discarding samples from the audio performance signal when the audio performance signal lags behind the video performance signal implements a discarding technique selected from among the techniques of;

discarding samples evenly dispersed throughout the audio data bitstream;

discarding samples dispersed at random in the audio data bitstream; and

discarding samples having an amplitude lower than a preselected threshold amplitude.

6. An apparatus according to claim 4 wherein:

the routine for adding samples to the audio performance signal when the audio performance signal leads the video performance signal implements an adding technique selected from among the techniques of;

interpolating samples in the audio data bitstream; and

duplicating samples in the audio data bitstream.

7. An apparatus according to claim 2 wherein:

the audio/video synchrony monitoring and restoring routines include a routine for receiving the video data bitstream at a vertical sync rate and converting the video data bitstream into video performance signals at a video frame rate, the video frame rate being slower than the vertical sync rate so that the audio/video synchrony monitoring and restoring routines include a routine for displaying a selected number of video data bitstream data frames for each video performance signal frame;

the audio/video synchrony monitoring routine includes a routine for determining whether the audio performance signal leads the video performance signal or the audio performance signal lags behind the video performance signal; and

the audio/video synchrony restoring routine includes;

a routine for displaying a fewer number of the plurality of video data bitstream data frames for each video performance signal frame when the video performance signal lags behind the audio performance signal; and

a routine for displaying a larger number of the plurality of video data bitstream data frames for each video performance signal frame when the video performance signal leads the audio performance signal.

8. A method of synchronizing audio and video presentation signals in a multiprocessor multimedia system comprising:

supplying a video data bitstream and an audio data bitstream;

formatting the video data bitstream for display including;

supplying a first timing signal for formatting the video data bitstream;

receiving the video data bitstream;

converting the video data bitstream into video performance signals; and

transmitting the video performance signals to a video display;

formatting the audio data bitstream for performance including;

supplying a second timing signal for formatting the audio data bitstream, the second timing signal being supplied having a timing independent from the first timing signal;

receiving the audio data bitstream;

converting the audio data bitstream into audio performance signals, and

transmitting the audio performance signals to an audio speaker;

determining whether the audio performance signals are asynchronous to the video performance signals; and

restoring audio/video synchrony when the audio performance signals are asynchronous to the video performance signals.

9. A method according to claim 8 further comprising:

generating an audio end of process (aEOP) signal upon transmitting the audio performance signals;

generating a video end of process (vEOP) signal upon transmitting the video performance signals;

counting the relative number of aEOP and vEOP signals to detect when the audio performance signals are asynchronous to the video performance signals.

10. A method according to claim 9 further comprising:

comparing the count of aEOP signals received to an expected number of aEOP signals after receipt of a predefined number of vEOP signals are received.

11. A method according to claim 9 further comprising:

determining whether the audio performance signal leads the video performance signal or the audio performance signal lags behind the video performance signal; and

restoring audio/video synchrony including;

discarding samples from the audio performance signal when the audio performance signal lags behind the video performance signal; and

adding samples to the audio performance signal when the audio performance signal leads the video performance signal.

12. A method according to claim 11 further comprising:

discarding samples from the audio performance signal when the audio performance signal lags behind the video performance signal implements a discarding technique selected from among the techniques of;

discarding samples evenly dispersed throughout the audio data bitstream;

discarding samples dispersed at random in the audio data bitstream; and

discarding samples having an amplitude lower than a preselected threshold amplitude.

13. A method according to claim 11 further comprising:

adding samples to the audio performance signal when the audio performance signal leads the video performance signal implements an adding technique selected from among the techniques of;

interpolating samples in the audio data bitstream; and

duplicating samples in the audio data bitstream.

14. A method according to claim 9 further comprising:

receiving the video data bitstream at a vertical sync rate;

converting the video data bitstream into video performance signals at a video frame rate, the video frame rate being slower than the vertical sync rate, further including;

displaying a plurality of video data bitstream data frames for each video performance signal frame;

determining whether the audio performance signal leads the video performance signal or the audio performance signal lags behind the video performance signal;

displaying a fewer number of the plurality of video data bitstream data frames for each video performance signal frame when the video performance signal lags behind the audio performance signal; and

displaying a larger number of the plurality of video data bitstream data frames for each video performance signal frame when the video performance signal leads the audio performance signal.

15. A multimedia system comprising:

a data source for supplying a video data bitstream and an audio data bitstream;

a video signal interface coupled to the data source for receiving the video data bitstream, converting the video data bitstream into video performance signals, and transmitting the video performance signals to a video display, the video signal interface having a first timing signal source;

an audio signal interface coupled to the data source for receiving the audio data bitstream, converting the audio data bitstream into audio performance signals, and transmitting the audio performance signals to an audio speaker, the audio signal interface having a second timing signal source having a timing independent from the first timing signal source;

a controller coupled to the video signal interface, the audio signal interface, and the data source, the controller including a means for synchronizing audio and video presentation signals, the synchronizing means including;

means for monitoring audio/video synchrony;

means for determining whether the audio performance signals are asynchronous to the video performance signals; and

means for restoring audio/video synchrony operative when the audio performance signals are asynchronous to the video performance signals.

16. A multimedia system according to claim 15 wherein:

the audio signal interface includes means for generating an audio end of process (aEOP) signal upon transmitting the audio performance signals;

the video signal interface includes means for generating a video end of process (vEOP) signal upon transmitting the video performance signals;

the audio/video synchrony monitoring means includes means for counting the relative number of aEOP and vEOP signals to detect when the audio performance signals are asynchronous to the video performance signals.

17. A multimedia system according to claim 16 wherein:

the audio/video synchrony monitoring means includes an asynchrony detection means for comparing the count of aEOP signals received to an expected number of aEOP signals after receipt of a predefined number of vEOP signals are received.

18. A multimedia system according to claim 16 wherein:

the audio/video synchrony monitoring means includes means for determining whether the audio performance signal leads the video performance signal or the audio performance signal lags behind the video performance signal; and

the audio/video synchrony restoring means includes;

means for discarding samples from the audio performance signal when the audio performance signal lags behind the video performance signal; and

means for adding samples to the audio performance signal when the audio performance signal leads the video performance signal.

19. A multimedia system according to claim 18 wherein:

the means for discarding samples from the audio performance signal when the audio performance signal lags behind the video performance signal implements a discarding technique selected from among the techniques of;

discarding samples evenly dispersed throughout the audio data bitstream;

discarding samples dispersed at random in the audio data bitstream; and

discarding samples having an amplitude lower than a preselected threshold amplitude.

20. A multimedia system according to claim 18 wherein:

the means for adding samples to the audio performance signal when the audio performance signal leads the video performance signal implements an adding technique selected from among the techniques of;

interpolating samples in the audio data bitstream; and

duplicating samples in the audio data bitstream.

21. A multimedia system according to claim 16 wherein:

the audio/video synchrony monitoring and restoring routines include a routine for receiving the video data bitstream at a vertical sync rate and converting the video data bitstream into video performance signals at a video frame rate, the video frame rate being slower than the vertical sync rate so that the audio/video synchrony monitoring and restoring routines include a routine for displaying a selected number of video data bitstream data frames for each video performance signal frame;

the audio/video synchrony monitoring routine includes a routine for determining whether the audio performance signal leads the video performance signal or the audio performance signal lags behind the video performance signal; and

the audio/video synchrony restoring routine includes;

a routine for displaying a fewer number of the plurality of video data bitstream data frames for each video performance signal frame when the video performance signal lags behind the audio performance signal; and

a routine for displaying a larger number of the plurality of video data bitstream data frames for each video performance signal frame when the video performance signal leads the audio performance signal.

22. A method of providing an apparatus for synchronizing audio and video presentation signals in a multiprocessor multimedia system comprising:

providing a data source for supplying a video data bitstream and an audio data bitstream;

providing a video signal interface coupled to the data source for receiving the video data bitstream, converting the video data bitstream into video performance signals, and transmitting the video performance signals to a video display, the video signal interface having a first timing signal source;

providing an audio signal interface coupled to the data source for receiving the audio data bitstream, converting the audio data bitstream into audio performance signals, and transmitting the audio performance signals to an audio speaker, the audio signal interface having a second timing signal source having a timing independent from the first timing signal source;

providing a controller coupled to the video signal interface, the audio signal interface, and the data source, the controller having a program code including;

an audio/video synchrony monitoring routine for determining whether the audio performance signals are asynchronous to the video performance signals; and

an audio/video synchrony restoring routine operative when the audio performance signals are asynchronous to the video performance signals.