Method and apparatus for synchronizing audio and video data

• US 20030128294A1
• Filed: 01/04/2002
• Published: 07/10/2003
• Est. Priority Date: 01/04/2002
• Status: Active Grant

• Current Assignee: Rovi Technologies Corporation (Adeia Inc.)
• Original Assignee: Microsoft Corporation
• Inventors: Lundblad, James, Khanna, Ramaneek
• Granted Patent: US 6,906,755 B2
• Time in Patent Office: Days
• US Class Current: 348/515
• CPC Class Codes: H04J 3/0638 Clock or time synchronisation among nodes; Internode synchronisation synchronization for ring networks H04L12/422; data switching networks with synchronous transmission H04L12/43
• CPC Class Codes: H04N 21/2368 Multiplexing of audio and video streams
• CPC Class Codes: H04N 21/426 Internal components of the client ; Characteristics thereof H04N5/44 takes precedence
• CPC Class Codes: H04N 21/4305 Synchronising client clock from received content stream, e.g. locking decoder clock with encoder clock, extraction of the PCR packets arrangements for synchronising receiver with transmitter by comparing receiver clock with transmitter clock H04L7/0012; arrangements for synchronising receiver with transmitter wherein the receiver takes measures against momentary loss of synchronisation H04L7/0083
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4341 Demultiplexing of audio and video streams
• CPC Class Codes: H04N 21/4392 involving audio buffer management
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56
• CPC Class Codes: H04N 5/602 for digital sound signals

Abstract

A system receives a transport stream containing video data and audio data. A determination is made regarding the time required to process the video data contained in the transport stream and the time required to process the audio data contained in the transport stream. The system then determines a difference in time to process the video contained in the transport stream as compared to the audio data contained in the transport stream. Presentation of the audio data is delayed by this difference in time to synchronize presentation of the audio data with presentation of the video data.

Claims

1. A method comprising:

receiving a transport stream containing video data and audio data;

determining a time required to process the video data contained in the transport stream;

determining a time required to process the audio data contained in the transport stream;

determining a difference in time to process the video data contained in the transport stream as compared to the audio data contained in the transport stream; and

delaying presentation of the audio data by the difference in time to process the video data contained in the transport stream as compared to the audio data contained in the transport stream;

2. A method as recited in claim 1, wherein delaying presentation of the audio data by the difference in time to process the video data contained in the transport stream as compared to the audio data contained in the transport stream is performed if the difference in time exceeds a threshold.

3. A method as recited in claim 1, wherein determining a time required to process the video data contained in the transport stream includes receiving a video presentation delay from a video display software routine.

4. A method as recited in claim 1, wherein determining a time required to process the video data contained in the transport stream includes calculating a video presentation delay by comparing a presentation time stamp and a system time clock.

5. A method as recited in claim 1, wherein the method of claim 1 is repeated at periodic intervals.

6. A method as recited in claim 1, wherein the method of claim 1 is performed for each received frame of video data.

7. A method as recited in claim 1, wherein delaying presentation of the audio data by the difference in time to process the video data contained in the transport stream as compared to the audio data contained in the transport stream includes storing the audio data in a DMA buffer with a delay that corresponds to the difference in time to process the video data contained in the transport stream as compared to the audio data contained in the transport stream.

8. A method as recited in claim 1, further comprising decoding the video data received in the transport stream.

9. A method as recited in claim 1, further comprising decoding the audio data received in the transport stream.

10. A method comprising:

receiving a transport stream containing video data and audio data;

determining a time required to process the video data contained in the transport stream;

determining a time required to process the audio data contained in the transport stream;

determining a difference in time to process the video data contained in the transport stream as compared to the audio data contained in the transport stream;

if the time required to process the video data is greater than the time required to process the audio data, delaying presentation of the audio data by the difference in time to process the video data contained in the transport stream as compared to the audio data contained in the transport stream; and

if the time required to process the audio data is greater than the time required to process the video data, delaying presentation of the video data by the difference in time to process the video data contained in the transport stream as compared to the audio data contained in the transport stream.

11. A method as recited in claim 10, wherein determining a time required to process the video data contained in the transport stream includes receiving a video presentation delay.

12. A method as recited in claim 10, wherein determining a time required to process the video data contained in the transport stream includes calculating a video presentation delay by comparing a presentation time stamp and a system time clock.

13. A method as recited in claim 10, further comprising decoding the video data received in the transport stream.

14. A method as recited in claim 10, further comprising decoding the audio data received in the transport stream.

15. A method comprising:

receiving a transport stream containing video data and audio data;

identifying a presentation time stamp in the transport stream;

identifying a value associated with a system time clock;

determining a time required to process the video data contained in the transport stream by comparing the presentation time stamp and the system time clock; and

delaying presentation of the audio data by the time required to process the video data contained in the transport stream.

16. A method as recited in claim 15, wherein delaying presentation of the audio data by the time required to process the video data contained in the transport stream is performed if the time required to process the video data contained in the transport stream exceeds a threshold.

17. A method as recited in claim 15, wherein delaying presentation of the audio data by the time required to process the video data contained in the transport stream includes storing the audio data in a buffer with a delay that corresponds to the time required to process the video data contained in the transport stream.

18. A method as recited in claim 15, wherein delaying presentation of the audio data by the time required to process the video data contained in the transport stream includes:

determining a position of a DMA read pointer; and

storing the audio data in a DMA buffer with a delay that matches the time required to process the video data contained in the transport stream.

19. A method as recited in claim 15, further comprising decoding the received video data.

20. A method as recited in claim 15, further comprising decoding the received audio data.

21. An apparatus comprising:

a transport stream decoder coupled to receive a transport stream and configured to separate audio data and video data from the transport stream;

a video processing module configured to receive video data from the transport stream decoder;

an audio processing module configured to receive audio data from the transport stream decoder; and

a clock control module coupled to the transport stream decoder to receive timing data from the transport stream, the clock control module further coupled to the video processing module and the audio processing module and further configured to delay presentation of the audio data by a difference in time to process the video data as compared to the audio data.

22. An apparatus as recited in claim 21, wherein the audio processing module delays presentation of the audio data by storing the audio data in a buffer with a delay that corresponds to the difference in time to process the video data as compared to the audio data.

23. An apparatus as recited in claim 21, wherein the transport stream decoder is further configured to decode the video data and the audio data contained in the transport stream.

24. An apparatus as recited in claim 21, wherein the transport stream decoder is further configured to decode the video data and the audio data contained in the transport stream as well as timing information contained in the transport stream.

25. An apparatus comprising:

a system time clock coupled configured to maintain a current system;

a video display software routine executing on a processor and configured to receive a first time stamp from a transport stream and receive a current system time from the system time clock, the video display software routine further configured to determine a video presentation delay based on the first time stamp and the current system time; and

an audio software routine executing on the processor and configured to receive the video presentation delay from the video display software routine and delay presentation of audio data contained in the transport stream based on the video presentation delay.

26. An apparatus as recited in claim 25, wherein the audio software routine delay s presentation of audio data contained in the transport stream by storing the audio data in a buffer with a delay that corresponds to the video presentation delay.

27. An apparatus as recited in claim 26, further comprising audio reproduction hardware configured to retrieve audio data stored in the buffer and generate an audio analog signal associated with the audio data.

28. An apparatus as recited in claim 25, wherein the video display software routine determines the video presentation delay each time a vertical retrace sync signal is received.

29. An apparatus as recited in claim 25, wherein the video display software routine determines the video presentation delay at periodic intervals.

30. One or more computer-readable media having stored thereon a computer program that, when executed by one or more processors, causes the one or more processors to:

receive a transport stream containing video data and audio data;

identify a time stamp in the transport stream;

determine a current system time;

determine a time required to process the video data contained in the transport stream by comparing the presentation time stamp and the current system time; and

delay presentation of the audio data by the time required to process the video data contained in the transport stream.

31. One or more computer-readable media as recited in claim 30, wherein delaying presentation of the audio data by the time required to process the video data contained in the transport stream includes storing the audio data in a buffer with a delay that corresponds to the time required to process the video data contained in the transport stream.

32. One or more computer-readable media as recited in claim 30, further causing the one or more processors to decode the audio data and decode the video data.