System for frame based audio synchronization and method thereof

• US 7,130,316 B2
• Filed: 04/11/2001
• Issued: 10/31/2006
• Est. Priority Date: 04/11/2001
• Status: Active Grant

• Current Assignee: Avago Technologies International Sales Pte Limited (Broadcom, Inc.)
• Original Assignee: ATI Technologies Incorporated (Advanced Micro Devices, Inc.)
• Inventors: Kovacevic, Branko D.
• Primary Examiner(s): Vu, Huy D.
• Assistant Examiner(s): RYMAN, DANIEL J
• Application Number:
  US09/833,493
• Publication Number:
  US 20020150126A1
• Time in Patent Office: 2,029 Days
• Field of Search: 370/503, 370/509, 370/510, 348/512, 348/513, 348/515, 348/516, 348/518, 348/521, 348/552
• US Class Current: 370/509
• CPC Class Codes: G11B 20/10 Digital recording or reproducing
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 69/22 Parsing or analysis of headers
• CPC Class Codes: H04N 21/2368 Multiplexing of audio and video streams
• CPC Class Codes: H04N 21/426 Internal components of the client ; Characteristics thereof H04N5/44 takes precedence
• CPC Class Codes: H04N 21/42607 for processing the incoming bitstream
• CPC Class Codes: H04N 21/4305 Synchronising client clock from received content stream, e.g. locking decoder clock with encoder clock, extraction of the PCR packets arrangements for synchronising receiver with transmitter by comparing receiver clock with transmitter clock H04L7/0012; arrangements for synchronising receiver with transmitter wherein the receiver takes measures against momentary loss of synchronisation H04L7/0083
• CPC Class Codes: H04N 21/4341 Demultiplexing of audio and video streams
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56
• CPC Class Codes: H04N 5/60 for the sound signals

Abstract

A system and method is provided for synchronizing the presentation of audio data with video data. Audio transport packets are received through a demultiplexer from a multimedia transport stream. A transport stream synchronization manager is used to lock a system time clock, local to the demultiplexer, to a program clock reference provided through the multimedia transport stream. Presentation time stamps are provided with the audio transport packets to indicate when decoded audio data is to be output. A packetized elementary stream synchronization manager maintains synchronization by adding or dropping audio packets from the audio transport packets. If the packetized elementary stream manager is unable to acquire synchronization it must defer synchronization back to the transport stream synchronization manager. Otherwise, processed audio packets are passed to an elementary stream synchronization manager that attempts to synchronize the delivery of audio data through a sample rate conversion of the audio data. If the elementary synchronization manager is unable to acquire synchronization, it defers synchronization to the packetized elementary stream synchronization manager.

Claims

1. A method comprising:

determining a synchronization state of an audio data relative to a system clock;

when the synchronization state is in a first state maintaining a current playback

when the synchronization state is in a second state making a first playback adjustment to the audio data, wherein the first playback adjustment includes performing a sample rate conversion by interpolation of one or more audio data samples of the audio data; and

when the synchronization state is in a third state making a second playback adjustment to the audio data, the second playback adjustment to provide a coarser playback adjustment than the first playback adjustment.

2. The method as in claim 1, further comprising:

when the synchronization state is in a fourth state initializing the system clock to a predefined value.

3. The method as in claim 2, wherein the predefined value is equal to a representation of a system clock associated with a source of data.

4. The method as in claim 3, wherein the representation of the system clock associated with the source of the audio data is a program counter clock associated with MPEG-type data.

5. The method as in claim 1, wherein the second playback adjustment includes adjustments to PES packets.

6. The method as in claim 5, wherein the second playback adjustment includes repeating PES packets.

7. The method as in claim 5, wherein the second playback adjustment includes dropping PES packets.

8. The method as in claim 1, wherein determining a synchronization state includes comparing a PTS value to an STC value.

9. The method as in claim 8, wherein a difference between the PTS value and the STC value is compared to a delta value.

10. The method as in claim 9, wherein a delta value associated with the first state indicates a time difference equivalent to playback of a single audio data sample.

11. The method as in claim 10, wherein the time to play a single audio sample is calculated by determining an audio data type.

12. The method as in claim 11, wherein determining the audio data type is based upon a data stream type.

13. The method as in claim 11, wherein determining the audio data type is based upon a data stream identifier.

14. The method as in claim 9, wherein a delta value associated with the second state indicates a range of time from the time required for playback of 2 audio samples to the time required for playback of 32 audio data samples.

15. The method as in claim 14, wherein the time to play an audio sample is calculated by determining an audio data type.

16. The method as in claim 15, wherein determining the audio data type is based upon a data stream type.

17. The method as in claim 15, wherein determining the audio data type is based upon a data stream identifier.

18. The method as in claim 9, wherein a delta value associated with the third state indicates a range of time from the time required for playback of 1 audio frame to the time required for playback of 3 audio data frames.

19. The method as in claim 18, wherein the time required for playing audio frames is calculated by determining an audio data type.

20. The method as in claim 19, wherein determining the audio data type is based upon a data stream type.

21. The method as in claim 19, wherein determining the audio data type is based upon a data stream identifier.

22. The method as in claim 9, wherein the delta value is variable.

23. The method of claim 1, wherein the first playback adjustment further comprises filtering of the one or more audio data samples.

24. A system comprising:

a data processor having an I/O buffer;

a first data processing component maintain playback of audio data samples according to a first state of synchronization;

a second data processing component to perform a first playback adjustment to maintain a second state of synchronization, wherein the first playback adjustment includes performing a sample rate conversion by interpolation of one or more of the audio data samples;

a third data processing component to perform a second playback adjustment to maintain a third state of synchronization, the second playback adjustment to provide a coarser playback adjustment than the first playback adjustment; and

a system time clock to track presentation times associated with the audio data.

25. The system as in claim 24, further including:

a fourth data processing component to initialize the system time clock to a predefined value to maintain a fourth state of synchronization.

26. The system as in claim 25, wherein the predefined value is equal to a representation of a system clock associated with a source of data.

27. The system as in claim 26, wherein the representation of the system clock associated with the source of the audio data is a program counter clock associated with MPEG-type data.

28. The system as in claim 24, wherein the second playback adjustment includes adjustments to PES packets.

29. The system as in claim 28, wherein the second playback adjustment includes repeating PES packets.

30. The system as in claim 28, wherein the second playback adjustment includes dropping PES packets.

31. The system as in claim 24, wherein operations within the data processing components are determined by comparing a PTS value to an STC value.

32. The system as in claim 31, wherein a difference between the PTS value and the STC value is compared to a delta value.

35. The system as in claim 32, wherein a delta value associated with the second state indicates a range of time from the time required for playback of 2 audio samples to the time required for playback of 32 audio data samples.

36. The system as in claim 32, wherein a delta value associated with the third state indicates a range of time from the time required for playback of 1 audio frame to the time required for playback of 3 audio frames.

37. The system as in claim 32, wherein the delta value is variable.

38. The system as in claim 24, further comprising memory operably coupled to the data processor to store code for the first, second and third data processing components.

39. The system as in claim 24, wherein the first, second and third data processing components are represented in hardware.

40. The system as in claim 24, wherein the first data processing component includes a demultiplexer.

41. The system as in claim 24, wherein the presentation time is determined through a presentation time stamp associated with a data stream.

42. The system as in claim 24, wherein the presentation time of the data packets are interpolated from the presentation time stamp associated with the data stream.

43. The method as in claim 24, wherein the second data processing component further processes data packets into data samples.

44. The system of claim 24, wherein the first playback adjustment further comprises filtering of the one or more audio data samples.

33. The system as in 32, wherein a delta value associated with the first state indicates a time difference equivalent to playback of a single audio data sample.

34. The system as in claim 33, wherein the time to play a single audio sample is calculated based on a determination of an audio data type.

45. A computer readable medium tangibly embodying a plurality of programs of instructions, the plurality of programs including:

a first data processing component to maintain playback of audio data samples according to a first state of synchronization,

a second data processing component to perform a first playback adjustment to maintain a second state of synchronization, wherein the first playback adjustment includes performing a sample rate conversion by interpolation of one or more of the audio data samples;

a third data processing component to perform a second playback adjustment to maintain a third state of synchronization, the second playback adjustment to provide a coarser playback adjustment than the first playback adjustment.

46. The computer readable medium as in claim 45, further including a fourth data processing component to initialize the system time clock to a predefined value to maintain a fourth state of synchronization.

47. The computer readable medium as in claim 45, wherein the third data processing component controls hardware used to process the data packets into data samples.

48. The computer readable medium as in claim 45, wherein the second data processing component controls hardware for processing the data samples.

49. The computer readable medium as in claim 45, wherein operations within the data processing components are determined by comparing a PTS value to an STC value.

50. The computer readable medium of claim 45, wherein the first playback adjustment further comprises filtering of the one or more audio data samples.

51. A method comprising:

receiving an MPEG-type transport stream;

demultiplexing the MPEG-type transport stream to generate transport packets;

synchronizing a system time clock to a program clock reference received through the MPEG-type transport stream;

determining if a PTS value associated with the transport packets is within a predefined value of the system time clock;

when the PTS value is within the predefined value, performing a sample rate conversion by interpolation of audio samples related to the transport packets; and

when the PTS value is not within the predefined value, adjusting PES packets related to the transport packets.

52. The method as in claim 51, wherein the transport packets are processed into the PES packets.

53. The method as in claim 52, wherein the PES packets are processed into the audio samples.

54. The method as in claim 51, wherein the predetermined value indicates a range of time from the time required for playback of 1 audio frame to the time required for playback of 3 audio data frames.

55. The method as in claim 54, wherein the time required for playing audio frames is calculated by determining an audio data type.

56. The method of claim 51, wherein the first playback adjustment further comprises filtering of the audio data samples.