Programmable audio-video synchronization method and apparatus for multimedia systems

• US 5,594,660 A
• Filed: 09/30/1994
• Issued: 01/14/1997
• Est. Priority Date: 09/30/1994
• Status: Expired due to Term

• Current Assignee: Cirrus Logic Incorporated
• Original Assignee: Cirrus Logic Incorporated
• Inventors: Sung, Chih-Ta, Chan, Tzoyao, Chang, Richard, Rosenau, Mark A., Ort, Jeffrey G., Daum, Daniel T., Sun, Yuanyuan
• Primary Examiner(s): Ramirez, Ellis B.
• Application Number:
  US08/316,015
• Time in Patent Office: 837 Days
• Field of Search: 364/514 R, 364/514 C, 395/153, 348/510, 348/512, 348/515, 348/423, 348/465, 370/100.1, 375/362, 375/375
• US Class Current: 715/203
• CPC Class Codes: G09G 5/393 Arrangements for updating the contents of the bit-mapped memory
• CPC Class Codes: G11B 20/00007 Time or data compression or expansion audio compression based on psychoacoustics G10L19/00; data processing for reproducing audio data at different playback speeds G10L21/04; video compression H04N19/00; data compression per se H03M7/30
• CPC Class Codes: G11B 20/10527 Audio or video recording; Data buffering arrangements G11B20/12 - G11B20/18 take precedence
• CPC Class Codes: G11B 2220/213 Read-only discs
• CPC Class Codes: G11B 2220/2545 CDs
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/3027 used signal is digitally coded
• CPC Class Codes: G11B 27/3036 Time code signal
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/42 characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation H04N19/635 takes precedence
• CPC Class Codes: H04N 19/423 characterised by memory arrangements H04N19/433 takes precedence
• CPC Class Codes: H04N 19/427 Display on the fly, e.g. simultaneous writing to and reading from decoding memory
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 21/4143 embedded in a Personal Computer [PC]
• CPC Class Codes: H04N 21/4305 Synchronising client clock from received content stream, e.g. locking decoder clock with encoder clock, extraction of the PCR packets arrangements for synchronising receiver with transmitter by comparing receiver clock with transmitter clock H04L7/0012; arrangements for synchronising receiver with transmitter wherein the receiver takes measures against momentary loss of synchronisation H04L7/0083
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 5/775 between a recording apparatus and a television receiver
• CPC Class Codes: H04N 5/85 on discs or drums
• CPC Class Codes: H04N 9/8042 : involving data reduction
• CPC Class Codes: H04N 9/8063 : using time division multiplex of the PCM audio and PCM video signals
• CPC Class Codes: H04N 9/8216 : using time division multiplex
• CPC Class Codes: H04N 9/87 : Regeneration of colour television signals H04N9/80 takes precedence

Abstract

A multimedia system including an audio/video decoder/decompresser for decoding/decompressing a compressed encoded audio/video data stream to generate video images for display and audio signals for audible reproduction. The multimedia system includes an integrated system and video decoder with an audio/video synchronization circuit for substancially synchronizing the display of video images with audio playback. A method is described for detecting when the playback of audio and the display of video images are out of synchronization. The circuitry includes three programmable registers, a finite state machine and one full adder using an audio presentation time stamp and the video presentation time stamp. The method uses a rounded programmable bias value which is compared with the difference between the video presentation time stamp and the audio presentation time stamps. The difference may included the latency delay of the audio and video display devices to more properly reflect the synchronization status between audio and video. The circuit area used by the circuitry has been minimized by selective rounding or truncation of the audio presentation time stamp and the video presentation time stamp without sacrificing the accuracy or speed of determining the synchronization between audio and video.

Claims

1. A multimedia system having an audio/video decoding/decompressing circuit for decoding an encoded/compressed data stream, the audio/video decoding/decompressing circuit comprising:

a parser for separating the encoded/compressed data stream into encoded/compressed audio data, encoded/compressed video data, an audio time value, and a video time value, wherein each of the audio time value and the video time value comprise a time stamp for the encoded/compressed audio data and the encoded/compressed video data respectively;

a memory having addressable storage, coupled to the parser, addressing addressable storage to selectively store and read the encoded/compressed audio data, the encoded/compressed video data, or decoded/decompressed video data to and from addressable storage;

a first decoder, coupled to the memory, for decoding the encoded/compressed video data and generating the decoded/decompressed video data; and

a synchronizer, coupled to the first decoder and the memory, for comparing the video time value with the audio time value and when the comparison exceeds a predetermined threshold, substantially synchronizing the generation of the decoded/decompressed video data by the first decoder with the addressing of addressable storage by the memory.

2. The audio/video decoding circuit of claim 1 wherein the synchronizer determines that the video time value exceeds the audio time value by the predetermined threshold, and

a sequence of addressing addressable storage containing decoded/decompressed video data by the memory is modified by repeating a sequence of addresses to substantially synchronize the generation of the decoded/decompressed video data by the first decoder with the addressing of addressable storage by the memory.

3. The audio/video decoding circuit of claim 1 wherein the synchronizer determines that the audio time value exceeds the video time value by the predetermined threshold, and

a sequence of addressing addressable storage containing encoded/compressed video data by the memory is modified by skipping a sequence of addresses to substantially synchronize the generation of the decoded/decompressed video data by the first decoder with the addressing of addressable storage by the memory.

4. The audio/video decoding circuit of claim 1 further comprising a second decoder, coupled to the memory, for decoding the encoded/compressed audio data into decoded/decompressed audio data.

5. The audio/video decoding circuit of claim 1 further comprising a graphics controller, coupled to the first decoder and a video monitor, for displaying the decoded/decompressed video data as an image on the video monitor.

6. The audio/video decoding circuit of claim 1 further comprising a graphics controller, coupled to the first decoder and a digital display, for displaying the decoded/decompressed video data as an image on the digital display.

7. The audio/video decoding circuit of claim 1 further comprising a television signal encoder, coupled to the first decoder and a television display, for generating television data from the decoded/decompressed video data and displaying the television data as an image on the television display.

8. The audio/video decoding circuit of claim 4 further comprising a digital to analog converter, coupled to the second decoder for generating an analog audio signal from the decoded/decompressed audio data.

9. The audio/video decoding circuit of claim 8 further comprising an audio amplifier, coupled to the digital to analog converter, for amplifying the magnitude of the analog audio signal.

10. A method for determining if synchronization of audio playback having a plurality of compressed and decompressed audio data packets, with video display images having a plurality of compressed and decompressed video data frames, is required, the method for determining comprising the steps of:

determining a time difference between receiving an audio time value and a video time value within a data stream of interspersed compressed audio data packets and compressed video data frames;

adding the time difference to the audio time value to generate a normalized audio time value; and

comparing the normalized audio time value with the video time value to determine whether the video display is lagging the audio playback and requires synchronization.

11. The method of claim 10 further comprising the step of:

skipping M compressed video data frames to substantially synchronize a compressed video data frame with a compressed audio data packet when it is determined in the first comparing step that the video display is lagging the audio playback by more than a lag threshold.

12. The method of claim 11 wherein the lag threshold is three frames of a video frame display time and the value of M is three.

13. The method of claim 11 wherein the lag threshold is one frame of a video frame display time and the value of M is one.

14. The method of claim 11 wherein the lag threshold is one half of a video frame display time and the value of M is one.

15. The method of claim 11 wherein the lag threshold is one half of a video frame display time and the value of M is one half.

16. The method of claim 10 further comprising the step of:

comparing the normalized audio time value with the video time value to determine whether the video display is leading the audio playback and requires synchronization.

17. The method of claim 16 further comprising the step of:

repeating N decompressed video data frames to substantially synchronize a decompressed video data frame with a decompressed audio data packet when it is determined in the second comparing step that the video display is leading the audio playback by more than a lead threshold.

18. The method of claim 17 wherein the lead threshold is three frames of a video frame display time and the value of N is three.

19. The method of claim 17 wherein the lead threshold is one frame of a video frame display time and the value of N is one.

20. The method of claim 17 wherein the lead threshold is one half of a video frame display time and the value of N is one.

21. The method of claim 17 wherein the lead threshold is one half of a video frame display time and the value of N is one half.

22. A method for determining if synchronization of video display images having a plurality of compressed and decompressed video data frames, with audio playback having a plurality of compressed and decompressed audio data packets, is required, the method for determining comprising the steps of:

determining a time difference between receiving a video time value and an audio time value within a data stream of interspersed compressed video data frames and compressed audio data packets;

adding the time difference to the video time value to generate a normalized video time value; and

comparing the normalized video time value with the audio time value to determine whether the video display is lagging the audio playback and requires synchronization.

23. The method of claim 22 further comprising the step of:

skipping M compressed video data frames to substantially synchronize a compressed video data frame with a compressed audio data packet when it is determined in the first comparing step that the video display is lagging the audio playback by more than a lag threshold.

24. The method of claim 23 wherein the lag threshold is two and one half frames of a video frame display time and the value of M is three.

25. The method of claim 23 wherein the lag threshold is one frame of a video frame display time and the value of M is one.

26. The method of claim 23 wherein the lag threshold is one half of a video frame display time and the value of M is one.

27. The method of claim 23 wherein the lag threshold is one half of a video frame display time and the value of M is one half.

28. The method of claim 22 further comprising the step of:

comparing the normalized video time value with the audio time value to determine whether the video display is leading the audio playback and requires synchronization.

29. The method of claim 28 further comprising the step of:

repeating N decompressed video data frames to substantially synchronize a decompressed video data frame with a decompressed audio data packet when it is determined in the second comparing step that the video display is leading the audio playback by more than a lead threshold.

30. The method of claim 29 wherein the lead threshold is three frames of a video frame display time and the value of N is three.

31. The method of claim 29 wherein the lead threshold is one frame of a video frame display time and the value of N is one.

32. The method of claim 29 wherein the lead threshold is one half of a video frame display time and the value of N is one.

33. The method of claim 29 wherein the lead threshold is one half of a video frame display time and the value of N is one half.

34. An audio/video synchronization circuit for substantially synchronizing video display images having a plurality of compressed and decompressed video data frames, with audio playback having a plurality of compressed and decompressed audio data packets, the audio/video synchronization circuit comprising:

a counter initialized to an audio time value upon receiving the audio time value within an audio/video encoded data stream and incremented by a clock to generate a normalized audio time value, the audio time value comprising a time stamp;

a memory means for storing a video time value upon receiving the video time value within the audio/video encoded data stream, the video time value also comprising a time stamp; and

a subtracter coupled to the memory means,

for subtracting the video time value stored in said memory means from the normalized audio time value to generate a time drift value,

for comparing the time drift value with a positive lag threshold to determine whether the video display is lagging the audio playback by the lag threshold and when exceeding the lag threshold skipping M compressed video data frames to substantially synchronize a compressed video data frame with a compressed audio data packet, and

for comparing the time drift value with the negative value of a lead threshold to determine whether the video display is leading the audio playback by the lead threshold and when exceeding the lead threshold repeating N decompressed video data frames to substantially synchronize a decompressed video data frame with a decompressed audio data packet.

35. The audio/video synchronization circuit of claim 34 wherein the lag threshold and lead threshold is three frames of a video frame display time and the value for M and N is three.

36. The audio/video synchronization circuit of claim 34 wherein the lag threshold and lead threshold is one frame of a video frame display time and the value for M and N is one.

37. The audio/video synchronization circuit of claim 34 wherein the lag threshold and lead threshold is one half of a video frame display time and the value for M and N is one.

38. The audio/video synchronization circuit of claim 34 wherein the lag threshold and lead threshold is one half of a video frame display time and the value for M and N is one half.

39. An audio/video synchronization circuit for substantially synchronizing video display images having a plurality of compressed and decompressed video data frames with audio playback having a plurality of compressed and decompressed audio data packets, the audio/video synchronization circuit comprising:

a frequency divider for selectively dividing the frequency of a clock in response to a display frame rate to generate a divided frequency clock;

a counter, coupled to the frequency divider, initialized to an audio time value upon receiving the audio time value within an audio/video encoded data stream, and incremented by the divided frequency clock to generate a normalized audio time value., wherein the audio time value comprises a time stamp;

a first memory means for storing a video time value upon receiving the video time value within the audio/video encoded data stream, wherein the video time value also comprises a time stamp;

a second memory means for storing a threshold value; and

an arithmetic means selectively coupled to the first memory means, the second memory means, and the counter, for selectively adding or subtracting the threshold value and the video time value to generate a first result and selectively adding or subtracting the normalized audio time value and the first result to generate a second result, wherein the second result indicates that the video display lags the audio playback by more than the threshold value; and

a synchronizer for causing M compressed video data frames to be skipped in order to substantially synchronize a decompressed video data frame with a decompressed audio data packet.

40. The audio/video synchronization circuit of claim 39 wherein

the audio time value and video time value have selective bits truncated, such that the audio time value and video time value are N bits wide to represent a narrower range of values,

the frequency divider is further responsive to the selected bits of the audio time value and video time value being truncated, and

the counter and arithmetic means are N+1 bits wide.

41. The audio/video synchronization circuit of claim 39 wherein the type of compressed video data frames that are caused to be skipped are MPEG encoded video frames within the audio/video encoded data stream.

42. The audio/video synchronization circuit of claim 39 wherein the type of compressed video data frames that are skipped is a B-type MPEG encoded video frame or a P-type MPEG encoded video frame which immediately precedes an I-type MPEG encoded video frame within the audio/video encoded data stream.

43. The audio/video synchronization circuit of claim 39 wherein the type of compressed video data frames that are desirable to skip is a B-type MPEG encoded video frame or a P-type MPEG encoded video frame which immediately precedes an I-type MPEG encoded video frame within the audio/video encoded data stream, and

when the desirable compressed video data frames that are desirable to skip are not contained within the audio/video encoded data stream for a predetermined period of time, the audio/video synchronization circuit generates an interrupt.

44. The audio/video synchronization circuit of claim 39 wherein the second result indicates that the video display lags the audio playback by more than the threshold value,

the audio video synchronization circuit signals to skip M compressed video data frames of a B-type MPEG encoded video frame or a P-type MPEG encoded video frame which immediately precedes an I-type MPEG encoded video frame within the audio/video encoded data stream to substantially synchronize a decompressed video data frame with a decompressed audio data packet, and

when B-type MPEG encoded video frames and P-type MPEG encoded video frames which immediately precede an I-type MPEG encoded video frame to be skipped are unavailable for a predetermined period of time within the audio/video encoded data stream,

the audio video synchronization circuit causes a delay of D compressed audio data packets.

45. The audio/video synchronization circuit of claim 39 wherein the threshold is three frames of a video frame display time and M is three.

46. The audio/video synchronization circuit of claim 39 wherein the threshold is one frame of a video frame display time and M is one.

47. The audio/video synchronization circuit of claim 39 wherein the threshold is one half of a video frame display time and M is one.

48. The audio/video synchronization circuit of claim 39 wherein the threshold is one half of a video frame display time and M is one half.

49. The audio/video synchronization circuit of claim 39 wherein the second result indicates that the video display leads the audio playback.

50. The audio/video synchronization circuit of claim 49 wherein the second result indicates that the video display leads the audio playback by more than the threshold value, and

the audio video synchronization circuit causes N decompressed video data frames to be repeated in order to substantially synchronize a decompressed video data frame with a decompressed audio data packet.

51. The audio/video synchronization circuit of claim 50 wherein the threshold is three frames of a video frame display time and N is three.

52. The audio/video synchronization circuit of claim 50 wherein the threshold is one frame of a video frame display time and N is one.

53. The audio/video synchronization circuit of claim 50 wherein the threshold is one half of a video frame display time and N is one.

54. The audio/video synchronization circuit of claim 50 wherein the threshold is one half of a video frame display time and N is one half.

55. An audio/video synchronization circuit for substantially synchronizing video display images having a plurality of compressed and decompressed video data frames with audio playback having a plurality of compressed and decompressed audio data packets, the audio/video synchronization circuit comprising:

a frequency divider for selectively dividing the frequency of a clock in response to a display frame rate to generate a divided frequency clock;

a counter, coupled to the frequency divider, initialized to a video time value upon receiving the video time value within an audio/video encoded data stream, and incremented by the divided frequency clock to generate a normalized video time value, wherein the video time value comprises a time stamp;

a first memory means for storing an audio time value upon receiving the audio time value within the audio/video encoded data stream, wherein the audio time value also comprises a time stamp;

a second memory means for storing a threshold value; and

an arithmetic means selectively coupled to the first memory means, the second memory means, and the counter, for selectively adding or subtracting the threshold value and the audio time value to generate a first result and selectively adding or subtracting the normalized video time value and the first result to generate a second result, wherein the second result indicates that the video display lags the audio playback by more than the threshold value; and

a synchronization circuit for causing M compressed video data frames to be skipped in order to substantially synchronize a decompressed video data frame with a decompressed audio data packet.

56. The audio/video synchronization circuit of claim 55 wherein the audio time value and video time value have selective bits truncated, such that the audio time value and video time value are N bits wide to represent a narrower range of values,

the frequency divider is further responsive to the selected bits of the audio time value and video time value being truncated, and

the counter and arithmetic means are N+1 bits wide.

57. The audio/video synchronization circuit of claim 55 wherein the threshold is three frames of a video frame display time and M is three.

58. The audio/video synchronization circuit of claim 55 wherein the threshold is one frame of a video frame display time and M is one.

59. The audio/video synchronization circuit of claim 55 wherein the threshold is one half of a video frame display time and M is one.

60. The audio/video synchronization circuit of claim 55 wherein the threshold is one half of a video frame display time and M is one half.

61. The audio/video synchronization circuit of claim 55 wherein the second result indicates that the video display leads the audio playback.

62. The audio/video synchronization circuit of claim 61 wherein the second result indicates that the video display leads the audio playback by more than the threshold value, and

the audio video synchronization circuit causes N decompressed video data frames to be repeated in order to substantially synchronize a decompressed video data frame with a decompressed audio data packet.

63. The audio/video synchronization circuit of claim 62 wherein the threshold is three frames of a video frame display time and N is three.

64. The audio/video synchronization circuit of claim 62 wherein the threshold is one frame of a video frame display time and N is one.

65. The audio/video synchronization circuit of claim 62 wherein the threshold is one half of a video frame display time and N is one.

66. The audio/video synchronization circuit of claim 62 wherein the threshold is one half of a video frame display time and N is one half.

67. A multimedia system for reading, decoding, and decompressing encoded audio/video data into video data for display on a plurality of display types and into audio data for reproduction on a plurality of sound reproduction devices, the multimedia system comprising:

a reader, receiving the encoded audio/video data, for reading the encoded audio/video data and generating a first encoded audio/video bit stream;

a first memory for storing the first encoded audio/video bit stream;

a processor, coupled to the reader, the first memory, and a system bus, for receiving the first encoded audio/video bit stream and writing first encoded audio/video bit stream into and a first memory, and reading the first encoded audio/video bit stream from the first memory being stored therein and outputing the first encoded audio/video bit stream onto the bus; and

an audio/video decoding circuit, coupled to the system bus, for receiving the first encoded audio/video bit stream and for decoding and decompressing the encoded audio/video data into video data and audio data, the audio/video decoding circuit comprising;

a parser for separating the encoded data stream into encoded audio data, encoded video data, an audio time value, and a video time value, the audio time value and the video time value comprising a time stamp;

a second memory having addressable storage, coupled to the parser, coupled to the parser, addressing addressable storage to selectively store and read the encoded audio data, the encoded video data, or decoded video data to and from addressable storage;

a first decoder, coupled to the second memory, for decoding and decompressing encoded video data and generating decompressed video data in response to the video time value;

a synchronizing circuit, coupled to the first decoder and the second memory, for comparing the video time value with the audio time value and when the comparison exceeds a predetermined threshold, substantially synchronizing the generation of the decompressed video data by the first decoder with the addressing of addressable storage by the second memory; and

a second decoder, coupled to the second memory, for decoding and decompressing encoded audio data to generate decompressed audio data.

68. The multimedia system of claim 67 wherein the synchronizer determines that the video time value exceeds the audio time value by the predetermined threshold, and

a sequence of addressing addressable storage containing decoded/decompressed video data by the memory is modified by repeating a sequence of addresses to substantially synchronize the generation of the decoded/decompressed video data by the first decoder with the addressing of addressable storage by the memory.

69. The multimedia system of claim 67 wherein the synchronizer determines that the audio time value exceeds the video time value by the predetermined threshold, and

a sequence of addressing addressable storage containing encoded video data by the memory is modified by skipping a sequence of addresses to substantially synchronize the generation of the decoded/decompressed video data by the first decoder with the addressing of addressable storage by the memory.

70. The multimedia system of claim 67 further comprising a graphics controller, coupled to the first decoder and a video monitor, for displaying the decoded/decompressed video data as an image on the video monitor.

71. The multimedia system of claim 67 further comprising a graphics controller, coupled to the first decoder and a digital display, for displaying the decoded/decompressed video data as an image on the digital display.

72. The multimedia system of claim 67 further comprising a television signal encoder, coupled to the first decoder and a television display, for generating television data from the decoded/decompressed video data and displaying the television data as an image on the television display.

73. The multimedia system of claim 67 further comprising a digital to analog converter, coupled to the second decoder for generating an analog audio signal from the decoded/decompressed audio data.

74. The multimedia system of claim 67 further comprising an audio amplifier, coupled to the digital to analog converter, for amplifying the magnitude of the analog audio signal.