Audio/video decoding system

• US 5,668,601 A
• Filed: 02/13/1995
• Issued: 09/16/1997
• Est. Priority Date: 02/17/1994
• Status: Expired due to Term

• Current Assignee: Godo Kaisha IP Bridge 1 (IP Bridge, Inc.)
• Original Assignee: Sanyo Electric Company Limited (Panasonic Holdings Corporation)
• Inventors: Okada, Shigeyuki, Yamauchi, Hideki
• Primary Examiner(s): Britton, Howard W.
• Application Number:
  US08/387,581
• Time in Patent Office: 946 Days
• Field of Search: 348/423, 370/100.1, 370/503
• US Class Current: 375/240.25
• CPC Class Codes: H04N 21/4305 Synchronising client clock from received content stream, e.g. locking decoder clock with encoder clock, extraction of the PCR packets arrangements for synchronising receiver with transmitter by comparing receiver clock with transmitter clock H04L7/0012; arrangements for synchronising receiver with transmitter wherein the receiver takes measures against momentary loss of synchronisation H04L7/0083
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4341 Demultiplexing of audio and video streams

Abstract

An audio/video decoding system capable of providing sufficient synchronization between an audio output and a video output. The decoding system has a parser, an audio decoder and a video decoder. The parser extracts an SCR (System Clock Reference), an audio time stamp and a video time stamp from a system stream, and demultiplexing the system stream into an audio stream and a video stream. The audio decoder computes an output timing for an audio output based on its decoding time, the SCR and the audio time stamp, and decodes the audio stream in accordance with the output timing. The video decoder computes an output timing for a video output based on its decoding time, the SCR and the video time stamp, and decodes the video stream in accordance with the output timing.

Claims

1. A Moving Picture Experts Group (MPEG) compliant audio/video decoding system for decoding a multiplexed system stream of audio and video streams having audio and video time stamps and a system clock reference (SCR) into synchronized audio output and video output from said decoding system, said system comprising:

a parser for extracting audio and video streams, audio and video time stamps, and said SCR from said system stream;

an audio stream processor responsive to the audio stream to decode said audio stream into audio data;

an audio controller responsive to said audio time stamp and SCR for controlling the output of audio data from said audio stream processor based on an audio decoding time of said audio stream processor, said SCR and said audio time stamp;

a video stream processor responsive to the video stream to decode said video stream into video data; and

a video controller responsive to video time stamp and SCR for controlling the output of video data from said video stream processor based on a video decoding time of said video stream processor, said SCR and said video time stamp.

2. A Moving Picture Experts Group (MPEG) compliant audio/video decoding system, responsive to a multiplexed system stream of audio and video streams having a system clock reference (SCR), an audio time stamp and a video time stamp, to provide decoded audio data and decoded video data as output from said decoding system, said system comprising:

a parser for extracting the SCR, audio time stamp and video time stamp from the system stream, said parser including a demultiplexer for demultiplexing the audio and video streams from said system stream, wherein said audio stream, video stream, SCR, audio time stamp and video time stamp are provided as outputs from said parser;

an audio stream processor responsive to the audio stream from said parser to decode said audio stream into audio data, said audio data is provided as output from said audio stream processor;

an audio controller, responsive to said SCR and said audio time stamp provided by said parser, for computing an audio decoding time based on an internal operational delay time of said audio stream processor, and for controlling the output of the audio data from said audio stream processor based on said audio decoding time, said SCR and said audio time stamp;

a video stream processor responsive to the video stream from said parser to decode said video stream into video data, said video data is provided as output from said video stream processor; and

a video controller, responsive to said SCR and said video time stamp provided by said parser, for computing a video decoding time based on an internal operational delay time of said video stream processor, and for controlling the output of the video data from said video stream processor based on said video decoding time, said SCR and said video time stamp.

3. The decoding system according to claim 2, wherein said audio stream processor includes:

an audio buffer for temporarily holding said audio stream output from said parser; and

an audio decode core circuit for decoding said audio stream held in said audio buffer; and

wherein said video stream processor includes;

a video buffer for temporarily holding said video stream output from said parser; and

a video decode core circuit for decoding said video stream held in said video buffer.

4. The decoding system according to claim 3, wherein the internal operational delay time of said audio stream processor is set as the sum of the internal delay time of said audio buffer and audio decode core circuit;

and

wherein the internal operational delay time of said video stream processor is set as the sum of the internal delay time of said video buffer and video decode core circuit.

5. A Moving Picture Experts Group (MPEG) compliant audio/video decoding system for decoding a multiplexed system stream of audio and video streams having audio and video time stamps and a system clock reference (SCR) into synchronized audio output and video output from said decoding system, said system comprising:

a parser for extracting audio and video streams, audio and video time stamps, and said SCR from said system stream;

an audio stream processor responsive to the audio stream to decode said audio stream into audio data;

an audio controller responsive to said audio time stamp and SCR for controlling the output of audio data from said audio stream processor based on, said SCR and said audio time stamp;

an video stream processor responsive to the video stream to decode said video stream into video data; and

a video controller responsive to video time stamp and audio time stamp for controlling the output of video data from said video stream processor based on a video decoding time of said video stream processor, an audio decoding time of said audio stream processor, said audio time stamp and said video time stamp.

6. A Moving Picture Experts Group (MPEG) compliant audio/video decoding system, responsive to a multiplexed system stream of audio and video streams with a system clock reference (SCR), an audio time stamp and a video time stamp, to provide decoded audio data and decoded video data synchronized with said audio data as outputs from said decoding system, said system comprising:

a parser for extracting the SCR, audio time stamp and video time stamp from the system stream, said parser including a demultiplexer for demultiplexing the audio and video streams from said system stream, wherein said audio stream, video stream, SCR, audio time stamp and video time stamp are provided as outputs from said parser;

an audio stream processor responsive to the audio stream from said parser to decode said audio stream into audio data, said audio data is provided as output from said audio stream processor;

an audio controller, responsive to said SCR and said audio time stamp provided by said parser for controlling the output of the audio data from said audio stream processor based on said SCR and said audio time stamp;

a time stamp generator, responsive to said audio time stamp from said parser, for computing an audio decoding time based on an internal operational delay time of said audio stream processor, and for generating a first time stamp based on the computed decoding time and said audio time stamp;

a video stream processor responsive to the video stream from said parser to decode said video stream into video data said video data is provided as output from said video stream processor; and

a video controller, responsive to said video time stamp provided from said parser and to the first time stamp generated by said time stamp generator, for computing a video decoding time based on an internal operational delay time of said video stream processor, and for controlling the output of the video data from said video stream processor based on said video time stamp said computed video decoding time and said first time stamp.

7. The decoding system according to claim 6, wherein said audio time stamp includes a presentation time stamp indicating an output timing of said audio output and a decoding time stamp indicating a decoding start timing of said audio stream;

and

said video time stamp includes a presentation time stamp indicating an output timing of said video output and a decoding time stamp indicating a decoding start timing of said video stream.

8. The decoding system according to claim 6, wherein said audio stream processor includes:

an audio register for receiving and temporarily holding said audio time stamp, and for providing said audio time stamp for said audio controller and said time stamp generator;

an audio buffer for temporarily holding said audio stream output from said parser; and

an audio decode core circuit for decoding said audio stream held in said audio buffer; and

wherein said video stream processor includes;

a video register for receiving and temporarily holding said video time stamp, and for providing said video time stamp for said video controller;

a video buffer for temporarily holding said video stream output from said parser; and

a video decode core circuit for decoding said video stream held in said video buffer.

9. The decoding system according to claim 8, wherein said time stamp generator computes a sum of an internal delay time of said audio buffer and an internal delay time of said audio decode core circuit, said sum being considered as a decoding time of said audio stream processor;

and

said video controller computes a sum of an internal delay time of said video buffer and an internal delay time of said video decode core circuit, said sum being considered as a decoding time of said video stream processor.

10. The decoding system according to claim 8, wherein said time stamp generator includes:

a delay time calculator for computing said internal delay time of said audio buffer; and

an audio subtracter for subtracting a sum of said internal delay time of said audio buffer and said internal delay time of said audio decode core circuit from said audio time stamp to produce said first time stamp.

11. The decoding system according to claim 10, wherein said time stamp generator includes:

a sampling frequency detector for detecting a sampling frequency from said audio stream held in said audio buffer and generating a clock signal corresponding to said sampling frequency; and

an adder for adding a primary first time stamp produced by said audio subtracter and said clock signal to produce a secondary first time stamp.

12. The decoding system according to claim 11, wherein said sampling frequency is 44.1 kHz.

13. The decoding system according to claim 8, wherein said video controller includes:

a mapping processor for mapping said video time stamp and a picture included in said video stream to obtain an internal operational delay time of said video buffer; and

a generator for generating a signal for controlling an output timing based on said internal delay time of said video decode core circuit, said video time stamp and said first time stamp in such a way as to cause said video decode core circuit to perform a skip operation or a repeat operation,

wherein said video decode core circuit abandons a picture included in said video stream held in said video buffer in accordance with a skip control signal and repeats video outputting of said picture in accordance with a repeat control signal.

14. The decoding system according to claim 13, further comprising a repeat determining circuit for determining if said repeat control signal generated by said generator is effective, and for generating a new repeat control signal for a repeat operation when said repeat control signal is effective.

15. The decoding system according to claim 14, wherein said repeat determining circuit includes:

a counter for counting a number of consecutive generations of said repeat control signal; and

a comparator for comparing a count value of said counter with a value set by an input device, and generating a new repeat control signal when said count value is greater than said set value,

said counter resetting said count value to zero when said comparator generates said new repeat control signal.

16. The decoding system according to claim 14, wherein said repeat determining circuit includes:

a counter for starting a counting operation when said repeat control signal is generated by a predetermined number of times; and

a comparator for comparing a count value of said counter with a value set by an input device, and generating a new repeat control signal when said count value is greater than said set value,

said counter resetting said count value to zero when said comparator generates said new repeat control signal.

17. The decoding system according to claim 13, wherein said generator generates a normal-operational control signal for controlling an output timing so as to cause said video decode core circuit to perform a normal operation;

and

said video decode core circuit decodes said video stream held in said video buffer in a normal manner in accordance with said normal-operation control signal.

18. The decoding system according to claim 17, further comprising:

a repeat determining circuit for determining if said repeat control signal generated by said generator is effective, and generating a new repeat control signal when said repeat control signal is effective; and

a skip determining circuit for determining if said skip control signal generated by said generator is effective, and generating a new skip control signal when said skip control signal is effective;

wherein said repeat determining circuit stops generating said new repeat control signal when said normal-operation control signal is generated by said generator; and

said skip determining circuit stops generating said new skip control signal when said normal-operation control signal is generated by said generator.

19. The decoding system according to claim 18, wherein said repeat determining circuit includes:

a first counter for counting a number of consecutive generations of said repeat control signal;

a first comparator for comparing a count value of said first counter with a value set by an input device, and generating a new repeat control signal when said count value is greater than said set value;

a second comparator for generating a count start signal when said count value of said first counter is greater than a predetermined value;

a second counter for starting a counting operation in response to said count start signal; and

a third comparator for comparing a count value of said second counter with a value set by an input device, and generating a new repeat control signal when said count value is greater than said set value,

said first counter resetting said count value to zero when said generator generates said normal-operation control signal or one of said first and third comparators generates said new repeat control signal during counting,

said second counter resetting said count value to zero when said third comparator generates said new repeat control signal.

20. The decoding system according to claim 18, wherein said skip determining circuit includes:

a first counter for counting a number of consecutive generations of said skip control signal;

a first comparator for comparing a count value of said first counter with a value set by an input device, and generating a new skip control signal when said count value is greater than said set value;

a second comparator for generating a count start signal when said count value of said first counter is greater than a predetermined value;

a second counter for starting a counting operation in response to said count start signal; and

a third comparator for comparing a count value of said second counter with a value set by an input device, and generating a new skip control signal when said count value is greater than said set value,

said first counter resetting said count value to zero when said generator generates said normal-operation control signal or one of said first and third comparators generates said new skip control signal during counting,

said second counter resetting said count value to zero when said third comparator generates said new skip control signal.

21. The decoding system according to claim 13, wherein said mapping processor includes:

a write address detector for detecting an address of a video time stamp on an assumption that said video stream held in said video buffer is affixed with said video time stamp, said address being temporarily held in association with said video time stamp in said video register;

a read address detector for detecting and address of said video stream read from said video buffer;

a picture header detector for extracting a picture header from a head of each picture in said video stream held in said video buffer, and for detecting a picture type defined in said picture header;

a first comparator for comparing said address of said video stream read from said video buffer with said address associated with said video time stamp and read from said video register, and determining if both addresses coincide with each other; and

a mapping circuit for mapping said video time stamp and a picture based on a result of determination by said first comparator and a result of detection by said picture header detector.

22. The decoding system according to claim 21, wherein said generator includes:

a first register for temporarily holding said video time stamp and exchanging a video time stamp associated with an I picture or a P picture with a video time stamp associated with a B picture, based on said picture type detected by said picture header detector;

a video subtracter for subtracting said internal delay time of said video decode core circuit and said video time stamp held in said first register from said first time stamp to produce a second time stamp;

a second comparator for comparing a first value set by an input device with said second time stamp; and

a sync determining circuit for generating said control signals after execution of mapping by said mapping circuit, based on a result of comparison by said second comparator.

23. The decoding system according to claim 22, wherein said first value is set greater than a half of a reproduction time of one picture.

24. The decoding system according to claim 22, wherein said video subtracter includes:

a first video subtracter subtracting a sum of said internal delay time of said video decode core circuit and a second value set by an input device from said video time stamp held in said first register, thereby producing a primary second time stamp; and

a second video subtracter for subtracting said primary second time stamp from said first time stamp to produce a secondary second time stamp.

25. The decoding system according to claim 13, further comprising a skip determining circuit for determining if said skip control signal generated by said generator is effective, and generating a new skip control signal when said skip control signal is effective.

26. The decoding system according to claim 25, wherein said skip determining circuit includes:

a counter for counting a number of consecutive generations of said skip control signal; and

a comparator for comparing a count value of said counter with a value set by an input device, and generating a new skip control signal when said count value is greater than said set value,

said counter resetting said count value to zero when said comparator generates said new skip control signal.

27. The decoding system according to claim 25, wherein said skip determining circuit includes:

a counter for starting a counting operation when said skip control signal is generated by a predetermined number of times; and

a comparator for comparing a count value of said counter with a value set by an input device, and generating a new skip control signal when said count value is greater than said set value,

said counter resetting said count value to zero when said comparator generates said new skip control signal.

28. The decoding system according to claim 25, wherein said mapping processor includes a picture header detector for extracting a picture header from a head of each picture in said video stream written in said video buffer and detecting a picture type defined in said picture header;

and

said skip determining circuit includes a priority processing circuit for generating a skip control signal to skip a B picture with a priority over an I picture and a P picture based on said new skip control signal and said picture type detected by said picture header detector.

29. A Moving Picture Experts Group (MPEG) compliant audio/video decoding system, responsive to a multiplexed system stream of audio and video streams with a system clock reference (SCR), an audio time stamp and a video time stamp, to provide decoded audio data and decoded video data synchronized with said audio data as outputs from said decoding system, said system comprising:

a parser for extracting the SCR, audio time stamp and video time stamp from the system stream, said parser including a demutiplexer for demultiplexing the audio and video streams from said system stream, wherein said audio stream, video stream, SCR, audio time stamp and video time stamp are provided as outputs from said parser;

an audio stream processor responsive to the audio stream from said parser to decode said audio stream into audio data, said audio data is provided as output from said audio stream processor;

an audio controller, responsive to said SCR and said audio time stamp provided by said parser, for controlling the output of the audio data from said audio stream processor based on said SCR and said audio time stamp;

a video stream processor responsive to the video stream from said parser to decode said video stream into video data, said video data being provided as output from said video stream processor; and

a video controller, responsive to said audio time stamp and said video time stamp provided by said parser, for computing a video decoding time based on an internal operational delay time of said video stream processor, and for controlling the output of the video data from said video stream processor based on said video decoding time, said audio time stamp and said video time stamp.

30. A Moving Picture Experts Group (MPEG) compliant audio decoder for use in an audio/video decoding system, said audio decoder being responsive to an audio stream, a system clock reference (SCR) and audio time stamp to produce decoded audio data, said audio decoder comprising:

an audio stream processor responsive to the audio stream to decode said audio stream into audio data, said audio data is provided as output from said audio stream processor; and

an audio controller, responsive to said SCR and said audio time stamp, for computing an audio decoding time based on an internal operational delay time of said audio stream processor, and for controlling the output of the audio data from said audio stream processor based on said audio decoding time, said SCR and said audio time stamp.

31. The audio decoder according to claim 30, wherein said audio stream processor includes:

a buffer for temporarily holding said audio stream; and

a decode core circuit for decoding said audio stream held in said buffer.

32. The audio decoder according to claim 31, wherein said audio controller computes a sum of an internal delay time of said buffer and an internal delay time of said decode core circuit, said sum being considered as a decoding time of said audio stream processor.

33. A Moving Picture Experts Group (MPEG) compliant video decoder for use in an audio/video decoding system, said video decoder being responsive to a video stream, a system clock reference (SCR) and a video time stamp to produce decoded video data, the video decoder comprising:

a video stream processor responsive to the video stream to decode said video stream into video data, said video data is provided as output from said video stream processor; and

a video controller, responsive to said SCR and said video time stamp, for computing an video decoding time based on an internal operational delay time of said video stream processor, and for controlling the output of the video data from said video stream processor based on said video decoding time, said SCR and said video time stamp.

34. The video decoder according to claim 33, wherein said video stream processor includes:

a buffer for temporarily holding said video stream; and

a decode core circuit for decoding said video stream held in said buffer.

35. The video decoder according to claim 34, wherein said video controller computes a sum of an internal delay time of said buffer and an internal delay time of said decode core circuit, said sum being considered as a decoding time of said video stream processor.

36. A Moving Picture Experts Group (MPEG) compliant system decoder for use in an audio/video decoding system, said system decoder being responsive to a multiplexed system stream of audio and video streams having system clock reference (SCR), audio time stamp and video time stamp, to provide an audio decoder with said audio stream and to provide a video decoder with said video stream, the system decoder comprising:

a parser for extracting the SCR, audio time stamp and video time stamp from the system stream, said parser including a demultiplexer for demultiplexing the audio and video streams from said system stream, wherein said audio stream, video stream, SCR, audio time stamp and video time stamp are provided as outputs from said parser;

an audio controller, responsive to said SCR and audio time stamp provided by said parser, for computing an audio decoding time based on an internal operational delay time of said audio decoder, and for controlling said audio decoder based on said audio decoding time, said SCR and said audio time stamp; and

an video controller, responsive to said SCR and video time stamp provided by said parser, for computing a video decoding time based on an internal operational delay time of said video decoder, and for controlling said video decoder based on said video decoding time, said SCR and said video time stamp.

37. A method for synchronizing the output of audio and video data from an audio/video decoding system compliant with Moving Picture Experts Group (MPEG) standards, said audio/video decoding system operating to decode an system stream into audio data and video data, said system stream including audio and video streams multiplexed with a System Clock Reference (SCR), an audio time stamp and a video time stamp, the method comprising the steps of:

extracting said SCR, said audio time stamp and said video time stamp from said system stream;

demultiplexing the system stream into an audio stream and video stream;

computing an audio decoding time, as the time needed to decode audio data from said audio stream;

timing the output of said audio data from said decoding system based on the audio decoding time, said SCR and said audio time stamp;

computing an video decoding time, as the time needed to decode video data from said video stream; and

timing the output of said video data from said decoder based on the video decoding time, said SCR and said video time stamp.

38. A method for synchronizing the output of audio and video data from an audio/video decoding system compliant with Moving Picture Experts Group (MPEG) standards, said audio/video decoding system operating to decode a system stream into audio data and video data, said system stream including audio and video streams multiplexed with a System Clock Reference (SCR), an audio time stamp and a video time stamp, the method comprising the steps of:

extracting said SCR, said audio time stamp and said video time stamp from said system stream;

demultiplexing the system stream into an audio stream and video streams;

computing an audio decoding time, as the time needed to decode audio data from said audio stream;

timing the output of said audio data from said decoding system based on said SCR and said audio time stamp;

generating a first time stamp based on said audio decoding time and said audio time stamp;

computing an video decoding time, as the time needed to decode video data from said video stream; and

timing the output of said video data from said decoding system based on the video decoding time, said video time stamp and said first time stamp.

39. The method according to claim 38, wherein said step of generating said first time stamp includes the steps of:

generating a primary first time stamp based on said computed decoding time and said audio time stamp;

detecting a sampling frequency from said audio stream;

generating a clock signal corresponding to said detected sampling frequency; and

adding said clock signal to said primary first time stamp to produce a secondary first time stamp.

40. The method according to claim 38, further comprising the step of receiving and temporarily holding said video stream before computing said video decoding time;

and

wherein said step for computing said video decoding time includes the steps of;

mapping said video time stamp and a picture included in said video stream to obtain a delay time originated from holding of said video stream; and

wherein said timing the output of said video data include the step of;

generating a control signal for one of skip and repeat operations based on the video decoding time said video stream, said video time stamp and said first time stamp.

41. The method according to claim 40, wherein said step of generating said control signal includes the steps of:

generating a second time stamp based on said video decoding time, said video time stamp and said first time stamp; and

comparing said generated second time stamp with a previously set value and generating a skip control signal or a repeat control signal in accordance with a result of comparison.

42. The method according to claim 40, further comprising the steps of:

determining if, after generation of said skip control signal, there is a consecutiveness in said generation of said skip control signal; and

generating a new skip control signal when it is determined that there is a consecutiveness in said generation of said skip control signal.

43. The method according to claim 40, further comprising the steps of:

determining if, after generation of said repeat control signal, there is a consecutiveness in said generation of said repeat control signal; and

generating a new repeat control signal when it is determined that there is a consecutiveness in said generation of said repeat control signal.

44. The method according to claim 40, further comprising the steps of:

counting a time after generation of said skip control signal, for a predetermined time; and

generating a new skip control signal when said predetermined time passes.

45. The method according to claim 40, further comprising the steps of:

counting a time after generation of said repeat control signal, for a predetermined time; and

generating a new repeat control signal when said predetermined time passes.

46. The method according to claim 40, further comprising the steps of:

determining if, after generation of said skip control signal, there is a consecutiveness in said generation of said skip control signal;

generating a new skip control signal when it is determined that there is a consecutiveness in said generation of said skip control signal;

counting a time after generation of said skip control signal, for a predetermined time; and

generating a new skip control signal when said predetermined time passes.

47. The method according to claim 40, further comprising the steps of:

determining if, after generation of said repeat control signal, there is a consecutiveness in said generation of said repeat control signal;

generating a new repeat control signal when it is determined that there is a consecutiveness in said generation of said repeat control signal;

counting a time after generation of said repeat control signal, for a predetermined time; and

generating a new repeat control signal when said predetermined time passes.

48. The method according to claim 47, further comprising the steps of:

extracting a picture header from a head of each picture in said video stream;

detecting a picture type defined in said picture header; and

generating a skip control signal to skip a B picture with a priority over an I picture and a P picture based on said new skip control signal and said picture type.

49. A method for synchronizing the output of audio and video data from an audio/video decoding system compliant with Moving Picture Experts Group (MPEG) standards, said audio/video decoding system operating to decode a system stream into audio data and video data, said system stream including audio and video streams multiplexed with a System Clock Reference (SCR), an audio time stamp and a video time stamp, the method comprising the steps of:

extracting said SCR, said audio time stamp and said video time stamp from said system stream;

demultiplexing the system stream into an audio stream and video streams;

computing an audio decoding time, as the time needed to decode audio data from said audio stream;

timing the output of said audio data from said decoding system based on the audio decoding time, said SCR and said audio time stamp;

computing an video decoding time, as the time needed to decode video data from said video stream; and

timing the output of said video signal from said decoder based on the video decoding time, said video time stamp and said audio time stamp.