Maintaining synchronization of streaming audio and video using internet protocol

• US 20050281255A1
• Filed: 06/18/2004
• Published: 12/22/2005
• Est. Priority Date: 06/18/2004
• Status: Active Grant

• Current Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Original Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Inventors: Richards, Martin John, Davies, Trevor, Lakshminarayanan, Gopi, Couling, John David
• Granted Patent: US 7,400,653 B2
• Time in Patent Office: Days
• US Class Current: 370/389
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4396 by muting the audio signal
• CPC Class Codes: H04N 21/64322 IP

Abstract

A playback system in a digital cinema network synchronizes the presentation of visual and aural content by deriving timing information for packets of information that are conveyed in video and audio data streams, examining the timing information to determine if any misalignment between the two data streams is likely to be perceptible and, if the misalignment is deemed to be perceptible, introducing delays into one or both data streams to correct the misalignment. If the audio data stream precedes the video data stream, the audio data stream is delayed by an integer number of audio sample periods. If the video data stream precedes the audio data stream, the video data stream is delayed by an integer number of video frames and the audio data stream is delayed by an integer number of audio sample periods.

Claims

1. A method for processing audio content and video content of an audio/video program, wherein the method comprises:

receiving a sequence of audio packets conveying first control information and the audio content and obtaining from the first control information audio time information for a respective audio packet in the sequence of audio packets;

receiving a sequence of video packets conveying second control information and the video content and obtaining from the second control information video time information for a respective video packet in the sequence of video packets;

determining from the audio time information and the video time information a relative time alignment of the respective audio packet with the respective video packet and applying a perceptual model to the relative time alignment to obtain a measure of perceived misalignment between the respective audio packet and the respective video packet;

if the measure of perceived misalignment indicates the respective audio packet precedes the desired time alignment with the respective video packet by an interval that exceeds a first threshold, delaying all or part of the respective audio packet by a first amount such that according to the perceptual model a perceived misalignment between the delayed respective audio packet and the respective video packet is reduced; and

if the measure of perceived misalignment indicates the respective audio packet follows the desired time alignment with the respective audio packet by an interval that exceeds a second threshold, delaying the respective video packet by a second amount and delaying all or part of the respective audio packet by a third amount such that according to the perceptual model a perceived misalignment between the delayed respective audio packet and the delayed respective video packet is reduced.

2. The method according to claim 1 wherein:

data conveyed in a respective audio packet represents an interval of aural information having an audio frame period and data conveyed in a respective video packet represents an interval of visual information having a video frame period;

the first amount is equal to an integer number of the audio frame periods;

the second amount is equal to an integer number of video frame periods; and

the third amount is equal to an integer number of audio frame periods.

3. The method according to claim 1 wherein:

data conveyed in a respective audio packet represents a plurality of samples each having an audio sample period and data conveyed in a respective video packet represents an interval of visual information having a video frame period;

the first amount is equal to an integer number of the audio sample periods;

the second amount is equal to an integer number of video frame periods; and

the third amount is equal to an integer number of audio sample periods.

4. The method according to claim 1 that comprises obtaining the audio time information by extrapolation from first control information that conveys timing for the beginning of the respective audio packet.

5. The method according to claim 1 that comprises:

inserting information into an audio data stream that is generated from all or part of the respective audio packet that is delayed; and

reducing playback levels of audio content preceding the inserted information and increasing playback levels of audio content following the inserted information.

6. The method according to claim 1 wherein:

the sequence of audio packets and the sequence of video patents is received from a communications network and stored in one or more buffers in a device; and

a protocol used to control communication in the network is used to control storage of information in the one or more buffers.

7. The method according to claim 1 that determines the first amount, the second amount and the third amount so that the perceived misalignment is reduced to such an extent that it is not perceptible according to the perceptual model.

8. A medium conveying a program of instructions that is executable by a device to perform a method for processing audio content and video content of an audio/video program, wherein the method comprises:

receiving a sequence of audio packets conveying first control information and the audio content and obtaining from the first control information audio time information for a respective audio packet in the sequence of audio packets;

receiving a sequence of video packets conveying second control information and the video content and obtaining from the second control information video time information for a respective video packet in the sequence of video packets;

determining from the audio time information and the video time information a relative time alignment of the respective audio packet with the respective video packet and applying a perceptual model to the relative time alignment to obtain a measure of perceived misalignment between the respective audio packet and the respective video packet;

if the measure of perceived misalignment indicates the respective audio packet precedes the desired time alignment with the respective video packet by an interval that exceeds a first threshold, delaying all or part of the respective audio packet by a first amount such that according to the perceptual model a perceived misalignment between the delayed respective audio packet and the respective video packet is reduced; and

if the measure of perceived misalignment indicates the respective audio packet follows the desired time alignment with the respective audio packet by an interval that exceeds a second threshold, delaying the respective video packet by a second amount and delaying all or part of the respective audio packet by a third amount such that according to the perceptual model a perceived misalignment between the delayed respective audio packet and the delayed respective video packet is reduced.

9. The medium according to claim 8 wherein:

data conveyed in a respective audio packet represents an interval of aural information having an audio frame period and data conveyed in a respective video packet represents an interval of visual information having a video frame period;

the first amount is equal to an integer number of the audio frame periods;

the second amount is equal to an integer number of video frame periods; and

the third amount is equal to an integer number of audio frame periods.

10. The medium according to claim 8 wherein:

data conveyed in a respective audio packet represents a plurality of samples each having an audio sample period and data conveyed in a respective video packet represents an interval of visual information having a video frame period;

the first amount is equal to an integer number of the audio sample periods;

the second amount is equal to an integer number of video frame periods; and

the third amount is equal to an integer number of audio sample periods.

11. The medium according to claim 8 wherein the method comprises obtaining the audio time information by extrapolation from first control information that conveys timing for the beginning of the respective audio packet.

12. The medium according to claim 8 wherein the method comprises:

inserting information into an audio data stream that is generated from all or part of the respective audio packet that is delayed; and

reducing playback levels of audio content preceding the inserted information and increasing playback levels of audio content following the inserted information.

13. The medium according to claim 8 wherein:

the sequence of audio packets and the sequence of video patents is received from a communications network and stored in one or more buffers in the device; and

a protocol used to control communication in the network is used to control storage of information in the one or more buffers.

14. The medium according to claim 8 wherein the method determines the first amount, the second amount and the third amount so that the perceived misalignment is reduced to such an extent that it is not perceptible according to the perceptual model.