Synchronisation of audio and video signals

• US 20040205214A1
• Filed: 12/16/2003
• Published: 10/14/2004
• Est. Priority Date: 09/14/2000
• Status: Active Grant

• Current Assignee: Telefonaktiebolaget LM Ericsson
• Original Assignee: Telefonaktiebolaget LM Ericsson
• Inventors: Bång, Goran, Lovgren, Tage
• Granted Patent: US 7,295,247 B2
• Time in Patent Office: Days
• US Class Current: 709/231
• CPC Class Codes: H04N 21/2187 Live feed
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4341 Demultiplexing of audio and video streams

Abstract

The present invention relates to adaptive time-shifting of audio frames (F_a) in encoded audio signal relative to video frames (F_v) in encoded video signal, which together form an audio/video message steam such that the corresponding decoded audio and video signals are aligned in time within the maximum degree of deviation. The invention accomplishes this alignment by determining a desired delay of the encoded audio signal for the decoded signals to meet deviation requirement The desired (D_d) delay is then converted into a frame delay number (n_D), which is calculated as one of the two neighbouring integers to a ratio between the desired delay (D_d) and an interval (T_a) at which the audio frames (F_a) are received. A number of audio frames (F_a) equal to the frame delay number (n_D) are alternately stored into respective read out from FIFO-buffers (309; 310) such that sufficiently delayed audio frames (F_a) are delivered to an audio decoding unit (313). After signal decoding (313; 315) synchronised audio and video information can thus be presented via a loudspeaker (314) and a video monitor (316) respectively.

Claims

1. A method of adaptively time-shifting an encoded audio signal relative to an encoded video signal in a received audio/video message stream such that a corresponding decoded audio signal and a corresponding decoded video signal are aligned in time within a maximum degree of deviation (Δ

_max),

the encoded audio signal being divided into audio frames (F_a) having a first frame interval (T_a),

the encoded video signal being divided into video frames (F_v) having a second frame interval (T_v),

determining a desired delay (D_d) of the encoded audio signal for the corresponding decoded audio and video signals to be aligned in time within the maximum degree of deviation (Δ

_max), the method characterised by the steps of;

calculating a frame delay number (n_D) as one of the two neighbouring integers to a ratio between the desired delay (D_d) and the first frame interval (T_a),

storing sequentially a number of audio frames (F_a(1)-F_a(5)) equal to the frame delay number (n_D) before decoding a matching video frame (F_v(1)),

decoding the matching video frame (F_v(1)),

decoding the stored audio frames (F_a(1)-F_a(5)) in the sequential order of storage, and

repeating the above steps for remaining audio frames ((F_a(6)-F_a(k)) respectiye video frames (F_v(2)-F_v(m)) of the audio/video message stream.

2. A method according to claim 1, characterised by the frame delay number (n_D) being equal to the integer part of the ratio between the desired delay (D_d) and the first frame interval (T_a).

3. A method according to claim 1, characterised by the frame delay number (n_D) being equal to the integer part of the ratio between the desired delay (D_d) and the first frame interval (T_a) plus one.

4. A method according to any of the preceding claims, characterised by the audio/video message stream being received under a transport protocol which provides support for timing reconstruction.

5. A method according to claim 4, characterised by each audio frame (F_a) and each video frame (F_v) being allocated a timestamp indicative of a point in time when a particular frame (F_a;

F_v) was created.

6. A method according to claim 5, characterised by the desired delay (D_d) being determined from the timestamps allocated to the audio frames (F_v) and the video frames (F_v) respectively in the received audio/video message stream.

7. A computer program directly loadable into the internal memory of a computer, comprising software for performing the steps of any of the claims 1-6 when said program is run on the computer.

8. A computer readable medium, having a program recorded thereon, where the program is to make a computer perform the steps of any of the claims 1-6.

9. An apparatus for adaptively time-shifting an encoded audio signal relative to an encoded video signal in a received audio/video message stream such that a corresponding decoded audio signal respective decoded video signal are aligned in time within a maximum degree of deviation, the encoded audio signal being divided into audio frames (F_a) having a first frame interval (T_a), the encoded video signal being divided into video frames (F_v) having a second frame interval (T_v), the apparatus characterised in that it comprises

a first buffer (309) sequentially storing received audio frames (F_a),

a second buffer (310) sequentially storing received audio frames (F_a), and

a controller (312)

storing a first frame interval value (T_a) representing a time span between two consecutively generated audio frames (T_a),

deriving and storing a desired delay value (D_d) from timing indications (ts_a, ts_v) of the received audio frames (F_a) and the received video frames (F_v) in the received audio/video message stream, the desired delay value (D_d) indicating a necessary delay of the audio frames (F_a) for the decoded audio and video signals to be aligned in time within the maximum degree of deviation,

calculating a frame delay number (n_D) as one of the two neighbouring integers to a ratio between the desired delay value (D_d) and the first frame interval value (T_a), an,d

controlling an alternate storage of delay number (n_D) audio frames (F_a) into the first buffer (309) during read-out of delay number (n_D) audio frames (F_a) from the second buffer (309) respective storage of delay number (n_D) audio frames (F_a) into the second buffer (310) during read-out of delay number (n_D) audio frames (F_a) from the first buffer (309).

10. An apparatus according to claim 9, characterised in that it comprises a first buffer switch (B) and a second buffer switch (C),

the controller (312) controlling the first buffer switch (B) to alternately store audio frames (F_a) into the first buffer (309) or into the second buffer (310), and

the controller (312) controlling the second buffer switch (C) to alternately read out audio frames from the first buffer (309) or from the second buffer (310).

11. An apparatus according to any of the claims 9 or 10, characterised in that it comprises a first delay switch (A) and a second delay switch (D),

the controller (312) controlling the first delay switch (A) to alternately add a delay to the received audio signal by feeding its corresponding audio frames (F_a) through the buffers (309;

310) or forward its corresponding audio frames (F_a) without additional delay, and

the controller (312) controlling the second delay switch (D) to alternately read out delayed audio frames (F_a) from the buffers (309;

310) or non-delayed audio frames (F_a) via the first delay switch (A).

12. An apparatus according to any of the claims 9-11, characterised in that it comprises a de-multiplexer (307), the de-multiplexer (307)

receiving the audio/video message stream and separating it into audio frames (F_a) respective video frames (F_v),

arranging the audio frames (F_a) and the video frames (F_v) in a time sequential order based on a timing indication (ts_a) allocated to each frame.

forwarding audio frames (F_a) and timing indications (ts_a, ts_v) of the audio frames (F_a) and the video frames (F_v) to the controller (312), and

13. An apparatus according to claims 12, characterised in that it comprises an input data forming unit (308) and an output data delivery unit (311),

the input data forming unit (308) receiving audio frames (F_a) to be delayed from the de-multiplexer (307) and arranging such frames according to a data structure which is suitable for sequential storage in a delay buffer,

the output data delivery unit (311) re-arranging delayed audio frames (F_a) according to a data structure which is suitable for decoding and delivering the frames for subsequent decoding into a decoded audio signal.

14. A system for transmission of real-time audio and video information from a transmitter station to a receiver station via a transmission medium (306),

the transmitter station comprising an audio encoding unit (303) encoding audio information into audio frames (F_a), a video encoding unit (304) encoding video information into video frames (F_v) and a multiplexer combining the audio frames (F_a) and the video frames (F_v) to an audio/video message stream to be sent through the transmission medium (306), and

the receiver station comprising an apparatus according to any of the claims 9-13, an audio decoding unit (313) extracting an audio signal from received audio frames (F_a) and a video decoding unit (315) extracting a video signal from received video frames (F_v).

15. A system according to claim 14, characterised in that the transmitter station comprises

a microphone (301) registering acoustic data and forwarding an audio signal to the audio encoding unit (303), and

a video camera (302) registering image data and forwarding a video signal to the video encoding unit (304), and

the receiver station comprises

a loudspeaker (314) receiving an audio signal from the audio decoding unit (313) and delivering a corresponding acoustic signal, and

a video monitor (316) receiving a video signal from the video decoding unit (315) and presenting a corresponding image signal.