Method and system for precise synchronization of audio and video streams during a distributed communication session with multiple participants

• US 7,949,890 B2
• Filed: 01/31/2008
• Issued: 05/24/2011
• Est. Priority Date: 01/31/2007
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Net Power & Light, Inc (Amazon.com, Inc.)
• Inventors: Shtayura, Vadim, Vonog, Stanislav, Surin, Nikolay, Iskhodzhanov, Timur
• Primary Examiner(s): Lee; Thomas
• Assistant Examiner(s): NGUYEN, PHIL K
• Application Number:
  US12/024,089
• Publication Number:
  US 20080181260A1
• Time in Patent Office: 1,209 Days
• Field of Search: None
• US Class Current: 713/400
• CPC Class Codes: G06F 1/14 Time supervision arrangements, e.g. real time clock
• CPC Class Codes: H04J 3/0632 Synchronisation of packets and cells, e.g. transmission of voice via a packet network, circuit emulation service [CES] queuing arrangements in packet switching elements H04L49/90; synchronising systems for the synchronous transmission of a pulse code modulated video signal with one or more other pulse code modulated signals H04N7/56
• CPC Class Codes: H04N 5/0736 using digital storage buffer techniques
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Described are the architecture of such a system, algorithms for time synchronization during a multiway conferencing session, methods to fight with network imperfections such as jitter to improve synchronization, methods of introducing buffering delays to create handicaps for players with faster connections, methods which help players with synchronization (such as a synchronized metronome during a music conferencing session), methods for synchronized recording and live delivery of synchronized data to the audience watching the distributed interaction live over the Internet.

Claims

1. A method for clock synchronization by calculating a correction value to be added or subtracted from a local time calculated comprising:

defining a first computer A'"'"'s time as an absolute time T_A and a second computer N'"'"'s time as non-absolute time T_N;

assigning a current best approximation of a network round trip time between computers A and N to RTT_app;

setting initial values of registers t_A,last and t_N,last to zero for computer A;

exchanging synchronization packets periodically;

computer N computing t₁=T_LocalTimeN, wherein T_LocalTimeN is a local time of computer N;

computer N writing t₁ into a synchronization section in a synchronization request;

computer N sending an audio packet to computer A;

computer A receiving the audio packet;

computer A computing t₂=T_LocalTimeA, wherein T_LocalTimeA is a local time of computer A;

computer A setting t_A,last to the value of t₂ and setting t_N,last to the value of t₁ from the audio packet if t₁>

t_N,last;

computer A calculating t′

₂ and writing (t′

₁=t₁+t′

₂−

t₂, t′

₂) into the synchronization section in a synchronization reply if the t_N,last was not sent in a synchronization reply, wherein t′

₂ is a local time of computer N just before an audio packet is sent to computer N;

computer A sending an audio packet to computer N;

computer N receiving the audio packet;

computer N computing t₃=T_LocalTimeN;

computer N computing RTT=t₃−

t′

₁;

computer N assigning RTT_app=RTT if RTT_app is not defined;

computer N assigning the value of minimum(RTT_app+RTT_drift, RTT) to RTT_app if RTT<

1.05*RTT_app, wherein RTT_drift is a drift in packet round trip time;

assigning a mean value of (t′

₂−

(t′

₁+t₃)/2) of buffered data from previous synchronization cycles to Δ

T<sub>N−

A</sub>;

synchronizing clock by adding a correction value Δ

T<sub>N−

A</sub> to a local time T_N.

2. The method of claim 1, wherein the local time T_N is calculated by processes including:

using a plurality of samples of digitized audio stream which comes from a network at any given time as a primary tool for synchronizing audio streams, wherein a length of incoming sound is divided by an audio driver into a plurality of buffers of a constant time duration T_Buffersize and an audio time increases with steps equal to T_Buffersize;

setting T_Start equal to T_SystemTime, wherein T_SystemTime is a current time on a local computer;

setting T_{Last Local Time Result} to zero;

calculating T_{Local time}=maximum(T_{Last Local Time Result}, minimum(T_SystemTime−

T_Start, T_Buffersize)+N_Samples/R_Sample), wherein N_Samples is a number of audio samples and R_Sample is a sample rate;

storing T_{Local Time} to T_N.

3. The method of claim 1, wherein the period of synchronization cycle ranges from 3 to 12 seconds.

4. The method of claim 1, wherein the mean value of (t′

₂−

(t′

₁+t₃)/2) of buffered data from previous synchronization cycles is calculated using 1000 or less data samples.

5. The method of claim 1, further comprising discarding a synchronization cycle if RTT>

D=1.05*RTT_app.

6. The method of claim 1, further comprising:

assigning a value obtained by 1 ms divided by the synchronization cycle rate in kHz to RTT_drift.

7. The method of claim 1, further comprising:

assigning Δ

T_N=Δ

T<sub>N−

A</sub>/2;

calculating Δ

T<sub>A−

N</sub> treating computer N as having an absolute time;

assigning Δ

T_A=Δ

T<sub>A−

N</sub>/2.

8. The method of claim 7, wherein both computer A and N make synchronization replies only after receiving synchronization requests.

9. The method of claim 7 further comprising:

calculating Δ

Ti=Σ

Δ

T<sub>i−

j</sub>/M, wherein Δ

T_i is a correction value for a computer i for synchronization among M computers, wherein M is an integer greater than 2 and Δ

T<sub>i−

i</sub> is zero.

10. The method of claim 9 further comprising sending synchronization replies to other computers in a round-robin manner.

11. The method of claim 9 further comprising sending an identification number of a sender of the synchronization request for the synchronization reply.

12. A system comprising:

a plurality of computers used for a distributed communication session with multiple participants using audio and video streams, wherein each computer includes;

a processor;

a clock;

a computer readable medium with instructions to carry out processes including;

defining a first computer A'"'"'s time as an absolute time T_A and a second computer N'"'"'s time as non-absolute time T_N;

assigning a current best approximation of a network round trip time between computers A and N to RTT_app;

setting initial values of registers t_A,last and t_N,last to zero for computer A;

exchanging synchronization packets periodically;

computer N computing t₁=T_LocalTimeN, wherein T_LocalTimeN is a local time of computer N;

computer N writing t₁ into a synchronization section in a synchronization request;

computer N sending an audio packet to computer A;

computer A receiving the audio packet;

computer A computing t₂=T_LocalTimeA, wherein T_LocalTimeA is a local time of computer A;

computer A setting t_A,last to the value of t₂ and setting t_N,last to the value of t₁ from the audio packet if t₁>

t_N,last;

computer A calculating t′

₂ and writing (t′

₁=t₁+t′

₂−

t₂, t′

₂) into the synchronization section in a synchronization reply if the t_N,last was not sent in a synchronization reply, wherein t′

₂ is a local time of computer N just before an audio packet is sent to computer N;

computer A sending an audio packet to computer N;

computer N receiving the audio packet;

computer N computing t₃=T_LocalTimeN;

computer N computing RTT=t₃−

t′

₁;

computer N assigning RTT_app=RTT if RTT_app is not defined;

computer N assigning the value of minimum(RTT_app+RTT_drift, RTT) to RTT_app if RTT<

1.05*RTT_app, wherein RTT_drift is a drift in packet round trip time;

assigning a mean value of (t′

₂−

(t′

₁+t₃)/2) of buffered data from previous synchronization cycles to Δ

T<sub>N−

A</sub>;

synchronizing clock by adding a correction value Δ

T<sub>N−

A</sub> to a local time T_N;

a network interface for communication among the plurality of computers.