Synchronization of audio and video signals from remote sources over the internet

• US 8,301,790 B2
• Filed: 02/22/2008
• Issued: 10/30/2012
• Est. Priority Date: 05/30/2007
• Status: Active Grant

• Current Assignee: Connectionopen Inc.
• Original Assignee: Lawrence Morrison, Randy Morrison
• Inventors: Morrison, Randy, Morrison, Lawrence
• Primary Examiner(s): Pollack, Melvin H
• Application Number:
  US12/070,983
• Publication Number:
  US 20090172200A1
• Time in Patent Office: 1,712 Days
• Field of Search: 709/231, 709/248, 709/230
• US Class Current: 709/231
• CPC Class Codes: G10H 1/0058 Transmission between separate instruments or between individual components of a musical system G10H1/0083 takes precedence
• CPC Class Codes: G10H 2240/175 for jam sessions or musical collaboration through a network, e.g. for composition, ensemble playing or repeating; Compensation of network or internet delays therefor
• CPC Class Codes: G10H 2240/305 Internet or TCP/IP protocol use for any electrophonic musical instrument data or musical parameter transmission purposes
• CPC Class Codes: G10H 2240/325 Synchronizing two or more audio tracks or files according to musical features or musical timings synchronised lyrics, e.g. for karaoke G10H2220/011

Abstract

The present invention is an architecture and technology for a method for synchronizing multiple streams of time-based digital audio and video content from separate and distinct remote sources, so that when the streams are joined, they are perceived to be in unison.

Claims

1. A method for providing synchronous delivery and playback of three or more electronic audio or video files, having differing arrival latencies, from participants from multiple locations, during an on-line session, the synchronous delivery and playback means comprising:

a. a session server having a master timestamp, said master timestamp used as a time reference by all participants;

b. a client application, said client application connecting a participant to the session server and to other participants and having a client timestamp and utilizing a formalized Internet time standard, said Internet time standard being the Network Time Protocol (NTP) which is used as the predictive successive approximation of the time of day for the client and the server, said client and server timestamp is synchronized with the master timestamp;

c. a timing mechanism, said timing mechanism synchronizing the client timestamp in the client application of the other participants and increasing the frequency of polling of the NTP so that the master timestamp and all client timestamps are synchronized to a precision of at least 10 milliseconds;

d. a file calibrating mechanism, said file calibrating mechanism having a buffer, a mixer, and a delayed timestamp, said buffer having a means for analyzing the difference in arrival latencies in real time of files by all participants, and a means for synchronizing the files, by which the arrival latency of any participant'"'"'s file may be increased so that all files by all participants arrive at the same time, and said mixer compiling the synchronized files into multiple files which are then returned to the participants, and said delayed timestamp being the timing means of the files after the files have been synchronized;

e. respective receivers at each client and the session server receiving packets of information from each client, the receiver decoding the timestamp from each client and comparing it with the timestamp of the master timestamp, keeping a record for each client of the difference in time of the time stream from the master timestamp, the stream with the highest difference designated as the delay reference stream and the timestamp from the delay reference stream is used as a reference time delayed timestamp; and

f. once the delayed reference stream has been determined, its data is immediately decoded and rendered to the client having the delayed reference stream, other incoming streams are then decoded and then paused until their timestamp agrees with the delayed timestamp and only then are they rendered to the client having that respective stream so that all incoming streams are in sync with the delayed timestamp and are therefore in unison with one another.

2. The synchronous delivery and playback means in accordance with claim 1, wherein said synchronous delivery and playback means further comprises a reference timestamp, said reference timestamp controlled by one of the participants, and constantly monitoring the NTP so as to continuously adjust the timing conditions.

3. An apparatus to provide synchronous delivery and playback of three or more electronic audio or video files, having differing arrival latencies, from participants from multiple locations, during an on-line session, the synchronous delivery and playback apparatus comprising:

a. a session server having a master timestamp, said master timestamp used as a time reference by all participants;

b. a client application, said client application connecting a participant to the session server and to other participants and having a client timestamp, and utilizing a formalized Internet time standard, said Internet time standard being the Network Time Protocol (NTP) which is used as the predictive successive approximation of the time of day for the client and the server, said client timestamp is synchronized with the master timestamp;

c. a timing mechanism, said timing mechanism synchronizing the client timestamp in the client application of the other participants and increasing the frequency of polling of the NTP so that the master timestamp and all client timestamps are synchronized to a precision of at least 10 milliseconds;

d. a file calibrating mechanism, said file calibrating mechanism having a buffer, said buffer having a means for analyzing the difference in arrival latencies in real time of files by all participants, and a means for synchronizing the files, by which the arrival latency of any participant'"'"'s file may be increased so that all files by all participants arrive at the same time;

e. a receiver at the session server receiving packets of information from each client, the receiver decoding the timestamp from each client and comparing it with the timestamp of the master timestamp, keeping a record for each client of the difference in time of the time stream from the master timestamp, the stream with the highest difference designated as the delay reference stream and the timestamp from the delay reference stream is used as a reference time delayed timestamp; and

f. once the delayed reference stream has been determined, its data is immediately decoded and rendered to the client having the delayed reference stream, other incoming streams are then decoded and then paused until their timestamp agrees with the delayed timestamp and only then are they rendered to the client having that respective stream so that all incoming streams are in sync with the delayed timestamp and are therefore in unison with one another.

4. The synchronous delivery and playback apparatus in accordance with claim 3, wherein said file calibrating mechanism further mixes the synchronized files into one file which is then returned to the participants.

5. The synchronous delivery and playback apparatus in accordance with claim 3, wherein said file calibrating mechanism further mixes the synchronized files into one file which is then returned simultaneously to the participants.

6. The synchronous delivery and playback apparatus in accordance with claim 3, wherein said synchronous delivery and playback means further comprises a reference timestamp, said reference timestamp controlled by one of the participants, and constantly monitoring the NTP so as to continuously adjust the timing conditions.

7. The synchronous delivery and playback apparatus in accordance with claim 3, wherein said file calibrating mechanism further comprises a delayed timestamp, said delayed timestamp being the timing of the files after the files have been synchronized.

8. A method to provide synchronous delivery and playback of three or more electronic audio or video files, having differing arrival latencies, from participants from multiple locations, during an on-line session, the synchronous delivery and playback method comprising:

a. creating a session on a server;

b. allowing participants to request to join the session;

c. approving or denying the participant'"'"'s request to join the session;

d. only after approval, joining the participant to the session and timestamping the participant'"'"'s session, and utilizing a formalized Internet time standard, said Internet time standard being the Network Time Protocol (NTP) which is used as the predictive successive approximation of the time of for the client and the server;

e. enabling a client application, said client application calculating each respective client'"'"'s and server'"'"'s reference time and factoring in a delay time;

f. starting a reference timestamp, said reference timestamp synchronized to the time reference of the server and is given simultaneously to all participants, increasing the polling of the NTP so that the master timestamp and all participant timestamps are synchronized to a precision of at least 10 milliseconds;

g. connection by the client application of each participant to the client application of the other participants and determination of each participant'"'"'s time differentials in real time;

h. adjusting constantly of the reference timestamp to the changes in the network conditions;

i. buffering and synchronizing the participants'"'"' multimedia streams so that all streams are transmitted so as to arrive at the same time as the slowest stream;

j. creating a delayed timestamp, said delayed timestamp in time with the buffered and synchronized multimedia stream;

k. utilizing the embedded timestamping within the transmitted streams to determine which stream has the greatest latency as compared to the reference timestamp;

l. decoding all streams as they arrive at the server;

m. designating the stream with the greatest latency as the delay reference stream;

n. buffering all other streams until each stream'"'"'s timestamp matches that of the delay reference stream;

o. rendering the all outgoing streams to all participants such that the participant with the least latency receives its stream at the same time as the participant with the greatest latency;

p. a receiver at the session server receiving packets of information from each client, the receiver decoding the timestamp from each client and comparing it with the timestamp of the master timestamp, keeping a record for each client of the difference in time of the time stream from the master timestamp, the stream with the highest difference designated as the delay reference stream and the timestamp from the delay reference stream is used as a reference time delayed timestamp; and

q. once the delayed reference stream has been determined, its data is immediately decoded and rendered to the client having the delayed reference stream, other incoming streams are then decoded and then paused until their timestamp agrees with the delayed timestamp and only then are they rendered to the client having that respective stream so that all incoming streams are in sync with the delayed timestamp and are therefore in unison with one another.

9. The synchronous delivery and playback method in accordance with claim 8, wherein said synchronous delivery and playback method further mixing the synchronized files into one file which is then returned to the participants.

10. The synchronous delivery and playback method in accordance with claim 8, wherein said synchronous delivery and playback method further comprises mixing the synchronized files into one file which is then returned simultaneously to the participants.

11. The synchronous delivery and playback method in accordance with claim 8, wherein said synchronous delivery and playback method utilizes a formalized Internet time standard, said Internet time standard being the Network Time Protocol (NTP).

12. The synchronous delivery and playback method in accordance with claim 8, wherein said synchronous delivery and playback method further comprising a reference timestamp, said reference timestamp controlled by one of the participants, and constantly monitoring the NTP so as to continuously adjust the timing conditions.