Synchronization of audio and video signals from remote sources over the internet

• US 20090172200A1
• Filed: 02/22/2008
• Published: 07/02/2009
• Est. Priority Date: 05/30/2007
• Status: Active Grant

• Current Assignee: Connectionopen Inc.
• Original Assignee: Lawrence Morrison, Randy Morrison
• Inventors: Morrison, Randy, Morrison, Lawrence
• Granted Patent: US 8,301,790 B2
• Time in Patent Office: Days
• US Class Current: 709/248
• CPC Class Codes: G10H 1/0058 Transmission between separate instruments or between individual components of a musical system G10H1/0083 takes precedence
• CPC Class Codes: G10H 2240/175 for jam sessions or musical collaboration through a network, e.g. for composition, ensemble playing or repeating; Compensation of network or internet delays therefor
• CPC Class Codes: G10H 2240/305 Internet or TCP/IP protocol use for any electrophonic musical instrument data or musical parameter transmission purposes
• CPC Class Codes: G10H 2240/325 Synchronizing two or more audio tracks or files according to musical features or musical timings synchronised lyrics, e.g. for karaoke G10H2220/011

Abstract

The present invention is an architecture and technology for a method for synchronizing multiple streams of time-based digital audio and video content from separate and distinct remote sources, so that when the streams are joined, they are perceived to be in unison.

Claims

1. A means for providing synchronous delivery and playback of three or more electronic audio or video files, having differing arrival latencies, from participants from multiple locations, during an on-line session, the synchronous delivery and playback means comprising:

a. a session server having a master metronome;

said master metronome used as a time reference by all participants;

b. a client application, said client application connecting a participant to the session server and to other participants and having a client metronome and utilizing a formalized Internet time standard, said Internet time standard being the Network Time Protocol (NTP), said client metronome is synchronized with the master metronome;

c. a timing mechanism, said timing mechanism synchronizing the client metronome in the client application of the other participants; and

d. a file calibrating mechanism, said file calibrating mechanism having a buffer, a mixer, and a delayed metronome, said buffer having a means for analyzing the difference in arrival latencies of files by all participants, and a means for synchronizing the files, by which the arrival latency of any participant'"'"'s file may be increased so that all files by all participants arrive at the same time, and said mixer compiling the synchronized files into one file which is then returned to the participants, and said delayed metronome being the timing means of the files after the files have been synchronized.

2. The synchronous delivery and playback means in accordance with claim 1, wherein said synchronous delivery and playback means further comprises a reference metronome, said reference metronome controlled by one of the participants, and constantly monitoring the NTP so as to continuously adjust the timing conditions.

3. An apparatus to provide synchronous delivery and playback of three or more electronic audio or video files, having differing arrival latencies, from participants from multiple locations, during an on-line session, the synchronous delivery and playback apparatus comprising:

a. a session server having a master metronome;

said master metronome used as a time reference by all participants;

b. a client application, said client application connecting a participant to the session server and to other participants and having a client metronome, said client metronome is synchronized with the master metronome;

c. a timing mechanism, said timing mechanism synchronizing the client metronome in the client application of the other participants; and

d. a file calibrating mechanism, said file calibrating mechanism having a buffer, said buffer having a means for analyzing the difference in arrival latencies of files by all participants, and a means for synchronizing the files, by which the arrival latency of any participant'"'"'s file may be increased so that all files by all participants arrive at the same time.

4. The synchronous delivery and playback apparatus in accordance with claim 3, wherein said file calibrating mechanism further mixes the synchronized files into one file which is then returned to the participants.

5. The synchronous delivery and playback apparatus in accordance with claim 3, wherein said file calibrating mechanism further mixes the synchronized files into one file which is then returned simultaneously to the participants.

6. The synchronous delivery and playback apparatus in accordance with claim 3, wherein said client application utilizes a formalized Internet time standard, said Internet time standard being the Network Time Protocol (NTP).

7. The synchronous delivery and playback apparatus in accordance with claim 3, wherein said synchronous delivery and playback means further comprises a reference metronome, said reference metronome controlled by one of the participants, and constantly monitoring the NTP so as to continuously adjust the timing conditions.

8. The synchronous delivery and playback apparatus in accordance with claim 3, wherein said file calibrating mechanism further comprises a delayed metronome, said delayed metronome being the timing of the files after the files have been synchronized.

9. A method to provide synchronous delivery and playback of three or more electronic audio or video files, having differing arrival latencies, from participants from multiple locations, during an on-line session, the synchronous delivery and playback method comprising:

a. creating a session on a server;

b. allowing participants to request to join the session;

c. approving or denying the participant'"'"'s request to join the session;

d. only after approval, joining the participant to the session and time stamping the participant'"'"'s session;

e. enabling a client application, said client application calculating the server'"'"'s reference time and factoring in a delay time;

f. starting a reference metronome, said reference metronome synchronized to the time reference of the server and is given simultaneously to all participants;

g. connection by the client application of each participant to the client application of the other participants and determination of each participant'"'"'s time differentials;

h. adjusting constantly of the reference metronome to the changes in the network conditions;

i. buffering and synchronizing the participants'"'"' multimedia streams so that all streams are transmitted so as to arrive at the same time as the slowest stream;

j. creating a delayed metronome, said delayed metronome in time with the buffered and synchronized multimedia stream;

k. utilizing the embedded time stamp within the transmitted streams to determine which stream has the greatest latency as compared to the reference metronome;

l. decoding all streams as they arrive at the server;

m. designating the stream with the greatest latency as the delay reference stream;

n. buffering all other streams until each stream'"'"'s time stamp matches that of the delay reference stream; and

o. rendering the all outgoing streams to all participants such that the participant with the least latency receives its stream at the same time as the participant with the greatest latency.

10. The synchronous delivery and playback method in accordance with claim 9, wherein said synchronous delivery and playback method further mixing the synchronized files into one file which is then returned to the participants.

11. The synchronous delivery and playback method in accordance with claim 9, wherein said synchronous delivery and playback method further comprises mixing the synchronized files into one file which is then returned simultaneously to the participants.

12. The synchronous delivery and playback method in accordance with claim 9, wherein said synchronous delivery and playback method utilizes a formalized Internet time standard, said Internet time standard being the Network Time Protocol (NTP).

13. The synchronous delivery and playback method in accordance with claim 9, wherein said synchronous delivery and playback method further comprising a reference metronome, said reference metronome controlled by one of the participants, and constantly monitoring the NTP so as to continuously adjust the timing conditions.