Synchronization of audio and video signals from remote sources over the internet

• US 8,918,541 B2
• Filed: 04/08/2010
• Issued: 12/23/2014
• Est. Priority Date: 02/22/2008
• Status: Active Grant

• Current Assignee: Connectionopen Inc.
• Original Assignee: Lawrence Morrison, Randy Morrison
• Inventors: Morrison, Randy, Morrison, Lawrence
• Primary Examiner(s): Zele, Krista
• Assistant Examiner(s): DESAI, MARGISHI V
• Application Number:
  US12/798,619
• Publication Number:
  US 20100198992A1
• Time in Patent Office: 1,720 Days
• Field of Search: 386/220, 709/248
• US Class Current: 709/248
• CPC Class Codes: G06F 16/4393 Multimedia presentations, e.g. slide shows, multimedia albums
• CPC Class Codes: G10H 1/0058 Transmission between separate instruments or between individual components of a musical system G10H1/0083 takes precedence
• CPC Class Codes: G10H 2240/175 for jam sessions or musical collaboration through a network, e.g. for composition, ensemble playing or repeating; Compensation of network or internet delays therefor
• CPC Class Codes: G10H 2240/305 Internet or TCP/IP protocol use for any electrophonic musical instrument data or musical parameter transmission purposes
• CPC Class Codes: G10H 2240/325 Synchronizing two or more audio tracks or files according to musical features or musical timings synchronised lyrics, e.g. for karaoke G10H2220/011
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel

Abstract

The present invention is an architecture and technology for a method for synchronizing multiple streams of time-based digital audio and video content from separate and distinct remote sources, so that when the streams are joined, they are perceived to be in unison.

Claims

1. A method for providing synchronous delivery and playback of three or more electronic audio or video files from multiple clients, having differing arrival latencies, from clients from multiple locations communicating through the Internet, during an on-line session, the synchronous delivery and playback method comprising:

a. creating a session server having a list of Network Time Protocol (NTP) servers, passing the list to each client through the Internet so that each client will acquire that client'"'"'s own time reference;

b. enabling a client application to provide authorization so that the client application enables each client to connect each client to each other client and to the session server, each client utilizing a formalized Internet time standard, said Internet time standard being the NTP, each client having an NTP clock which is shared with all clients;

c. starting the NTP clock, the NTP clock synchronized to the time stamp of its chosen NTP server and given to all clients, providing a timing adjustor for adjusting each client'"'"'s NTP timestamp in the client application of all clients, enabling the NTP timestamp adjustor of each client to acquire the NTP timestamp from all clients, in the session and the session server, enabling the session server to calculate the distance of each client and synch any NTP timestamp of all clients, so that streams of the clients will be offset until each stream of a client is in synch with the stream of the furthest client;

d. constantly adjusting the NTP timestamp to change in the network conditions, and providing a file calibrator in each client application, said file calibrator having a buffer, a mixer, and the NTP timestamp, said buffer having a method for analyzing the difference in arrival latencies of streams by all clients, and a method for synchronizing the streams, by which the arrival latency of any client'"'"'s stream may be increased so that all streams by all clients arrive at the same time, and said mixer returns each stream to all clients, and said adjusted NTP timestamp from all clients being the timing method of each stream;

e. providing respective receivers at each client and at the session server receiving packets of information from each client, each receiver decoding the NTP timestamp from each client and comparing it with an NTP timestamp of the formalized Internet time standard, keeping a record for each client of the difference in time of the time stream from the formalized Internet time standard, the stream with the highest difference designated as the delay reference stream and the NTP timestamp from the delay reference stream is used as a reference time delayed NTP timestamp; and

f. once the delayed reference stream has been determined, its data is immediately decoded and rendered to the client having the delayed reference stream, other incoming streams are then decoded and then paused until their NTP timestamp agrees with the delayed NTP timestamp and only then are they rendered to the client having that respective stream so that all incoming streams are in sync with the delayed NTP timestamp and are therefore in unison with one another.

2. A method to provide synchronous delivery and playback of three or more electronic audio or video files from multiple clients, having differing arrival latencies, from clients from multiple locations communicating through the Internet, during an on-line session, the synchronous delivery and playback method comprising:

a. creating a session server having a list of Network Time Protocol (NTP) servers, the list is passed to each client through the Internet so that each client will acquire that client'"'"'s own time reference;

b. enabling a client application to provide authorization so that the client application enables each client to connect each client to each other client and to the session server, each client utilizing a formalized Internet time standard, said Internet time standard being the NTP, each client having an NTP clock which is shared with all clients;

c. providing to each client a timing adjustor for adjusting each client'"'"'s NTP timestamp in the client application of all clients, the timing adjustor for adjusting each client'"'"'s NTP timestamp in the client application of all clients, the timing adjustor enables each client to acquire the NTP timestamp from all clients, in the session and the session server, which will calculate the distance of each client and synch any NTP stamps of all clients, so that the streams of the clients will be offset until each stream of a client is in synch with the streams of the furthest client;

d. constantly adjusting the NTP timestamp to change the network conditions, a file calibrator in each client application, said file calibrator having a buffer, a mixer, and an NTP timestamp, said buffer having a method for analyzing the difference in arrival latencies of streams by all clients, and a method for synchronizing the streams, by which the arrival latency of any client'"'"'s stream may be increased so that all streams by all clients arrive at the same time, and said mixer returns each stream to all clients, and said adjusted NTP timestamp from all clients being the timing method of each stream, the file calibrator performs additional functions selected from the group consisting of the file calibrator synchronizes streams based on an NTP timestamp of each client so that the streams arrive at each client in synch with the other clients and the file calibrator aligns the streams with the other clients'"'"' streams and then returns the streams simultaneously to all clients;

e. at the session server receiving packets of information from each client, the receiver decoding the NTP timestamp from each client and comparing it with the NTP timestamp of the formalized Internet time standard, keeping a record for each client of the difference in time of the time stream from the formalized Internet time standard, the stream with the highest difference designated as the delay reference stream and the NTP timestamp from the delay reference stream is used as a reference time delayed NTP timestamp; and

f. once the delayed reference stream has been determined, its data is immediately decoded and rendered to the client having the delayed reference stream, other incoming streams are then decoded and then paused until their NTP timestamp agrees with the delayed NTP timestamp and only then are they rendered to the client having that respective stream so that all incoming streams are in sync with the delayed NTP timestamp and are therefore in unison with one another.

3. A method to provide synchronous delivery and playback of three or more electronic audio or video files from multiple clients in accordance with claim 2, further providing an NTP timestamp, said NTP timestamp is controlled by each of the clients, and constantly monitoring the NTP timestamp so as to continuously adjust the timing conditions.

4. A method to provide synchronous delivery and playback of three or more electronic audio or video files from multiple clients in accordance with claim 2, further comprising said file calibrator further comprises an NTP timestamp, said NTP timestamp being the timing of that client after the stream has been synchronized with the other streams.

5. A method to provide synchronous delivery and playback of three or more electronic audio or video files from multiple clients, having differing arrival latencies, from clients from multiple locations, during an on-line session, the synchronous delivery and playback method comprising:

a. creating a session on each client;

b. allowing clients to request to join the session;

c. approving or denying the client'"'"'s request to join the session;

d. acquiring a Network Time Protocol (NTP) server and timestamp and utilizing a formalized Internet time standard being the Network Time Protocol (NTP);

e. only after approval, joining the client to the session and time stamping the participant'"'"'s session;

f. enabling a client application, the client application calculating the NTP server'"'"'s NTP timestamp and factoring in a delay time, said client application connecting each client to each other client and to the session server, each client utilizing a formalized Internet time standard, said Internet time standard being the NTP, each client having an NTP clock which is shared with all clients;

g. starting an NTP clock, the clock synchronized to the timestamp of its chosen NTP server and is given to all clients, a timing adjustor for adjusting each client'"'"'s NTP timestamp in the client application of all clients, the timing adjustor enables each client to acquire the NTP stamp from all clients, in the session and the session server, which will calculate the distance of each client and synch any NTP timestamp of all clients, so that the streams of the clients will be offset until each stream of a client is in synch with the streams of the furthest client;

h. connection by the client application to the client application of the other participants and determination of each client'"'"'s time differentials;

i. constantly adjusting the NTP timestamp to changes in the network conditions, a file calibrator in each client application, said file calibrator having a buffer, a mixer, and an NTP timestamp, said buffer for analyzing the difference in arrival latencies of streams by all clients, and for synchronizing the streams, by which the arrival latency of any client'"'"'s stream may be increased so that all streams by all clients arrive at the same time, and said mixer returns each stream to all clients, and said adjusted NTP timestamp from all clients being the timing method of each stream;

j. buffering and synchronizing the clients'"'"' multimedia streams so that all streams are transmitted so as to arrive at the same time as the slowest stream;

k. creating a time stamped stream, said time stamped stream is buffered and synchronized multimedia streams;

l. utilizing the embedded time stamp within the transmitted streams to determine which stream has the greatest latency as compared to its own NTP timestamp;

m. decoding all streams as they arrive at each client;

n. designating the stream with the greatest latency as the delay reference stream;

o. buffering all other streams until each stream'"'"'s time stamp matches that of the delay reference stream;

p. rendering the all outgoing streams to all clients such that the client with the least latency receives its stream at the same time as the participant with the greatest latency;

q. at the session server receiving packets of information from each client, the receiver decoding the NTP timestamp from each client and comparing it with the NTP timestamp of the formalized Internet time standard, keeping a record for each client of the difference in time of the time stream from the formalized Internet time standard, the stream with the highest difference designated as the delay reference stream and the NTP timestamp from the delay reference stream is used as a reference time delayed NTP timestamp, and

r. once the delayed reference stream has been determined, its data is immediately decoded and rendered to the client having the delayed reference stream, other incoming streams are then decoded and then paused until their NTP timestamp agrees with the delayed NTP timestamp and only then are they rendered to the client having that respective stream so that all incoming streams are in sync with the delayed NTP timestamp and are therefore in unison with one another.

6. A method to provide synchronous delivery and playback of three or more electronic audio or video files from multiple clients in accordance with claim 5, further comprising said synchronous delivery and playback method further aligns the synchronized stream so that each client receives the stream in sync.

7. A method to provide synchronous delivery and playback of three or more electronic audio or video files from multiple clients in accordance with claim 5, further comprising said synchronous delivery and playback method further comprises aligning the streams so that all streams appear simultaneously to the clients.

8. A method to provide synchronous delivery and playback of three or more electronic audio or video files from multiple clients in accordance with claim 5, further comprising said synchronous delivery and playback method further comprises an NTP timestamp, said NTP timestamp controlled by each of the participants, and constantly monitoring the NTP timestamp so as to continuously adjust the timing conditions.