Synchronization of audio and video signals from remote sources over the internet

• US 20100198992A1
• Filed: 04/08/2010
• Published: 08/05/2010
• Est. Priority Date: 02/22/2008
• Status: Active Grant

• Current Assignee: Connectionopen Inc.
• Original Assignee: Lawrence Morrison, Randy Morrison
• Inventors: Morrison, Randy, Morrison, Lawrence
• Granted Patent: US 8,918,541 B2
• Time in Patent Office: Days
• US Class Current: 709/248
• CPC Class Codes: G06F 16/4393 Multimedia presentations, e.g. slide shows, multimedia albums
• CPC Class Codes: G10H 1/0058 Transmission between separate instruments or between individual components of a musical system G10H1/0083 takes precedence
• CPC Class Codes: G10H 2240/175 for jam sessions or musical collaboration through a network, e.g. for composition, ensemble playing or repeating; Compensation of network or internet delays therefor
• CPC Class Codes: G10H 2240/305 Internet or TCP/IP protocol use for any electrophonic musical instrument data or musical parameter transmission purposes
• CPC Class Codes: G10H 2240/325 Synchronizing two or more audio tracks or files according to musical features or musical timings
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel

Abstract

The present invention is an architecture and technology for a method for synchronizing multiple streams of time-based digital audio and video content from separate and distinct remote sources, so that when the streams are joined, they are perceived to be in unison.

Claims

1. A means for providing synchronous delivery and playback of three or more electronic audio or video files from multiple clients, having differing arrival latencies, from clients from multiple locations communicating through the internet, during an on-line session, the synchronous delivery and playback means comprising:

a. a session server having a list of Network Time Protocol (NTP) servers, the list is passed to each client through the internet so that each client can acquire its own time reference;

b. a client application, said client application connecting each client to each other and to the session server, each client utilizing a formalized Internet time standard, said Internet time standard being the NTP, each client having an NTP clock which is shared with all clients;

c. a timing mechanism, said timing mechanism adjusting each client'"'"'s NTP timestamp in the client application of all clients; and

d. a file calibrating mechanism in each client application, said file calibrating mechanism having a buffer, a mixer, and a n offset NTP timestamp, said buffer having a means for analyzing the difference in arrival latencies of streams by all clients, and a means for synchronizing the streams, by which the arrival latency of any client'"'"'s stream may be increased so that all streams by all clients arrive at the same time, and said mixer returns each stream to all clients, and said adjusted timestamp from all clients being the timing means of each stream.

2. The synchronous delivery and playback means in accordance with claim 1, wherein said synchronous delivery and playback means further comprises an offset NTP timestamp, said NTP timestamps are controlled by each of the clients, and each client is constantly monitoring the NTP server that each respective client choose so as to continuously adjust the timing conditions.

3. An apparatus to provide synchronous delivery and playback of three or more electronic audio or video files from multiple clients, having differing arrival latencies, from clients from multiple locations communicating through the Internet, during an on-line session, the synchronous delivery and playback apparatus comprising:

a. a session server having a list of Network Time Protocol (NTP) servers, the list is passed to each client through the internet so that each client can acquire its own time reference;

b. a client application, said client application connecting each client to each other and to the session server, each client utilizing a formalized Internet time standard, said Internet time standard being the NTP, each client having an NTP clock which is shared with all clients;

c. a timing mechanism, said timing mechanism adjusting each client'"'"'s NTP timestamp in the client application of all clients; and

d. a file calibrating mechanism in each client application, said file calibrating mechanism having a buffer, a mixer, and an offset NTP timestamp, said buffer having a means for analyzing the difference in arrival latencies of streams by all clients, and a means for synchronizing the streams, by which the arrival latency of any client'"'"'s stream may be increased so that all streams by all clients arrive at the same time, and said mixer returns each stream to all clients, and said adjusted timestamp from all clients being the timing means of each stream.

4. The synchronous delivery and playback apparatus in accordance with claim 3, wherein said file calibrating mechanism synchronizes streams based on the offset NTP timestamps of each client so that the streams arrive at each client in sync with the other clients.

5. The synchronous delivery and playback apparatus in accordance with claim 3, wherein said file calibrating mechanism aligns the streams with the other clients'"'"' streams and then returns the stream simultaneously to all clients.

6. The synchronous delivery and playback apparatus in accordance with claim 3, wherein said client application utilizes a formalized Internet time standard, said Internet time standard being the NTP.

7. The synchronous delivery and playback apparatus in accordance with claim 3, wherein said synchronous delivery and playback means further comprises an NTP timestamp, said timestamp is controlled by each of the clients, and constantly monitoring the NTP so as to continuously adjust the timing conditions.

8. The synchronous delivery and playback apparatus in accordance with claim 3, wherein said file calibrating mechanism further comprises an offset timestamp, said timestamp being the timing of that client after the stream has been synchronized with the other streams.

9. A method to provide synchronous delivery and playback of three or more electronic audio or video files from multiple clients, having differing arrival latencies, from clients from multiple locations, during an on-line session, the synchronous delivery and playback method comprising:

a. creating a session on each client;

b. allowing clients to request to join the session;

c. approving or denying the client'"'"'s request to join the session;

d. acquiring a Network Time Protocol (NTP) server and timestamp;

e. only after approval, joining the client to the session and time stamping the participant'"'"'s session;

f. enabling a client application, the client application calculating the NTP server'"'"'s timestamp and factoring in a delay time;

g. starting an NTP clock, the clock synchronized to the timestamp of its chosen NTP serve and is given to all clients;

h. connection by the client application to the client application of the other participants and determination of each client'"'"'s time differentials;

i. adjusting constantly the NTP timestamps to the changes in the network conditions;

j. buffering and synchronizing the clients'"'"' multimedia streams so that all streams are transmitted so as to arrive at the same time as the slowest stream;

k. creating a time stamped stream, said time stamped stream is buffered and synchronized multimedia streams;

l. utilizing the embedded time stamp within the transmitted streams to determine which stream has the greatest latency as compared to its ow NTP timestamp;

m. decoding all streams as they arrive at each client;

n. designating the stream with the greatest latency as the delay reference stream;

o. buffering all other streams until each stream'"'"'s time stamp matches that of the delay reference stream; and

p. rendering the all outgoing streams to all clients such that the client with the least latency receives its stream at the same time as the participant with the greatest latency.

10. The synchronous delivery and playback method in accordance with claim 9, wherein said synchronous delivery and playback method further aligns the synchronized stream so that each client receives the stream in sync.

11. The synchronous delivery and playback method in accordance with claim 9, wherein said synchronous delivery and playback method further comprises aligning the streams so that all streams appear simultaneously to the clients.

12. The synchronous delivery and playback method in accordance with claim 9, wherein said synchronous delivery and playback method utilizes a formalized Internet time standard, said time standard being the NTP.

13. The synchronous delivery and playback method in accordance with claim 9, wherein said synchronous delivery and playback method further comprises an NTP timestamp, said timestamp controlled by each of the participants, and constantly monitoring the NTP timestamp so as to continuously adjust the timing conditions.