Synchronization and mixing of audio and video streams in network-based video conferencing call systems
First Claim
Patent Images
1. A computer-implemented method for producing a video conference display for a receiver participant, the method comprising:
- receiving over a network one or more audio streams and one or more video streams from one or more sender participants, the audio streams divided into audio frames and the video stream(s) divided into video frames, wherein;
each audio frame retains in the composite stream an audio time marker from a corresponding sender participant, andeach video frame retains in a video stream a video time marker from a corresponding sender participant;
generating and playing a composite audio stream based on the one or more audio streams;
for each sender participant providing a video stream, matching playback of the video stream for each sender participant to the composite audio stream, the matching comprising;
repeatedly determining if a current frame of the video stream for a first sender participant should occur during a current audio frame of the composite audio stream by comparing the retained video time marker for the first sender participant to the retained audio time marker in the composite audio stream for the first sender participant;
if the current video frame should occur during a time period calculated for the current audio frame for the first sender participant based on the retained audio time marker for that participant, then playing the current video frame and moving to a next frame of the video stream;
if the current video frame should occur after said time period, then waiting; and
if the current video frame should have occurred before said time period, then skipping the current video frame and moving to a next frame of the video stream.
2 Assignments
0 Petitions
Accused Products
Abstract
In one aspect, audio streams are added to a mix until the mix is either complete (i.e., all audio streams have been added) or the mix is closed early (i.e., before the mix is complete). In another aspect, audio and video streams are synchronized by playing back the audio stream and then synchronizing display of the video frames to the playback of the audio stream.
-
Citations
21 Claims
-
1. A computer-implemented method for producing a video conference display for a receiver participant, the method comprising:
-
receiving over a network one or more audio streams and one or more video streams from one or more sender participants, the audio streams divided into audio frames and the video stream(s) divided into video frames, wherein; each audio frame retains in the composite stream an audio time marker from a corresponding sender participant, and each video frame retains in a video stream a video time marker from a corresponding sender participant; generating and playing a composite audio stream based on the one or more audio streams; for each sender participant providing a video stream, matching playback of the video stream for each sender participant to the composite audio stream, the matching comprising; repeatedly determining if a current frame of the video stream for a first sender participant should occur during a current audio frame of the composite audio stream by comparing the retained video time marker for the first sender participant to the retained audio time marker in the composite audio stream for the first sender participant; if the current video frame should occur during a time period calculated for the current audio frame for the first sender participant based on the retained audio time marker for that participant, then playing the current video frame and moving to a next frame of the video stream; if the current video frame should occur after said time period, then waiting; and if the current video frame should have occurred before said time period, then skipping the current video frame and moving to a next frame of the video stream. - View Dependent Claims (2, 3)
-
-
4. A computer-implemented method for playback of a recorded video conference call between two or more participants, the method comprising:
-
accessing one or more audio streams and one or more video streams from the participants, the audio streams divided into audio frames and the video stream(s) divided into video frames, wherein; each audio frame retains in the composite stream an audio time marker from a corresponding sender participant, and each video frame retains in a video stream a video time marker from a corresponding sender participant; generating and playing a playback audio stream based on the one or more audio streams; for each participant having provided a video stream, matching playback of the video stream for each participant to the playback audio stream, the matching comprising; repeatedly determining if a current frame of the video stream for a sender participant should occur during a current audio frame of the playback audio stream by comparing the retained video time marker for the first participant to the retained audio time marker in the playback audio stream for the sender participant; if the current video frame should occur during a time period calculated for the current audio frame for the first participant based on the retained audio time marker for that participant, then playing the current video frame and moving to a next frame of the video stream; if the current video frame should occur after said time period, then waiting; and if the current video frame should have occurred before said time period, then skipping the current video frame and moving to a next frame of the video stream. - View Dependent Claims (5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16)
-
-
17. A computer-implemented method for synchronizing an audio stream and a video stream, the method comprising:
-
playing the audio stream as a series of audio chunks, each audio chunk retaining an audio time marker from a corresponding sender of the audio stream and the video stream; matching playback of a video frame of the video stream from the sender to an audio chuck of the audio stream from the sender, the current frame of the video stream retaining a video time marker from the sender of the audio stream and the video stream, the matching comprising; determining if a current frame of the video stream should occur during the playing of a current audio chunk by comparing the retained video time marker for the sender participant to the retained audio time marker for the sender participant; if the current video frame should occur during a time period calculated for the current audio chunk, then playing the current video frame and moving to a next frame of the video stream; if the current video frame should occur after said time period, then waiting; and if the current video frame should have occurred before said time period, then skipping the current video frame and moving to a next frame of the video stream. - View Dependent Claims (18, 19)
-
-
20. A video conference call client for synchronizing and mixing audio and video streams received over a network from one or more sender participants, the video conference call client comprising:
-
an audio player module, for playing a composite audio stream based on one or more audio streams received from sender participants, each audio frame of the one or more audio streams retaining in the composite stream an audio time marker from a corresponding sender participant; a video player module, for playing one or more video streams received from sender participants, each video frame of each video stream retaining a video time marker from a corresponding sender participant; and an audio-video synchronization module for synchronizing the video streams to the audio stream, the synchronizing comprising; comparing the retained video time marker for a first sender participant to the retained audio time marker in the composite audio stream for the first sender participant; if the current video frame should occur during a time period calculated for the current audio frame, then playing the current video frame and moving to a next frame of the video stream; if the current video frame should occur after said time period, then waiting; and if the current video frame should have occurred before said time period, then skipping the current video frame and moving to a next frame of the video stream.
-
-
21. A video conference call client for playback of a recorded video conference call between two or more participants, the video conference call client comprising:
-
an audio player module, for playing one or more audio streams from the participants; a video player module, for playing one or more video streams from the participants; and an audio-video synchronization module for synchronizing the video streams to the audio streams, wherein the audio streams are divided into audio frames and the video stream(s) are divided into video frames, wherein; each audio frame retains an audio time marker from a corresponding sender participant, and each video frame retains in a video stream a video time marker from a sender participant; and for each sender providing a video stream, the audio-video synchronization module matching playback of a current frame of the video stream for each sender participant to a current audio frame for the corresponding sender participant, wherein the audio-video synchronization module; repeatedly determines if a current frame of the video stream for that participant should occur during a current audio by comparing a retained video time marker for the current frame of the video stream to the current frame of the audio stream for that sender participant; if the current video frame should occur during a time period calculated for the current audio frame, then playing the current video frame and moving to a next frame of the video stream; if the current video frame should occur after said time period, then waiting; and if the current video frame should have occurred before said time period, then skipping the current video frame and moving to a next frame of the video stream.
-
Specification