Synchronization and Mixing of Audio and Video Streams in Network-Based Video Conferencing Call Systems

• US 20120069137A1
• Filed: 11/23/2011
• Published: 03/22/2012
• Est. Priority Date: 09/30/2007
• Status: Active Grant

• Current Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Original Assignee: Optical Fusion Inc.
• Inventors: Thapa, Mukund N.
• Granted Patent: US 8,954,178 B2
• Time in Patent Office: Days
• US Class Current: 348/14.08
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/155 involving storage of or access to video conference sessions tracking arrangements for later retrieval of a computer conference content or participants activities H04L12/1831

Abstract

In one aspect, audio streams are added to a mix until the mix is either complete (i.e., all audio streams have been added) or the mix is closed early (i.e., before the mix is complete). In another aspect, audio and video streams are synchronized by playing back the audio stream and then synchronizing display of the video frames to the playback of the audio stream.

Claims

1. A computer-implemented method for producing a video conference display for a receiver participant, the method comprising:

receiving over a network one or more audio streams and one or more video streams from one or more sender participants, the audio streams divided into audio frames and the video stream(s) divided into video frames;

generating and playing a composite audio stream based on the one or more audio streams;

for each sender participant providing a video stream, repeatedly determining if a current frame of the video stream for that sender participant should occur during a current audio frame of the composite audio stream;

if the current video frame should occur during a time period calculated for the current audio frame, then playing the current video frame and moving to a next frame of the video stream;

if the current video frame should occur after said time period, then waiting; and

if the current video frame should have occurred before said time period, then skipping the current video frame and moving to a next frame of the video stream.

2. The computer-implemented method of claim 1 wherein the step of determining if a current frame of the video stream for that sender participant should occur during a current audio frame of the composite audio stream comprises:

determining whether a time stamp for the current video frame falls within a time tolerance of a time stamp for the current audio frame.

3. The computer-implemented method of claim 2 wherein two or more video streams are received from two or more sender participants.

4. A computer-implemented method for playback of a recorded video conference call between two or more participants, the method comprising:

accessing one or more audio streams and one or more video streams from the participants, the audio streams divided into audio frames and the video stream(s) divided into video frames;

generating and playing a playback audio stream based on the one or more audio streams;

for each participant, repeatedly determining if a current frame of the video stream for that participant should occur during a current audio frame of the playback audio stream;

if the current video frame should occur during a time period calculated for the current audio frame, then playing the current video frame and moving to a next frame of the video stream;

if the current video frame should occur after said time period, then waiting; and

if the current video frame should have occurred before said time period, then skipping the current video frame and moving to a next frame of the video stream.

5. The computer-implemented method of claim 4 further comprising:

during a fast forward mode;

advancing the audio time stamp faster than real-time; and

performing, for the video stream of only one of the participants, the step of determining if a current frame of the video stream for that participant should occur during a current audio frame; and

after the fast forward mode, synchronizing the video streams of the other participants to the time stamp of the current audio frame.

6. The computer-implemented method of claim 4 further comprising:

during a fast forward mode;

advancing the audio time stamp faster than real-time; and

performing, for the video streams of all participants, the step of determining if a current frame of the video stream for that participant should occur during a current audio frame.

7. The computer-implemented method of claim 4 further comprising:

during a rewind mode;

advancing the audio time stamps in a reverse direction; and

performing, for the video stream of only one of the participants, the step of determining if a current frame of the video stream for that participant should occur during a current audio frame; and

after the rewind mode, synchronizing the video streams of the other participants to the time stamp of the current audio frame.

8. The computer-implemented method of claim 4 further comprising:

during a rewind mode;

advancing the audio time stamps in a reverse direction; and

performing, for the video streams of all participants, the step of determining if a current frame of the video stream for that participant should occur during a current audio frame.

9. The computer-implemented method of claim 4 further comprising:

during a seek mode;

changing the audio time stamp to a seek location; and

synchronizing the video time stamps for the video streams to the audio time stamp of the seek location.

10. The computer-implemented method of claim 9 wherein the step of synchronizing the video time stamps comprises stepping through the video time stamps until the video time stamps are synchronized with the audio time stamp of the seek location.

11. The computer-implemented method of claim 9 wherein the step of synchronizing the video time stamps comprises using a binary search to synchronize the video time stamps with the audio time stamp of the seek location.

12. The computer-implemented method of claim 4 further comprising setting a video time stamp=NULL to indicate moving to a next video frame.

13. The computer-implemented method of claim 4 wherein the audio streams and the video streams are indexed, and the indices are used during irregular playback.

14. The computer-implemented method of claim 4 wherein a look ahead technique is used during irregular playback.

15. The computer-implemented method of claim 4 further comprising adjusting the number of video streams played during irregular playback based on the compute resources available for playback.

16. The computer-implemented method of claim 4 wherein there are two or more audio streams received from two or more participants, and the audio streams are synchronized relative to each other.

18. The computer-implemented method of claim 16 wherein the step of determining if a current frame of the video stream should occur during the playing of a current audio chunk comprises:

determining a time marker for the current video frame;

determining a start time and an end time for the time period calculated for the audio chunk; and

determining whether the time marker falls between the start time and the end time.

17. A computer-implemented method for synchronizing an audio stream and a video stream, the method comprising:

playing the audio stream as a series of audio chunks;

determining if a current frame of the video stream should occur during the playing of a current audio chunk;

if the current video frame should occur during a time period calculated for the current audio chunk, then playing the current video frame and moving to a next frame of the video stream;

if the current video frame should occur after said time period, then waiting; and

if the current video frame should have occurred before said time period, then skipping the current video frame and moving to a next frame of the video stream.

19. The computer-implemented method of claim 17 wherein the step of determining a start time and an end time comprises:

determining a nominal start time and a nominal end time for the audio chunk;

adjusting the nominal start time by a tolerance; and

adjusting the nominal end time by another tolerance.

20. A video conference call client for synchronizing and mixing audio and video streams received over a network from one or more sender participants, the video conference call client comprising:

an audio player module, for playing a composite audio stream based on one or more audio streams received from sender participants;

a video player module, for playing one or more video streams received from sender participants; and

an audio-video synchronization module for synchronizing the video streams to the audio stream, wherein the audio streams are divided into audio frames and the video stream(s) are divided into video frames; and

for each sender participant providing a video stream, the audio-video synchronization module repeatedly determines if a current frame of the video stream for that participant should occur during a current audio frame of the composite audio stream;

if the current video frame should occur during a time period calculated for the current audio frame, then playing the current video frame and moving to a next frame of the video stream;

if the current video frame should occur after said time period, then waiting; and

if the current video frame should have occurred before said time period, then skipping the current video frame and moving to a next frame of the video stream.

21. A video conference call client for playback of a recorded video conference call between two or more participants, the video conference call client comprising:

an audio player module, for playing one or more audio streams from the participants;

a video player module, for playing one or more video streams from the participants; and

an audio-video synchronization module for synchronizing the video streams to the audio streams, wherein the audio streams are divided into audio frames and the video stream(s) are divided into video frames; and

for each sender providing a video stream, the audio-video synchronization module repeatedly determines if a current frame of the video stream for that participant should occur during a current audio frame of the composite audio stream;

if the current video frame should occur during a time period calculated for the current audio frame, then playing the current video frame and moving to a next frame of the video stream;

if the current video frame should occur after said time period, then waiting; and

if the current video frame should have occurred before said time period, then skipping the current video frame and moving to a next frame of the video stream.