Synchronization and mixing of audio and video streams in network-based video conferencing call systems

• US 8,954,178 B2
• Filed: 11/23/2011
• Issued: 02/10/2015
• Est. Priority Date: 09/30/2007
• Status: Active Grant

• Current Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Original Assignee: Optical Fusion Inc.
• Inventors: Thapa, Mukund N.
• Primary Examiner(s): MCCORD, PAUL C
• Application Number:
  US13/304,105
• Publication Number:
  US 20120069137A1
• Time in Patent Office: 1,175 Days
• Field of Search: 700/94
• US Class Current: 700/94
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/155 involving storage of or access to video conference sessions tracking arrangements for later retrieval of a computer conference content or participants activities H04L12/1831

Abstract

In one aspect, audio streams are added to a mix until the mix is either complete (i.e., all audio streams have been added) or the mix is closed early (i.e., before the mix is complete). In another aspect, audio and video streams are synchronized by playing back the audio stream and then synchronizing display of the video frames to the playback of the audio stream.

Claims

1. A computer-implemented method for producing a video conference display for a receiver participant, the method comprising:

receiving over a network one or more audio streams and one or more video streams from one or more sender participants, the audio streams divided into audio frames and the video stream(s) divided into video frames, wherein;

each audio frame retains in the composite stream an audio time marker from a corresponding sender participant, and

each video frame retains in a video stream a video time marker from a corresponding sender participant;

generating and playing a composite audio stream based on the one or more audio streams;

for each sender participant providing a video stream, matching playback of the video stream for each sender participant to the composite audio stream, the matching comprising;

repeatedly determining if a current frame of the video stream for a first sender participant should occur during a current audio frame of the composite audio stream by comparing the retained video time marker for the first sender participant to the retained audio time marker in the composite audio stream for the first sender participant;

if the current video frame should occur during a time period calculated for the current audio frame for the first sender participant based on the retained audio time marker for that participant, then playing the current video frame and moving to a next frame of the video stream;

if the current video frame should occur after said time period, then waiting; and

if the current video frame should have occurred before said time period, then skipping the current video frame and moving to a next frame of the video stream.

2. The computer-implemented method of claim 1 wherein the step of determining if a current frame of the video stream for that sender participant should occur during a current audio frame of the composite audio stream comprises:

determining whether a time stamp for the current video frame falls within a time tolerance of a time stamp for the current audio frame.

3. The computer-implemented method of claim 2 wherein two or more video streams are received from two or more sender participants.

4. A computer-implemented method for playback of a recorded video conference call between two or more participants, the method comprising:

accessing one or more audio streams and one or more video streams from the participants, the audio streams divided into audio frames and the video stream(s) divided into video frames, wherein;

each audio frame retains in the composite stream an audio time marker from a corresponding sender participant, and

each video frame retains in a video stream a video time marker from a corresponding sender participant;

generating and playing a playback audio stream based on the one or more audio streams;

for each participant having provided a video stream, matching playback of the video stream for each participant to the playback audio stream, the matching comprising;

repeatedly determining if a current frame of the video stream for a sender participant should occur during a current audio frame of the playback audio stream by comparing the retained video time marker for the first participant to the retained audio time marker in the playback audio stream for the sender participant;

if the current video frame should occur during a time period calculated for the current audio frame for the first participant based on the retained audio time marker for that participant, then playing the current video frame and moving to a next frame of the video stream;

if the current video frame should occur after said time period, then waiting; and

if the current video frame should have occurred before said time period, then skipping the current video frame and moving to a next frame of the video stream.

5. The computer-implemented method of claim 4 further comprising:

during a fast forward mode;

advancing the audio time stamp faster than real-time; and

performing, for the video stream of only one of the participants, the step of determining if a current frame of the video stream for that participant should occur during a current audio frame; and

after the fast forward mode, synchronizing the video streams of the other participants to the time stamp of the current audio frame.

6. The computer-implemented method of claim 4 further comprising:

during a fast forward mode;

advancing the audio time stamp faster than real-time; and

performing, for the video streams of all participants, the step of determining if a current frame of the video stream for that participant should occur during a current audio frame.

7. The computer-implemented method of claim 4 further comprising:

during a rewind mode;

advancing the audio time stamps in a reverse direction; and

performing, for the video stream of only one of the participants, the step of determining if a current frame of the video stream for that participant should occur during a current audio frame; and

after the rewind mode, synchronizing the video streams of the other participants to the time stamp of the current audio frame.

8. The computer-implemented method of claim 4 further comprising:

during a rewind mode;

advancing the audio time stamps in a reverse direction; and

performing, for the video streams of all participants, the step of determining if a current frame of the video stream for that participant should occur during a current audio frame.

9. The computer-implemented method of claim 4 further comprising:

during a seek mode;

changing the audio time stamp to a seek location; and

synchronizing the video time stamps for the video streams to the audio time stamp of the seek location.

10. The computer-implemented method of claim 9 wherein the step of synchronizing the video time stamps comprises stepping through the video time stamps until the video time stamps are synchronized with the audio time stamp of the seek location.

11. The computer-implemented method of claim 9 wherein the step of synchronizing the video time stamps comprises using a binary search to synchronize the video time stamps with the audio time stamp of the seek location.

12. The computer-implemented method of claim 4 further comprising setting a video time stamp=NULL to indicate moving to a next video frame.

13. The computer-implemented method of claim 4 wherein the audio streams and the video streams are indexed, and the indices are used during irregular playback.

14. The computer-implemented method of claim 4 wherein a look ahead technique is used during irregular playback.

15. The computer-implemented method of claim 4 further comprising adjusting the number of video streams played during irregular playback based on the compute resources available for playback.

16. The computer-implemented method of claim 4 wherein there are two or more audio streams received from two or more participants, and the audio streams are synchronized relative to each other.

17. A computer-implemented method for synchronizing an audio stream and a video stream, the method comprising:

playing the audio stream as a series of audio chunks, each audio chunk retaining an audio time marker from a corresponding sender of the audio stream and the video stream;

matching playback of a video frame of the video stream from the sender to an audio chuck of the audio stream from the sender, the current frame of the video stream retaining a video time marker from the sender of the audio stream and the video stream, the matching comprising;

determining if a current frame of the video stream should occur during the playing of a current audio chunk by comparing the retained video time marker for the sender participant to the retained audio time marker for the sender participant;

if the current video frame should occur during a time period calculated for the current audio chunk, then playing the current video frame and moving to a next frame of the video stream;

if the current video frame should occur after said time period, then waiting; and

if the current video frame should have occurred before said time period, then skipping the current video frame and moving to a next frame of the video stream.

18. The computer-implemented method of claim 17 wherein the step of determining if a current frame of the video stream should occur during the playing of a current audio chunk comprises:

determining a time marker for the current video frame;

determining a start time and an end time for the time period calculated for the audio chunk; and

determining whether the time marker falls between the start time and the end time.

19. The computer-implemented method of claim 17 wherein the step of determining a start time and an end time comprises:

determining a nominal start time and a nominal end time for the audio chunk;

adjusting the nominal start time by a tolerance; and

adjusting the nominal end time by another tolerance.

20. A video conference call client for synchronizing and mixing audio and video streams received over a network from one or more sender participants, the video conference call client comprising:

an audio player module, for playing a composite audio stream based on one or more audio streams received from sender participants, each audio frame of the one or more audio streams retaining in the composite stream an audio time marker from a corresponding sender participant;

a video player module, for playing one or more video streams received from sender participants, each video frame of each video stream retaining a video time marker from a corresponding sender participant; and

an audio-video synchronization module for synchronizing the video streams to the audio stream, the synchronizing comprising;

comparing the retained video time marker for a first sender participant to the retained audio time marker in the composite audio stream for the first sender participant;

if the current video frame should occur during a time period calculated for the current audio frame, then playing the current video frame and moving to a next frame of the video stream;

if the current video frame should occur after said time period, then waiting; and

if the current video frame should have occurred before said time period, then skipping the current video frame and moving to a next frame of the video stream.

21. A video conference call client for playback of a recorded video conference call between two or more participants, the video conference call client comprising:

an audio player module, for playing one or more audio streams from the participants;

a video player module, for playing one or more video streams from the participants; and

an audio-video synchronization module for synchronizing the video streams to the audio streams, wherein the audio streams are divided into audio frames and the video stream(s) are divided into video frames, wherein;

each audio frame retains an audio time marker from a corresponding sender participant, and

each video frame retains in a video stream a video time marker from a sender participant; and

for each sender providing a video stream, the audio-video synchronization module matching playback of a current frame of the video stream for each sender participant to a current audio frame for the corresponding sender participant, wherein the audio-video synchronization module;

repeatedly determines if a current frame of the video stream for that participant should occur during a current audio by comparing a retained video time marker for the current frame of the video stream to the current frame of the audio stream for that sender participant;

if the current video frame should occur during a time period calculated for the current audio frame, then playing the current video frame and moving to a next frame of the video stream;

if the current video frame should occur after said time period, then waiting; and

if the current video frame should have occurred before said time period, then skipping the current video frame and moving to a next frame of the video stream.