Synchronization and Mixing of Audio and Video Streams in Network-Based Video Conferencing Call Systems

• US 20090088880A1
• Filed: 09/30/2008
• Published: 04/02/2009
• Est. Priority Date: 09/30/2007
• Status: Active Grant

• Current Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Original Assignee: Optical Fusion Inc.
• Inventors: Thapa, Mukund N.
• Granted Patent: US 8,583,268 B2
• Time in Patent Office: Days
• US Class Current: 700/94
• CPC Class Codes: H04L 65/1089 by adding media; by removing media
• CPC Class Codes: H04L 65/1093 by adding participants; by removing participants
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/4038 with floor control
• CPC Class Codes: H04L 65/4046 with distributed floor control
• CPC Class Codes: H04L 65/764 at the destination reformatting of additional data in video clients H04N21/4355
• CPC Class Codes: H04M 3/562 where the conference facilities are distributed
• CPC Class Codes: H04M 3/564 whereby the feature is a sub-conference
• CPC Class Codes: H04M 3/568 audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/155 involving storage of or access to video conference sessions tracking arrangements for later retrieval of a computer conference content or participants activities H04L12/1831

Abstract

In one aspect, audio streams are added to a mix until the mix is either complete (i.e., all audio streams have been added) or the mix is closed early (i.e., before the mix is complete). In another aspect, audio and video streams are synchronized by playing back the audio stream and then synchronizing display of the video frames to the playback of the audio stream.

Claims

1. A computer-implemented method for mixing audio streams from N senders, N at least two, the method comprising:

receiving the N audio streams over a network, the audio streams divided into audio chunks;

buffering the audio chunks;

opening a mix;

cycling through the N senders, for each sender on each cycle;

if the sender is not yet in the mix, adding the sender'"'"'s audio chunk to the mix if the missing audio chunk is available from the buffer; and

closing the mix if either audio chunks from all N senders are in the mix or if a predetermined early close condition is met.

2. The computer-implemented method of claim 1 wherein:

the step of opening a mix comprises resetting a counter for the number of senders in the mix;

the step of cycling through the N senders further comprises;

if the sender'"'"'s audio chunk is added to the mix in a cycle, incrementing the counter; and

the step of closing the mix further comprises closing the mix if the counter indicates N senders are in the mix.

3. The computer-implemented method of claim 1 wherein the step of cycling through the N senders further comprises:

on each cycle, if the sender'"'"'s audio chunk is already in the mix and the sender has at least one additional audio chunk buffered, incrementing a wait counter for that sender;

wherein the predetermined early close condition is based on the value of the wait counter(s).

4. The computer-implemented method of claim 3 wherein the predetermined early close condition occurs if any wait counter reaches a predetermined maximum value.

5. The computer-implemented method of claim 4 wherein the predetermined maximum value is three.

6. The computer-implemented method of claim 3 wherein:

the step of cycling through the N senders further comprises;

if the sender'"'"'s audio chunk is added to the mix in a cycle, decrementing the wait counter for that sender if the wait counter is greater than 0.

7. The computer-implemented method of claim 3 wherein:

the step of opening a mix comprises resetting the wait counters for all senders.

8. The computer-implemented method of claim 1 wherein each audio chunk contains audio of 40 ms duration.

9. The computer-implemented method of claim 1 wherein:

the step of receiving the N audio streams over a network comprises a central server receiving the N audio streams over a network from sender clients; and

the steps of buffering, opening a mix, cycling through the senders and closing the mix occur at the central server.

10. The computer-implemented method of claim 1 wherein:

the step of receiving the N audio streams over a network comprises a receiver client receiving the N audio streams over a network from a central server, the central server having received the N audio streams from sender clients; and

the steps of buffering, opening a mix, cycling through the senders and closing the mix occur at the receiver client.

11. The computer-implemented method of claim 1 wherein:

the step of receiving the N audio streams over a network comprises a receiver client receiving the N audio streams over a network from sender clients; and

the steps of buffering, opening a mix, cycling through the senders and closing the mix occur at the receiver client.

12. A computer-implemented method for mixing audio streams from N senders, N at least two, the method comprising:

opening a mix;

receiving the N audio streams over a network, the audio streams divided into audio chunks;

as each sender'"'"'s audio chunk is received;

if the sender is not yet in the mix and the received audio chunk is the correct audio chunk for the mix, adding the sender'"'"'s audio chunk to the mix;

otherwise, buffering the sender'"'"'s audio chunk for a future mix; and

closing the mix if either audio chunks from all N senders are in the mix or if a predetermined early close condition is met.

13. The computer-implemented method of claim 12 wherein:

the step of opening a mix comprises resetting a counter for the number of senders in the mix;

the step of adding a sender'"'"'s audio chunk to the mix further comprises incrementing the counter; and

the step of closing the mix further comprises closing the mix if the counter indicates N senders are in the mix.

14. The computer-implemented method of claim 12 wherein the step of buffering the sender'"'"'s audio chunk for a future mix further comprises:

incrementing a queue counter for that sender;

wherein the predetermined early close condition is based on the value of the queue counter(s).

15. The computer-implemented method of claim 14 wherein the predetermined early close condition occurs if any queue counter reaches a predetermined maximum value.

16. The computer-implemented method of claim 14 further comprising:

upon closing the mix, opening a new mix and determining whether any of the previously buffered audio chunks should be added to the new mix.

17. A computer-implemented method for synchronizing an audio stream and a video stream, the method comprising:

playing the audio stream as a series of audio chunks;

determining if a current frame of the video stream should occur during the playing of a current audio chunk;

if the current frame should occur during a time period calculated for the current audio chunk, then displaying the current frame and moving to a next frame;

if the current frame should occur after said time period, then waiting; and

if the current frame should have occurred before said time period, then discarding the current frame and moving to a next frame.

18. The computer-implemented method of claim 17 wherein the step of determining if a current frame of the video stream should occur during the playing of a current audio chunk comprises:

determining a time marker for the current video frame;

determining a start time and an end time for the time period calculated for the audio chunk; and

determining whether the time marker falls between the start time and the end time.

19. The computer-implemented method of claim 18 wherein the step of determining a start time and an end time comprises:

determining a nominal start time and a nominal end time for the audio chunk;

adjusting the nominal start time by a tolerance; and

adjusting the nominal end time by another tolerance.

20. A computer-implemented method for producing a video conference display for a receiver participant, the method comprising:

receiving over a network two or more audio streams and one or more video streams from two or more sender participants, the audio streams divided into audio chunks and the video stream(s) divided into video frames;

generating and playing a composite audio stream of the received audio streams, said generating and playing comprising repeating the steps of;

opening a mix;

adding sender'"'"'s audio chunks to the mix;

closing the mix if either audio chunks from all senders are in the mix or if a predetermined early close condition is met; and

playing the mix; and

repeatedly determining if a current frame of the video stream should occur during the playing of a current mix;

if the current frame should occur during a time period calculated for the current mix, then displaying the current frame and moving to a next frame;

if the current frame should occur after said time period, then waiting; and

if the current frame should have occurred before said time period, then discarding the current frame and moving to a next frame.

21. The computer-implemented method of claim 20 wherein the step of generating and playing a composite audio stream further comprises:

buffering the received audio chunks;

wherein the step of adding sender'"'"'s audio chunks to the mix comprises cycling through the senders, and for each sender on each cycle, if the sender is not yet in the mix, adding the sender'"'"'s audio chunk to the mix if the missing audio chunk is available from the buffer.

22. The computer-implemented method of claim 20 wherein the step of generating and playing a composite audio stream further comprises:

as each sender'"'"'s audio chunk is received, if the sender is not yet in the mix and the received audio chunk is the correct audio chunk for the mix, adding the sender'"'"'s audio chunk to the mix; and

, otherwise, buffering the sender'"'"'s audio chunk for a future mix.