Synchronization and mixing of audio and video streams in network-based video conferencing call systems

• US 8,583,268 B2
• Filed: 09/30/2008
• Issued: 11/12/2013
• Est. Priority Date: 09/30/2007
• Status: Active Grant

• Current Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Original Assignee: Optical Fusion Inc.
• Inventors: Thapa, Mukund N.
• Primary Examiner(s): MCCORD, PAUL C
• Application Number:
  US12/242,358
• Publication Number:
  US 20090088880A1
• Time in Patent Office: 1,869 Days
• Field of Search: 700/94
• US Class Current: 700/94
• CPC Class Codes: H04L 65/1089 by adding media; by removing media
• CPC Class Codes: H04L 65/1093 by adding participants; by removing participants
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/4038 with floor control
• CPC Class Codes: H04L 65/4046 with distributed floor control
• CPC Class Codes: H04L 65/764 at the destination reformatting of additional data in video clients H04N21/4355
• CPC Class Codes: H04M 3/562 where the conference facilities are distributed
• CPC Class Codes: H04M 3/564 whereby the feature is a sub-conference
• CPC Class Codes: H04M 3/568 audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/155 involving storage of or access to video conference sessions tracking arrangements for later retrieval of a computer conference content or participants activities H04L12/1831

Abstract

In one aspect, audio streams are added to a mix until the mix is either complete (i.e., all audio streams have been added) or the mix is closed early (i.e., before the mix is complete). In another aspect, audio and video streams are synchronized by playing back the audio stream and then synchronizing display of the video frames to the playback of the audio stream.

Claims

1. A computer-implemented method for mixing audio streams from N senders, N at least two, the method comprising:

receiving the N audio streams over a network, the audio streams divided into audio chunks;

buffering the audio chunks;

opening a mix;

cycling through the N senders, for each sender;

if the sender is not yet in the mix, and if a missing audio chunk is available from the buffer, adding the sender'"'"'s audio chunk to the mix;

if the sender is not yet in the mix and if the audio chunk is not available from the buffer, cycling through the remaining senders and rechecking whether the missing chunk is available from the buffer;

on each cycle, if the sender'"'"'s audio chunk is already in the mix and the sender has at least one additional audio chunk buffered, incrementing a wait counter for that sender;

wherein a predetermined early close condition is based on a value of the wait counter(s); and

closing the mix if either audio chunks from all N senders are in the mix or if the predetermined early close condition is met.

2. The computer-implemented method of claim 1 wherein:

the step of opening a mix comprises resetting a counter for the number of senders in the mix;

the step of cycling through the N senders further comprises;

if the sender'"'"'s audio chunk is added to the mix in a cycle, incrementing the counter; and

the step of closing the mix further comprises closing the mix if the counter indicates N senders are in the mix.

3. The computer-implemented method of claim 1 wherein the predetermined early close condition occurs if any wait counter reaches a predetermined maximum value.

4. The computer-implemented method of claim 3 wherein the predetermined maximum value is three.

5. The computer-implemented method of claim 1 wherein:

the step of cycling through the N senders further comprises;

if the sender'"'"'s audio chunk is added to the mix in a cycle, decrementing the wait counter for that sender if the wait counter is greater than 0.

6. The computer-implemented method of claim 1 wherein:

the step of opening a mix comprises resetting the wait counters for all senders.

7. The computer-implemented method of claim 1 wherein each audio chunk contains audio of 40 ms duration.

8. The computer-implemented method of claim 1 wherein:

the step of receiving the N audio streams over a network comprises a receiver client receiving the N audio streams over a network from sender clients; and

the steps of buffering, opening a mix, cycling through the senders and closing the mix occur at the receiver client.

9. A computer-implemented method for mixing audio streams from N senders, N at least two, the method comprising:

opening a mix;

receiving the N audio streams over a network, the audio streams divided into audio chunks;

if an audio chunk for the sender is not yet in the mix and has not yet been received, cycling through the remaining senders, and rechecking whether the chunk from the missing sender has been received;

as each sender'"'"'s audio chunk is received;

if the sender is not yet in the mix and the received audio chunk is the correct audio chunk for the mix, adding the sender'"'"'s audio chunk to the mix;

otherwise, buffering the sender'"'"'s audio chunk for a future mix and incrementing a wait counter for the sender, wherein a predetermined early close condition is based on a value of the wait counter(s); and

closing the mix if either audio chunks from all N senders are in the mix or if the predetermined early close condition is met.

10. The computer-implemented method of claim 9 wherein:

the step of opening a mix comprises resetting a counter for the number of senders in the mix;

the step of adding a sender'"'"'s audio chunk to the mix further comprises incrementing the counter; and

the step of closing the mix further comprises closing the mix if the counter indicates N senders are in the mix.

11. The computer-implemented method of claim 9 wherein the step of buffering the sender'"'"'s audio chunk for a future mix further comprises:

incrementing a queue counter for that sender;

wherein the predetermined early close condition is based on the value of the queue counter(s).

12. The computer-implemented method of claim 11 wherein the predetermined early close condition occurs if any queue counter reaches a predetermined maximum value.

13. The computer-implemented method of claim 11 further comprising:

upon closing the mix, opening a new mix and determining whether any of the previously buffered audio chunks should be added to the new mix.