Synchronization and mixing of audio and video streams in network-based video conferencing call systems

• US 9,654,537 B2
• Filed: 03/04/2014
• Issued: 05/16/2017
• Est. Priority Date: 09/30/2007
• Status: Active Grant

• Current Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Original Assignee: Optical Fusion Inc.
• Inventors: Thapa, Mukund N.
• Primary Examiner(s): MCCORD, PAUL C
• Application Number:
  US14/196,703
• Publication Number:
  US 20140184735A1
• Time in Patent Office: 1,169 Days
• Field of Search: 700 94
• CPC Class Codes: H04L 65/1089 by adding media; by removing media
• CPC Class Codes: H04L 65/1093 by adding participants; by removing participants
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/4038 with floor control
• CPC Class Codes: H04L 65/4046 with distributed floor control
• CPC Class Codes: H04L 65/764 at the destination reformatting of additional data in video clients H04N21/4355
• CPC Class Codes: H04M 3/562 where the conference facilities are distributed
• CPC Class Codes: H04M 3/564 whereby the feature is a sub-conference
• CPC Class Codes: H04M 3/568 audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/155 involving storage of or access to video conference sessions tracking arrangements for later retrieval of a computer conference content or participants activities H04L12/1831

Abstract

In one aspect, audio streams are added to a mix until the mix is either complete (i.e., all audio streams have been added) or the mix is closed early (i.e., before the mix is complete). In another aspect, audio and video streams are synchronized by playing back the audio stream and then synchronizing display of the video frames to the playback of the audio stream.

Claims

1. A computer-implemented method for synchronizing an audio stream and a video stream, the method comprising:

playing the audio stream as a series of audio chunks according to time markers for the audio chunks;

based on a comparison of a time marker for a current frame of the video stream to a calculated time period for playing of a current audio chunk, determining if the current frame of the video stream should occur during the playing of the current audio chunk;

if the current frame should occur during the time period calculated for the current audio chunk, then displaying the current frame and moving to a next frame;

if the current frame should occur after the time period, then waiting; and

if the current frame should have occurred before the time period, then discarding the current frame and moving to a next frame,

wherein the step of determining if a current frame of the video stream should occur during the playing of the current audio chunk comprises;

determining a time marker for the current video frame;

determining a start time and an end time for the time period calculated for the audio chunk, the determining comprising;

determining a nominal start time and a nominal end time for the audio chunk,

adjusting the nominal start time by a tolerance, and

adjusting the nominal end time by another tolerance; and

determining whether the time marker falls between the start time and the end time.

2. A computer program product for synchronizing an audio stream and a video stream, wherein the computer program product is stored on a non-transitory computer-readable medium that includes instructions that, when loaded into memory, cause a processor to perform a method, the method comprising:

playing the audio stream as a series of audio chunks according to time markers for the audio chunks;

based on a comparison of a time marker for a current frame of the video stream to a calculated time period for playing of a current audio chunk, determining if the current frame of the video stream should occur during the playing of the current audio chunk;

if the current frame should occur during the time period calculated for the current audio chunk, then displaying the current frame and moving to a next frame;

if the current frame should occur after the time period, then waiting; and

if the current frame should have occurred before the time period, then discarding the current frame and moving to a next frame,

wherein the step of determining if a current frame of the video stream should occur during the playing of the current audio chunk comprises;

determining a time marker for the current video frame;

determining a start time and an end time for the time period calculated for the audio chunk, the determining comprising;

determining a nominal start time and a nominal end time for the audio chunk,

adjusting the nominal start time by a tolerance, and

adjusting the nominal end time by another tolerance; and

determining whether the time marker falls between the start time and the end time.

3. A non-transitory computer-readable medium that includes instructions that, when loaded into memory, cause a processor to perform a method for synchronizing an audio stream and a video stream, the method comprising:

playing the audio stream as a series of audio chunks according to time markers for the audio chunks;

based on a comparison of a time marker for a current frame of the video stream to a calculated time period for playing of a current audio chunk, determining if the current frame of the video stream should occur during the playing of the current audio chunk;

if the current frame should occur during the time period calculated for the current audio chunk, then displaying the current frame and moving to a next frame;

if the current frame should occur after the time period, then waiting; and

if the current frame should have occurred before the time period, then discarding the current frame and moving to a next frame,

wherein the step of determining if a current frame of the video stream should occur during the playing of the current audio chunk comprises;

determining a time marker for the current video frame;

determining a start time and an end time for the time period calculated for the audio chunk, the determining comprising;

determining a nominal start time and a nominal end time for the audio chunk,

adjusting the nominal start time by a tolerance, and

adjusting the nominal end time by another tolerance; and

determining whether the time marker falls between the start time and the end time.