Lip synchronization for audio/video transmissions over a network

• US 7,693,190 B2
• Filed: 11/22/2006
• Issued: 04/06/2010
• Est. Priority Date: 11/22/2006
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Kadiyala, Madhavi, Baird, Randall B., Sarkar, Shantanu, Firestone, Scott, Bangalore, Manjunath S.
• Primary Examiner(s): Ngo; Ricky
• Assistant Examiner(s): OVEISSI, MANSOUR
• Application Number:
  US11/603,849
• Publication Number:
  US 20080117937A1
• Time in Patent Office: 1,231 Days
• Field of Search: 348/14.09, 348/14.08, 348/14.12, 348/466, 348/467, 715/500.1, 715/716, 370/503, 386/12, 386/17, 386/52
• US Class Current: 370/503
• CPC Class Codes: H04N 21/23406 involving management of server-side video buffer
• CPC Class Codes: H04N 21/23412 for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/2365 Multiplexing of several video streams
• CPC Class Codes: H04N 21/426 Internal components of the client ; Characteristics thereof H04N5/44 takes precedence
• CPC Class Codes: H04N 21/4305 Synchronising client clock from received content stream, e.g. locking decoder clock with encoder clock, extraction of the PCR packets arrangements for synchronising receiver with transmitter by comparing receiver clock with transmitter clock H04L7/0012; arrangements for synchronising receiver with transmitter wherein the receiver takes measures against momentary loss of synchronisation H04L7/0083
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4341 Demultiplexing of audio and video streams
• CPC Class Codes: H04N 21/4347 Demultiplexing of several video streams
• CPC Class Codes: H04N 21/44004 involving video buffer management, e.g. video decoder buffer or video display buffer
• CPC Class Codes: H04N 21/44012 involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
• CPC Class Codes: H04N 21/64723 Monitoring of network processes or resources, e.g. monitoring of network load traffic related reporting in data switching networks H04L43/062
• CPC Class Codes: H04N 21/8547 involving timestamps for synchronizing content
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56

Abstract

In one embodiment, a system includes a video mixer coupled with an audio mixer for exchange of information that includes a first set of delay values respecting input audio streams received by the audio mixer from a plurality of source endpoints, and output audio streams sent from the audio mixer to a plurality of destination endpoints. The information further including a second set of delay values respecting the corresponding input video streams. The audio mixer calculates end-to-end video delays, and the video mixer calculates end-to-end audio delays. The audio mixer delays the output audio streams to equalize the end-to-end audio and video delays in the event that the end-to-end audio delays are less than the end-to-end video delays, and the video mixer delays the output video streams to equalize the end-to-end audio and video delays in the event that the end-to-end video delays are less than the end-to-end audio delays.

Claims

1. A method comprising:

receiving, by a video mixer, information from an audio mixer that includes delay values respecting input audio streams received by the audio mixer from a plurality of source endpoints, and output audio streams sent without timestamps from the audio mixer to a plurality of destination endpoints that do not extract Real-Time Transport Control Protocol (RTCP) packets, each input and output audio stream being respectively associated with a corresponding input and output video stream, the video mixer and the audio mixer being physically located at different places on a communication network;

calculating, from the information received, an output delay value for each corresponding output video stream to ensure that source endpoint-to-destination endpoint audio and video delays are substantially equal;

buffering, by the video mixer, each of the corresponding input and output video streams; and

adjusting the buffering of each of the corresponding input and output video streams so as to delay each corresponding output video stream by the output delay value.

2. The method of claim 1 wherein the delay values include, for each of the input audio streams, a first delay from a source endpoint to a summation unit of the audio mixer, and for each of the output audio streams, a second delay from the summation unit to a destination endpoint.

3. The method of claim 1 further comprising:

calculating, from the information received, a source endpoint-to-destination endpoint audio delay from one of the source endpoints to one of the destination endpoints.

4. The method of claim 1 further comprising:

receiving, by the video mixer, new information from the audio mixer that includes updated delay values.

5. A method comprising:

receiving, by an audio mixer, information from an video mixer that includes delay values respecting input video streams received by the video mixer from a plurality of source endpoints, and output video streams sent without timestamps from the video mixer to a plurality of destination endpoints that do not extract Real-Time Transport Control Protocol (RTCP) packets, each input and output video stream being respectively associated with a corresponding input and output audio stream, the video mixer and the audio mixer being physically located at different places on a communication network, the delay values including, for each of the input video streams, a first delay from a source endpoint to a summation unit of the video mixer, and for each of the output video streams, a second delay from the summation unit to a destination endpoint;

calculating, from the information received, an output delay value for each corresponding output audio stream to ensure that source endpoint-to-destination endpoint audio end video delays are substantially equal;

buffering, by the audio mixer, each of the corresponding input and output audio streams;

adjusting the buffering of each of the corresponding input and output audio streams so as to delay each corresponding output audio stream by the output delay value.

6. The method of claim 5 further comprising:

calculating, from the information received, a source endpoint-to-destination endpoint video delay from one of the source endpoints to one of the destination endpoints.

7. The method of claim 5 further comprising:

receiving, by a audio mixer, new information from the video mixer that includes updated delay values.

8. A computer readable memory encoded with a computer program product, when executed the computer program product being operable to:

during a video conference session, receive information from an audio mixer on a communication network, the information including delay values respecting input audio streams received by the audio mixer from a plurality of source endpoints, and output audio streams sent without timestamps from the audio mixer to a plurality of destination endpoints that do not extract Real-Time Transport Control Protocol (RTCP) packets, each input and output audio stream being respectively associated with a corresponding input and output video stream, the delay values including, for each of the input audio streams, a first delay from a source endpoint to a summation unit of the audio mixer, and for each of the output audio streams, a second delay from the summation unit to a destination endpoint;

calculate, from the information received, an output delay value for each corresponding output video stream to ensure that source endpoint-to-destination endpoint audio and video delays are substantially equal;

buffer, by a video mixer, each of the corresponding input and output video streams; and

adjust the buffering so as to delay each corresponding output video stream by the output delay value.

9. A computer readable memory encoded with a computer program product, when executed the computer program product being operable to:

during a video conference session, receive information from a video mixer physically located at a different place on a communication network, the information including delay values respecting input video streams received by the video mixer from a plurality of source endpoints, and output video streams sent without timestamps from the video mixer to a plurality of destination endpoints that do not extract Real-Time Transport Control Protocol (RTCP) packets, each input and output video stream being respectively associated with a corresponding input and output audio stream, the delay values including, for each of the input video streams, a first delay from a source endpoint to a switch of the video mixer, and for each of the output video streams, a second delay from the switch to a destination endpoint;

calculate, from the information received, an output delay value for each corresponding output audio stream to ensure that source endpoint-to-destination endpoint audio and video delays are substantially equal;

buffer, by an audio mixer, each of the corresponding input and output audio streams; and

adjust the buffering so as to delay each corresponding output audio stream by the output delay value.