System and method for providing video conferencing synchronization

• US 7,084,898 B1
• Filed: 11/18/2003
• Issued: 08/01/2006
• Est. Priority Date: 11/18/2003
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Wu, Duanpei, Lantz, Keith A., Sarkar, Shantanu, Friedrich, Walter R., Firestone, Scott S., Ismail, Nermin M., Surazski, Luke K.
• Primary Examiner(s): Ramakrishnaiah, Melur
• Application Number:
  US10/715,687
• Time in Patent Office: 987 Days
• Field of Search: 348 1401- 1409, 348/14.1, 348/14.11, 348/14.12, 348/14.13
• US Class Current: 348/14.09
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

An audio mixer on a first device receives one or more incoming audio streams. Each of the one or more incoming audio streams has an associated timestamp. The audio mixer generates a mixed audio stream from the one or more incoming audio streams. The audio mixer determines differences in the time base of each of the one or more incoming audio streams and the time base for the mixed audio stream. The audio mixer generates mapping parameters associated with the determined differences and transforms the timestamp of each of the one or more incoming audio streams to a corresponding output timestamp associated with the mixed audio stream according to the mapping parameters. the mapping parameters are provided to a video mixer for similar processing and transformation such that the mixed audio stream is in synchronization with a mixed video stream.

Claims

1. A system for providing video conferencing synchronization, comprising:

an audio mixer on a first device operable to receive one or more incoming audio streams, each of the one or more incoming audio streams having an associated timestamp, the audio mixer operable to generate a mixed audio stream from the one or more incoming audio streams, the audio mixer operable to determine differences in the time base of each of the one or more incoming audio streams and the time base for the mixed audio stream, the audio mixer operable to generate mapping parameters associated with the determined differences, the audio mixer operable to transform the timestamp of each of the one or more incoming audio streams to a corresponding output timestamp associated with the mixed audio stream according to the mapping parameters;

a video mixer on a second device operable to receive one or more incoming video streams, each of the one or more incoming video streams having an associated timestamp, the video mixer operable to generate a mixed video stream from the one or more incoming video streams, the video mixer operable to receive the mapping parameters from the audio mixer, the video mixer operable to transform the timestamp of each of the one or more incoming video streams to a corresponding output timestamp associated with the mixed video stream according to the mapping parameters.

2. The system of claim 1, further comprising:

an endpoint operable to receive the mixed audio stream from the audio mixer and the mixed video stream from the video mixer, the endpoint operable to synchronize the mixed audio stream and the mixed video stream according to their respective output timestamps.

3. The system of claim 1, wherein the audio mixer is operable to determine when the mapping parameters are to be applied.

4. The system of claim 1, wherein the video mixer is operable to receive speaker selection information with the mapping parameters from the audio mixer.

5. The system of claim 1, wherein the mapping parameters include an offset and a scale factor, the offset and scale factor representing differences in time bases of the one or more incoming audio streams and the mixed audio stream.

6. A method for providing video conferencing synchronization, comprising:

receiving one or more incoming audio streams, each of the one or more incoming audio streams being associated with endpoint local timestamps;

converting the endpoint local timestamps to endpoint network timestamps for each of the one or more incoming audio streams;

generating a mixed audio stream from the one or more incoming audio streams, the mixed audio stream having mixer network timestamps;

determining mapping parameters associated with each incoming audio stream between the endpoint network timestamps and the mixer network timestamps.

7. The method of claim 6, further comprising:

receiving information indicating a relationship between the endpoint local timestamps and the endpoint network timestamps for each of the one or more incoming audio streams.

8. The method of claim 6, further comprising:

adjusting the mapping parameters in response to a change in average jitter buffer levels associated with each one of the incoming audio streams.

9. The method of claim 6, further comprising:

providing the mapping parameters to video stream processing in order to synchronize the mixed audio stream to an associated mixed video stream.

10. A method for providing video conferencing synchronization, comprising:

receiving one or more incoming audio streams, each of the one or more incoming audio streams being associated with endpoint local timestamps;

converting the endpoint local timestamps to endpoint network timestamps for each of the one or more incoming audio streams;

generating a mixed audio stream from the one or more incoming audio streams, the mixed audio stream having mixer network timestamps;

determining mapping parameters between the endpoint network timestamps and the mixer network timestamps;

comparing two sets of endpoint network timestamps to mixer network timestamps;

identifying an offset and a scale factor as the mapping parameters in response to timestamp comparison.

11. A system for providing video conferencing synchronization, comprising:

means for receiving one or more incoming audio streams, each of the one or more incoming audio streams being associated with endpoint local timestamps;

means for converting the endpoint local timestamps to endpoint network timestamps for each of the one or more incoming audio streams;

means for generating a mixed audio stream from the one or more incoming audio streams, the mixed audio stream having mixer network timestamps;

means for determining mapping parameters associated with each incoming audio stream between the endpoint network timestamps and the mixer network timestamps.

12. The system of claim 11, further comprising:

means for receiving information indicating a relationship between the endpoint local timestamps and the endpoint network timestamps for each of the one or more incoming audio streams.

13. The system of claim 11, further comprising:

means for adjusting the mapping parameters in response to a change in average jitter buffer levels associated with each one of the incoming audio streams.

14. The method of claim 11, further comprising:

means for providing the mapping parameters to video stream processing in order to synchronize the mixed audio stream to an associated mixed video stream.

15. A system for providing video conferencing synchronization, comprising:

means for receiving one or more incoming audio streams, each of the one or more incoming audio streams being associated with endpoint local timestamps;

means for converting the endpoint local timestamps to endpoint network timestamps for each of the one or more incoming audio streams;

means for generating a mixed audio stream from the one or more incoming audio streams, the mixed audio stream having mixer network timestamps;

means for determining mapping parameters between the endpoint network timestamps and the mixer network timestamps;

means for comparing two sets of endpoint network timestamps to mixer network timestamps;

means for identifying an offset and a scale factor as the mapping parameters in response to timestamp comparison.

16. A computer readable medium including code for providing video conferencing synchronization, the code operable to:

receive one or more incoming audio streams, each of the one or more incoming audio streams being associated with endpoint local timestamps;

convert the endpoint local timestamps to endpoint network timestamps for each of the one or more incoming audio streams;

generate a mixed audio stream from the one or more incoming audio streams, the mixed audio stream having mixer network timestamps;

determine mapping parameters associated with each incoming audio stream between the endpoint network timestamps and the mixer network timestamps.

17. The computer readable medium of claim 16, wherein the code is further operable to:

receive information indicating a relationship between the endpoint local timestamps and the endpoint network timestamps for each of the one or more incoming audio streams.

18. The computer readable medium of claim 16, wherein the code is further operable to:

adjust the mapping parameters in response to a change in average jitter buffer levels associated with each one of the incoming audio streams.

19. The computer readable medium of claim 16, wherein the code is further operable to:

provide the mapping parameters to video stream processing in order to synchronize the mixed audio stream to an associated mixed video stream.

20. A computer readable medium including code for providing video conferencing synchronization, the code operable to:

receive one or more incoming audio streams, each of the one or more incoming audio streams being associated with endpoint local timestamps;

convert the endpoint local timestamps to endpoint network timestamps for each of the one or more incoming audio streams;

generate a mixed audio stream from the one or more incoming audio streams, the mixed audio stream having mixer network timestamps;

determine mapping parameters between the endpoint network timestamps and the mixer network timestamps;

compare two sets of endpoint network timestamps to mixer network timestamps;

identify an offset and a scale factor as the mapping parameters in response to timestamp comparison.