METHOD AND SYSTEM FOR SYNCHRONIZING AUDIO AND VIDEO STREAMS IN MEDIA RELAY CONFERENCING

• US 20140118473A1
• Filed: 10/15/2013
• Published: 05/01/2014
• Est. Priority Date: 11/01/2012
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Polycom Incorporated (HP Inc.)
• Inventors: Halavy, Avishay, Yassur, Amir
• Granted Patent: US 9,426,423 B2
• Time in Patent Office: Days
• US Class Current: 348/14.09
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

A novel technique allows synchronizing a plurality of audio and video streams received at a receiving media relay endpoint via an intermediate node, such as a media relay multipoint control unit. The received streams were generated by a plurality of transmitting media relay endpoints and been relayed to the plurality of receiving media relay endpoints via the intermediate node, but are seen as being using a single time domain, in terms of wall clock and manipulated timestamps, while preserving the correct capture time.

Claims

1. A method, comprising:

receiving, at an intermediate device that is communicatively coupled between a plurality of media relay endpoints, a plurality of input streams from the plurality of media relay endpoints, wherein each stream comprises input data chunks;

parsing the input data chunks; and

converting a timestamp field of each input data chunk into a manipulated timestamp, reflecting a single time domain.

2. The method of claim 1, further comprising:

creating output data chunks, each associated with a relevant manipulated timestamp;

organizing the output data chunks from a subset of the plurality of media relay endpoints into one or more output streams; and

relaying the one or more output streams toward a receiving media relay endpoint.

3. The method of claim 2, further comprising:

interpreting the one or more output streams by the receiving media relay endpoint as being generated using a single time domain.

4. The method of claim 2, wherein organizing the output data chunks comprises:

writing the manipulated timestamp in a timestamp field of the corresponding output data chunk of the one or more output streams.

5. The method of claim 4,

wherein the one or more output streams comprise a video stream and an audio stream, and

wherein the video stream and the audio stream are synchronizable using the manipulated timestamp.

6. The method of claim 1, wherein the plurality of input streams comprise a plurality of video streams, further comprising:

organizing the plurality of input streams into one or more output video streams, configured for composition into a continuous presence video image.

7. The method of claim 1, wherein the plurality of input streams comprise a plurality of audio streams, further comprising:

organizing the plurality of audio streams into one or more output audio streams, mixable into a mixed audio stream.

8. The method of claim 1, wherein the single time domain is a time domain of the intermediate device.

9. The method of claim 1, wherein the intermediate device is a media relay multipoint control unit.

10. The method of claim 1, further comprising:

calculating a parameter corresponding to each input stream for converting the timestamp field of the input data chunks into the single time domain.

11. The method of claim 10, wherein calculating the parameter comprises:

estimating a difference between a clock of a transmitting media relay endpoint of the input stream and a clock of the intermediate device; and

determining the parameter as a function of the estimated difference, a timestamp clock rate associated with the input stream, a clock field and a timestamp field in a sender report corresponding to the input stream, a clock value of the intermediate device, and a timestamp value of the intermediate device,

wherein the clock value of the intermediate device and the timestamp value of the intermediate device are obtained at a same time.

12. The method of claim 10, wherein converting the timestamp field comprises modifying the timestamp field of the input data chunks by the parameter corresponding to the input stream.

13. The method of claim 1, further comprising:

combining the output data chunks into a single output stream.

14. The method of claim 1, further comprising:

sending a single sender report per media type from the intermediate device to each receiving media relay endpoint, representing the single time domain.

15. The method of claim 1, further comprising:

organizing the output data chunks from a subset of the plurality of media relay endpoints into one or more output streams; and

combining the one or more output streams into a continuous presence video image by the receiving media relay endpoint.

16. The method of claim 1, wherein the plurality of input streams comprise a plurality of input audio streams, further comprising:

associating the value of the manipulated timestamp of an input audio data chunk with an identifier of the input stream that carries that input audio data chunk;

decoding received input audio data streams into decoded audio streams;

selecting and mixing two or more decoded audio streams into a mixed audio stream;

compressing the mixed audio stream into an output mixed audio stream;

organizing the output mixed audio stream into a stream of output mixed audio data chunks; and

transferring the stream of output mixed audio data chunks toward a receiving media relay endpoint,

wherein the acts of decoding, selecting, compressing and organizing preserve an association between the output mixed audio data chunks, the corresponding manipulated timestamps and the stream identifiers.

17. The method of claim 16, wherein the act of organizing comprises:

manipulating each of the output mixed audio data chunks to include information related to the one or more stream identifiers and the one or more manipulated timestamps of input data chunks that are embedded in the mix.

18. The method of claim 16,

wherein the act of associating comprises:

aligning the input data chunks at the intermediate device according to their manipulated timestamps, prior to mixing them, and

wherein the act of organizing comprises;

inserting the manipulated timestamps as the manipulated timestamp of the output mixed audio data chunks.

19. A media relay multipoint controller comprising:

a signaling and control module configured to select one or more transmitting media relay endpoints of a plurality of transmitting media relay endpoints; and

a compressed media processor controlled by the signaling and control module, configured to;

parse input data chunks corresponding to a plurality of input streams; and

convert a timestamp field of the input data chunks from a received timestamp value to a manipulated timestamp value reflecting a single time domain.

20. The media relay multipoint controller of claim 19, wherein the compressed media processor is further configured to:

organize the input data chunks from selected transmitting media relay endpoints into one or more output streams, where each data chunk of each output stream is associated with a corresponding manipulated timestamp; and

transfer the one or more output streams toward a receiving media relay endpoint.

21. The media relay multipoint controller of claim 20,

wherein the one or more output streams comprise a video stream and an audio stream, and

wherein the timestamp field is used by the receiving media relay endpoint for synchronizing the video stream with the audio stream.

22. The media relay multipoint controller of claim 20, wherein the compressed media processor is configured to transfer the one or more output streams toward a receiving media relay endpoint via a single real time protocol connection between the media relay multipoint controller and the receiving media relay endpoint.

23. The media relay multipoint controller of claim 19,

wherein the plurality of input streams comprises a plurality of video streams.

24. The media relay multipoint controller of claim 19,

wherein the plurality of input streams comprises a plurality of audio streams.

25. The media relay multipoint controller of claim 19, wherein the single time domain is associated with the media relay multipoint controller.

26. The media relay multipoint controller of claim 19, wherein the compressed media processor is further configured to:

calculate a parameter for each input stream of the plurality of input streams for use in converting the timestamp fields of each input data chunk contained in the input stream.

27. The media relay multipoint controller of claim 26, wherein the compressed media processor is configured to calculate the parameter for an input stream by being configured to:

estimate a difference between a wall clock of each transmitting media relay endpoint and a wall clock of the media relay multipoint controller; and

determine the parameter as a function of the estimated difference, a timestamp clock rate used for the input stream, a value of a wall clock field and a timestamp field in a sender report corresponding to the input stream, and the wall clock value and a timestamp value of a relevant media clock of the media relay multipoint controller,

wherein the wall clock value and the timestamp value of relevant media clock of the media relay multipoint controller are obtained at a same time.

28. The media relay multipoint controller of claim 26, wherein the compressed media processor is configured to convert the timestamp field by being configured to

adjust the timestamp field by the parameter for the corresponding input stream.

29. The media relay multipoint controller of claim 19, wherein the compressed media processor is further configured to send a single sender report per media type to each receiving media relay endpoint, representing the single time domain.

30. The media relay multipoint controller of claim 19,

wherein plurality of input streams comprise a plurality of input audio streams, and

wherein the compressed media processor is further configured to:

associate the manipulated timestamp value of input audio data chunks with an identifier of the stream that caries that input audio data chunk;

decode obtained input audio data streams into decoded audio streams;

select and mix a plurality of the decoded audio streams into a mixed audio stream;

compress the mixed audio stream into a stream of compressed mixed audio;

organize the stream of compressed mixed audio into a stream of output audio data chunks;

transfer the stream of output audio data chunks toward a receiving media relay endpoint; and

preserve an association between the output audio data chunks, the corresponding manipulated timestamp values, and the stream identifiers.

31. The media relay multipoint controller of claim 30, wherein the compressed media processor is further configured to:

add information related to the stream identifier and the manipulated timestamp to each related output audio data chunk.