Systems and methods for scalable composition of media streams for real-time multimedia communication

• US 8,482,593 B2
• Filed: 05/11/2011
• Issued: 07/09/2013
• Est. Priority Date: 05/12/2010
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Blue Jeans Network, Inc. (Verizon Communications Inc.)
• Inventors: Periyannan, Alagu, Anand, Raghavan, Grupenhoff, Michael, Kalluri, Ravi Kiran, Weber, Emmanuel
• Primary Examiner(s): Nguyen, Joseph J
• Application Number:
  US13/105,691
• Publication Number:
  US 20110279635A1
• Time in Patent Office: 790 Days
• Field of Search: 348 1401- 1409
• US Class Current: 348/14.08
• CPC Class Codes: H04L 12/1827 Network arrangements for conference optimisation or adaptation
• CPC Class Codes: H04L 51/10 Multimedia information
• CPC Class Codes: H04N 5/265 Mixing
• CPC Class Codes: H04N 7/141 between two video terminals, e.g. videophone telephonic communication systems combined with television receiver for reception of entertainment or information matter H04M11/085
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

A new approach is proposed that contemplates systems and methods to support the operation of a Virtual Media Room or Virtual Meeting Room (VMR), wherein each VMR can accept from a plurality of participants at different geographic locations a variety of video conferencing feeds from video conference endpoints that can be either proprietary or standards-based and enable a multi-party video conferencing session among the plurality of participants by composing one composite audio and video stream for each of the participants. Each single VMR can be implemented across an infrastructure of globally distributed set of servers/media processing nodes co-located in Points of Presence (POPs) for Internet access. Each VMR also gives its users a rich set of conferencing and collaboration interaction hitherto not experienced by video conferencing users.

Claims

1. A system, comprising:

one or more media processing nodes, each configured to accept from a plurality of participants to a video conference a plurality of audio and video streams from a plurality of video conference endpoints each associated with one of the plurality of participants, wherein the plurality of video conference endpoints are of different types, and wherein each of the media processing nodes builds up a composite metadata field associated with the composite video stream in order to apply operations outlined in the metadata field to each of the audio and video streams of the composite video stream and each of the media processing nodes utilizes information from the composite metadata field for efficient transcoding and processing of the associated composite audio and video streams;

for each of the plurality of participants to the video conference, convert and compose the plurality of audio and video streams into one or more composite video and audio streams compatible with the video conference endpoint associated with the participant in real time;

render the composite video and audio stream compatible with the video conference endpoint associated with each of the plurality of participants to the video conference.

2. The system of claim 1, wherein:

the one or more media processing nodes are MCUs built from off-the-shelf components including Linux/x86 CPUs and PC GPUs (Graphics Processing Units) instead of custom hardware.

3. The system of claim 1, wherein:

each of the plurality of video conference endpoints is either standard-based or proprietary.

4. The system of claim 1, wherein:

the audio and video streams are each in the form of one or more of;

original compressed video, uncompressed raw video, a lower resolution compressed thumbnail video.

5. The system of claim 1, wherein:

each of the media processing nodes converts one or more of;

video encoding formats, audio encoding formats, communication protocols, video resolutions, screen ratios, encryption standards, and acoustic considerations of the audio and video streams.

6. The system of claim 1, wherein:

each of the media processing nodes transcodes, upscales, and downscales of the audio and video streams into the composite audio and video stream.

7. The system of claim 1, wherein:

each of the media processing nodes composes and renders the composite video and audio stream to closely match the capabilities of the video conference endpoint associated with each of the plurality of participants in order for the participant to have an effective meeting experience.

8. The system of claim 1, wherein:

each of the media processing nodes subscribes to the audio and video streams needed to compose and render the composite video and audio stream that has been requested by each of the participants.

9. The system of claim 1, wherein:

each of the media processing nodes exchanges with or subscribes to the composite video and audio streams composed by other media processing nodes.

10. The system of claim 1, wherein:

each of the media processing nodes transmits the composite video and audio streams over the public internet, over a private network or over a provisioned overlay network with service level guarantees.

11. The system of claim 1, wherein:

each of the media processing nodes extracts and gathers characteristics of each of the audio and video streams for composing the composite video streams.

12. The system of claim 1, wherein:

each of the media processing nodes utilizes information from the composite metadata field for non-real-time multimedia services.

13. The system of claim 1, wherein:

each of the media processing nodes enables highly scalable audio mixing that mixes the audio streams from various codec at the best sampling rate after preprocessing the audio and picking only active channels to avoid signal degradation.

14. The system of claim 1, wherein:

each of the media processing nodes enables real-time language translation and speech-to-text or speech-to-visual services.

15. The system of claim 1, wherein:

each of the media processing nodes enables automatic Internet based determination of the need for acoustic echo cancellation at each of the video conference endpoints.

16. A method, comprising:

accepting from a plurality of participants to a video conference a plurality of audio and video streams from a plurality of video conference endpoints each associated with one of the plurality of participants, wherein the plurality of video conference endpoints are of different types;

for each of the plurality of participants to the video conference, converting and composing the plurality of audio and video streams into one composite video and audio stream compatible with the video conference endpoint associated with the participant in real time;

rendering the composite video and audio stream compatible with the video conference endpoint associated with each of the plurality of participants to the video conference;

building up a composite metadata field associated with the composite video stream in order to apply operations outlined in the metadata field to each of the audio and video streams of the composite video stream; and

utilizing information from the composite metadata field for efficient transcodinq and processing of the associated composite audio and video stream.

17. The method of claim 16, further comprising:

converting one or more of;

video encoding formats, audio encoding formats, communication protocols, video resolutions, screen ratios, encryption standards, and acoustic considerations of the audio and video streams.

18. The method of claim 16, further comprising:

transcoding, upscaling, and downscaling of the audio and video streams into the composite audio and video streams.

19. The method of claim 16, further comprising:

composing and rendering the composite video and audio stream to closely match the capabilities of the video conference endpoint associated with each of the plurality of participants in order for the participant to have an effective meeting experience.

20. The method of claim 16, further comprising:

subscribing to the audio and video streams needed to compose and render the composite video and audio stream to each of the participants to the video conference.

21. The method of claim 16, further comprising:

exchanging with or subscribing to the composite video and audio streams composed by other media processing nodes.

22. The method of claim 16, further comprising:

transmitting the composite video and audio streams over the public internet, over a private network or over a provisioned overlay network with service level guarantees.

23. The method of claim 16, further comprising:

extracting and gathering characteristics of each of the audio and video streams for composing the composite video streams.

24. The method of claim 16, further comprising:

utilizing information from the composite metadata field for non-real-time multimedia services.

25. The method of claim 16, further comprising:

enabling highly scalable audio mixing that mixes the audio streams from various codec at the best sampling rate.

26. The method of claim 16, further comprising:

enabling real-time language translation and speech-to-text or speech-to-visual services.

27. The method of claim 16, further comprising:

enabling automatic Internet based determination of the need for acoustic echo cancellation at each of the video conference endpoints.