SYSTEM AND METHOD FOR VIDEOCONFERENCING USING SCALABLE VIDEO CODING AND COMPOSITING SCALABLE VIDEO CONFERENCING SERVERS

• US 20070200923A1
• Filed: 12/22/2006
• Published: 08/30/2007
• Est. Priority Date: 12/22/2005
• Status: Active Grant

• Current Assignee: Vidyo Incorporated (Enghouse Systems Limited)
• Original Assignee: Vidyo Incorporated (Enghouse Systems Limited)
• Inventors: Eleftheriadis, Alexandros, Shapiro, Ofer, Wiegand, Thomas, Chakareski, Jacob
• Granted Patent: US 8,436,889 B2
• Time in Patent Office: Days
• US Class Current: 348/14.80
• CPC Class Codes: H04N 19/31 in the temporal domain
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/36 Scalability techniques involving formatting the layers as a function of picture distortion after decoding, e.g. signal-to-noise [SNR] scalability
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/89 involving methods or arrangements for detection of transmission errors at the decoder
• CPC Class Codes: H04N 21/234327 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/4438 Window management, e.g. event handling following interaction with the user interface
• CPC Class Codes: H04N 21/4621 Controlling the complexity of the content stream or additional data, e.g. lowering the resolution or bit-rate of the video stream for a mobile client with a small screen arrangements for using the results of monitoring on user's side in broadcast systems H04H60/65; flow control in packet networks H04L47/10
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 21/6379 directed to encoder , e.g. for requesting a lower encoding rate
• CPC Class Codes: H04N 21/6405 Multicasting data broadcast and multicast in packet switching networks H04L12/18
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

Systems and methods for videoconferencing are provided. The systems use scalable video coding techniques and a compositing scalable video coding server (CSVCS) to composite input video signals from transmitting conferencing participants into one single output video signal forwarded to a receiving participant. The server is configured to composite the input video signals pictures without decoding, rescaling, and re-encoding of the signals.

Claims

1. A multi-endpoint video signal conferencing system for videoconferencing between a plurality of endpoints over a communications network, the system comprising:

a conferencing bridge (“

Compositing Scalable Video Coding Server”

(CSVCS)) linked to at least one receiving and at least one transmitting endpoint by at least one communication channel each,

at least one endpoint that transmits coded digital video using either a single-layer coding format or a scalable video coding format, and

at least one receiving endpoint that is capable of decoding at least one digital video stream coded in a scalable video coding format,

wherein the CSVCS is configured to compose input video signals received from transmitting endpoints into a single composite coded digital video output signal, and to forward the single composite coded digital video output signal to the at least one receiving endpoint.

2. The conferencing system of claim 1, wherein the CSVCS is configured to compose input video signals received from transmitting endpoints into a single composite video output signal, and to forward the single composite video output signal to the at least one receiving endpoint without decoding and/or re-coding the input video signal.

3. The system of claim 1, wherein the communications network comprises Quality of Service (QoS) support, such that high reliability and low reliability transport channels are available for transport of scalable coded video data from transmitting endpoints to the CSVCS and from the CSVCS to the receiving endpoints, and wherein at least the lowest temporal level of the base layer is transmitted reliably over a transport channel between the transmitting endpoints and the CSVCS or the CSVCS and the receiving endpoints.

4. The system of claim 3, wherein QoS is provided via positive acknowledgments, or negative acknowledgments, or both.

5. The system of claim 3, wherein QoS is provided via forward error correction (FEC) techniques.

6. The system of claim 1 where the CSVCS is configured to compose an output signal picture for at least one of the receiving endpoints by:

assigning a portion of the composite output picture'"'"'s area to each transmitting endpoint intended to be included in the composite output picture;

discarding incoming video signal data received from the transmitting endpoints that correspond to one of;

resolutions higher than the one intended for the composed picture;

data not needed for decoding at the resolution intended for the composed picture; and

a transmitting endpoint that is not to be included in the composite picture;

modifying the remaining data of the incoming coded video signals by changing header information so that they form proper data of the composite output video signal;

generating, when necessary, artificial layer data for at least one of the transmitting endpoint video signals; and

transmitting the modified data and any generated lower layer data to the one or more receiving endpoints,

such that no decoding or encoding of picture data takes place at the CSVCS.

7. The system of claim 6, wherein the at least one receiving endpoint is capable of decoding video coded in the H.264 SVC scalable video coding format, and wherein:

the assignment of portions of the composite output picture'"'"'s area to transmitting endpoints intended to be included in the composite output picture is performed by defining a slice group map in the Picture Parameter Set of the composite output signal, with each transmitting endpoint corresponding to one slice group, and

the assignment of portions of the composite output picture'"'"'s area to transmitting endpoints is communicated to the at least one receiving endpoints by transmitting the Picture Parameter Set to the at least one receiving endpoints.

8. The system of claim 7 configured to convey the Picture Parameter Set in-band or out-of band to the one or more receiving endpoints.

9. The system of claim 7 which is further configured so that the composite output picture is flagged as;

used-for-reference, when at least one of the input pictures received from the transmitting endpoints that is included in the composite output picture is flagged as used-for-reference; and

not-used-for-reference, when all of the input pictures received from the transmitting endpoints that are included in the composite output picture are flagged as not-used-for-reference,

wherein if the composite output picture is flagged as used-for-reference, reference frame reordering commands are inserted in the slices of pictures received subsequently from the transmitting endpoints prior to their transmission to the at least one receiving endpoints, so that proper operation of the reference picture buffers at the one or more receiving endpoints is ensured.

10. The system of claim 7 wherein the NAL extension headers for SVC of the composite output picture NAL units are set such that:

the same dependency_id value is used for the NAL units of the composite output picture that correspond to the highest scalable coding layer present in the composite output picture, and the same but successively lower dependency_id value is used for NAL units of successive lower layers;

and wherein the temporal_level is set such that;

when the arriving pictures from the at least one transmitting endpoints are composed so that the temporal levels are synchronized, then the same temporal_level value is used for the NAL units corresponding to the highest scalable coding layer, and successively lower temporal_level values are used for successive lower layers, and

when the arriving pictures from the at least one transmitting endpoints are not composed so that the temporal levels are synchronized, then the value 0 is used for all NAL units of the composite output picture.

11. The system of claim 6 wherein assignment of specific portions of the composite output video picture'"'"'s area to specific transmitting endpoint video signals by the CSVCS is predefined.

12. The system of claim 6 wherein the assignment of specific portions of the composite output video picture'"'"'s area to specific transmitting endpoint video signals is performed dynamically by the CSVCS, based on:

requests for specific spatial resolutions from the receiving endpoint,

requests for specific spatial positions within the composite output picture from the receiving endpoint, and

combinations thereof.

13. The system of claim 6 wherein the CSVCS assignment of specific portions of the composite output video picture'"'"'s area to specific transmitting endpoint video signals is performed by the CSVCS, taking into account the decoding capabilities or resolution preferences of the at least one receiving endpoints.

14. The system of claim 6 wherein the CSVCS is configured to respond to input video signals having different temporal rates or a shift in the arrival times of the input video signals by at least one of:

transmitting an output picture triggered by incoming pictures;

transmitting an output picture based on the maximum frame rate of the input video signals;

transmitting an output picture based on a predefined time schedule,

and wherein the CSVCS is further configured to respond when a new picture of the input video signals does not arrive on time for transmission by;

transmitting pre-coded slice data that instruct the at least one receiving endpoint to repeat data from a previous picture;

inserting reference picture list reordering commands in the picture headers of the following picture of the input video signals prior to transmission to the at least one receiving endpoint, to ensure that proper reference picture selection is performed for the said following picture.

15. The system of claim 14, wherein the CSVCS is further configured to discard and not forward received picture data of a transmitting endpoint if the elimination of the discarded picture data will not adversely affect the decoding process at the at least one receiving endpoint, and to transmit in place of the discarded picture data received picture data of a later picture of the same transmitting endpoint, so that the pictures of the transmitting endpoint are synchronized with those of the other transmitting endpoints in the composite output video signal that is transmitted to the at least one receiving endpoints.

16. The system of claim 6 wherein the CSVCS is further configured to decode at least the lowest temporal level of the lowest spatial and quality resolutions of the video signals received from the at least one transmitting endpoints, and wherein the CSVCS is further configured to generate an intra encoding for the video signal of the transmitting endpoint that is affected when the composite picture configuration for an existing receiving endpoint needs to change, and to transmit the intra encoding to the receiving endpoint in place of the corresponding coded picture data received from the said transmitting endpoint.

17. The system of claim 1 comprising a plurality of CSVCS in an cascading arrangement, wherein at least one CSVCS, which is not the last one in the cascading arrangement, is configured to optionally:

without processing, forward composite coded pictures received from CSVCS that are earlier in the cascading arrangement to other CSVCSs;

or

decompose the composite coded pictures received from CSVCS that are earlier in the cascading arrangement and recompose them using a different layout prior to forwarding them to other CSVCSs.

18. The system of claim 1 comprising at least one SVCS and at least one CSVCSs in an cascading arrangement, wherein:

the at least one SVCS is configured to only select some or all data of the video signals received from the transmitting endpoints or other SVCSs, and forward the selected data to other SVCSs or CSVCSs, and

a CSVCS is the last server in the cascading arrangement, which

creates the composite coded output picture for transmission to the one or more receiving endpoints.

19. The system of claim 1, wherein the CSVCS is configured to generate and transmit coded picture data for portions of the composite output video picture'"'"'s area that are not assigned to any transmitting endpoint.

20. The system of claim 19, wherein the transmitted coded picture data for the portions of the composite output video picture'"'"'s area that are not assigned to any transmitting endpoint are generated by one of:

coding of new content that is generated dynamically;

retrieval of coded data that has been previously computed and stored; and

a combination thereof.

21. The system of claim 1 further comprising a feedback channel between an endpoint and the CSVCS, wherein the CSVCS is further configured to respond to endpoint instructions communicated via intra content over the feedback channel.

22. The system of claim 1 wherein the coding dependency of the scalable video coding technique used is such that enhancement layers that are not used as references by other layers are flagged as discardable.

23. The system of claim 22, wherein the CSVCS is further configured to optionally discard enhancement layers that are flagged as discardable.

24. The system of claim 1 further comprising bidirectional control channels linking a transmitting endpoint with the CSVCS, and the CSVCS with a receiving endpoint.

25. The system of claim 24 which further configured to provide a capabilities exchange between the transmitting and receiving endpoints over the bi-directional control channel through the CSVCS, whereby the transmitting endpoint indicates its capabilities in terms of spatial, temporal, quality resolutions, and bit rates, and the receiving endpoint indicates which of these capabilities it supports.

26. The system of claim 25, wherein the CSVCS is configured to receive requests from a receiving endpoint for a desired spatial resolution of output video signals.

27. The system of claim 26, wherein the CSVCS is configured to modify the spatial layout in the composite output picture to accommodate different spatial resolution requests.

28. The system of claim 26, wherein the CSVCS is configured to instruct a transmitting endpoint to add or remove a spatial resolution from its transmitted video signal.

29. The system of claim 1, wherein the CSVCS is configured to include source identifying information or other information for display over one of in-band and out-of-band bitstreams.

30. The system of claim 1, wherein the CSVCS is configured to overlay source identifying information or other conveyed information on one of:

(1) the pixels of portions of the composite output picture'"'"'s area that are assigned to each participant in an output picture, and (2) the pixels of portions of the composite output picture'"'"'s area that are not assigned to any of the video signals of the transmitting participants.

31. The conferencing system of claim 1 wherein the CSVCS is configured to provide at least one of continuous presence, personalized layout, rate matching, error localization, and random entry features to at least one receiving endpoint linked through the CSVCS by selectively multiplexing the video signal layers received from transmitting endpoints as components of the composite output picture, and forwarding the components to the linked receiving endpoint, together with optional additional data generated at the CSVCS that ensures that the composite output signal received at the linked received endpoint is a valid coded video bitstream

32. The conferencing system of claim 31 wherein the CSVCS is further configured to respond to bandwidth conditions by at least one of:

statistically multiplexing video signals from a plurality of transmitting endpoints; and

synchronizing the composition and transmission of video signals received from transmitting endpoints to stagger larger-than-average video pictures in the composite output video signal.

33. The conferencing system of claim 31 wherein the CSVCS is further configured to:

modify the bitrate of the transmitted composite output signal by replacing coded picture data that is received from the at least one transmitting endpoint with coded data that indicates to the at least one receiving endpoint to copy the corresponding pixel data from a previous picture, and

transmit the replacement coded data,

so that the output bitrate can match desired characteristics.

34. The conferencing system of claim 1, wherein the CSVCS is further configured to provide at least one of session network border control, media proxy, firewall, and network address translation functions.

35. A method for videoconferencing between a plurality of endpoints over a communications network, the method comprising:

using a conferencing bridge (“

Compositing Scalable Video Coding Server”

(CSVCS)) linked to at least one receiving and at least one transmitting endpoint by at least one communication channel each;

transmitting coded digital video from at least one transmitting endpoint in either a single-layer coding format or a scalable video coding format; and

at the CSVCS, composing input video signals received from transmitting endpoints into a single composite coded digital video output signal, and forwarding the single composite coded digital video output signal to at least one receiving endpoint which is capable of decoding at least one digital video stream coded in a scalable video coding format.

36. The method of claim 35, wherein the CSVCS is configured to compose input video signals received from transmitting endpoints into a single composite video output signal, and wherein forwarding the single composite video output signal to the at least one receiving endpoint comprises doing so without decoding and/or re-coding the input video signal.

37. The method of claim 35, wherein the communications network comprises Quality of Service (QoS) support, such that high reliability and low reliability transport channels are available for transport of scalable coded video data from transmitting endpoints to the CSVCS and from the CSVCS to the receiving endpoints, the method further comprising reliably transmitting at least the lowest temporal level of the base layer over a transport channel between the transmitting endpoints and the CSVCS or the CSVCS and the receiving endpoints.

38. The method of claim 37, further comprising providing QoS via positive acknowledgments, or negative acknowledgments, or both.

39. The method of claim 37, further comprising providing QoS via forward error correction (FEC) techniques.

40. The method of claim 35, at the CSVCS, composing input video signals received from transmitting endpoints into a single composite coded digital video output signal comprises:

assigning a specific portion of the composite output picture'"'"'s area to each specific transmitting endpoint intended to be included in the composite output picture;

discarding incoming video signal data received from the transmitting endpoints that correspond to one of;

resolutions higher than the one intended for the composed picture;

data not needed for decoding at the resolution intended for the composed picture; and

a transmitting endpoint that is not to be included in the composite picture;

modifying the remaining data of the incoming coded video signals by changing header information so that they form proper data of the composite output video signal;

generating, when necessary, artificial layer data for at least one of the transmitting endpoint video signals; and

transmitting the modified data and any generated lower layer data to the one or more receiving endpoints,

such that no decoding or encoding of picture data takes place at the CSVCS.

41. The method of claim 40, wherein the at least one receiving endpoint is capable of decoding video coded in the H.264 SVC scalable video coding format, and wherein:

assigning a portion of the composite output picture'"'"'s area to each transmitting endpoint intended to be included in the composite output picture is performed by defining a slice group map in the Picture Parameter Set of the composite output signal, with each transmitting endpoint corresponding to one slice group, and

transmitting the Picture Parameter Set to the at least one receiving endpoints to communicate the assignment of specific portions of the composite output picture'"'"'s areas to specific transmitting endpoints to the at least one receiving endpoint.

42. The method of claim 41, further comprising conveying the Picture Parameter Set in-band or out-of band to at least one receiving endpoint.

43. The method of claim 41, further comprising flagging the composite output picture as;

used-for-reference, when at least one of the input pictures received from the transmitting endpoints that is included in the composite output picture is flagged as used-for-reference; and

not-used-for-reference, when all of the input pictures received from the transmitting endpoints that are included in the composite output picture are flagged as not-used-for-reference,

wherein if the composite output picture is flagged as used-for-reference, reference frame reordering commands are inserted in the slices of pictures received subsequently from the transmitting endpoints prior to their transmission to the at least one receiving endpoints, so that proper operation of the reference picture buffers at the at least one receiving endpoint is ensured.

44. The method of claim 41, further comprising setting the NAL extension headers for SVC of the composite output picture NAL units such that:

the same dependency_id value is used for the NAL units of the composite output picture that correspond to the highest scalable coding layer present in the composite output picture, and the same but successively lower dependency_id value is used for NAL units of successive lower layers;

and setting the temporal_level such that;

when the arriving pictures from the at least one transmitting endpoints are composed so that the temporal levels are synchronized, then the same temporal_level value is used for the NAL units corresponding to the highest scalable coding layer, and successively lower temporal_level values are used for successive lower layers; and

when the arriving pictures from the at least one transmitting endpoints are not composed such that the temporal levels are synchronized, then the value 0 is used for all NAL units of the composite output picture.

45. The method of claim 40, wherein assignment of specific portions of the composite output video picture'"'"'s area to specific transmitting endpoint video signals by the CSVCS is predefined.

46. The method of claim 40, wherein the assignment of specific portions of the composite output video picture'"'"'s area to specific transmitting endpoint video signals is performed dynamically by the CSVCS, based on:

requests for specific spatial resolutions from the receiving endpoint,

requests for specific spatial positions within the composite output picture from the receiving endpoint, and

combinations thereof.

47. The method of claim 40 further comprising taking into account the decoding capabilities or resolution preferences of the at least one receiving endpoints while assigning specific portions of the composite output video picture'"'"'s area to specific transmitting endpoint video signals.

48. The method of claim 40, wherein the CSVCS is configured to respond to input video signals having different temporal rates or a shift in the arrival times of the input video signals, the method further comprising responding by at least one of:

transmitting an output picture triggered by incoming pictures;

transmitting an output picture based on the maximum frame rate of the input video signals; and

transmitting an output picture based on a predefined time schedule,

and wherein the CSVCS is further configured to respond when a new picture of the input video signals does not arrive on time for transmission, the method further comprising responding by;

transmitting pre-coded slice data that instruct the at least one receiving endpoint to repeat data from a previous picture; and

inserting reference picture list reordering commands in the picture headers of the following picture of the input video signals prior to transmission to the at least one receiving endpoint, to ensure that proper reference picture selection is performed for the said following picture.

49. The method of claim 48, further comprising:

at the CSVCS, discarding and not forwarding received picture data of a transmitting endpoint if lack of the discarded picture data will not adversely affect the decoding process at the at least one receiving endpoint; and

transmitting received picture data of a later picture of the same transmitting endpoint in place of the discarded picture data, so that the pictures of the transmitting endpoint are synchronized with those of the other transmitting endpoints in the composite output video signal that is transmitted to the at least one receiving endpoints.

50. The method of claim 40, wherein the CSVCS is further configured to decode at least the lowest temporal level of the lowest spatial and quality resolutions of the video signals received from the at least one transmitting endpoints, the method further comprising:

at the CSVCS, generating an intra encoding for the video signal of the transmitting endpoint that is affected when the composite picture configuration for an existing receiving endpoint needs to change; and

transmitting the intra encoding to the receiving endpoint in place of the corresponding coded picture data received from the said transmitting endpoint.

51. The method of claim 35, in the case where the communication network comprises a plurality of CSVCS in an cascading arrangement, further comprising:

at least one CSVCS that is not the last one in the cascading arrangement, optionally forwarding without processing composite coded pictures received from CSVCS that are earlier in the cascading arrangement to other CSVCSs;

or decomposing the composite coded pictures received from CSVCS that are earlier in the cascading arrangement and recompose them using a different layout prior to forwarding them to other CSVCSs.

52. The method of claim 35, in the case where the communication network comprises at least one SVCS and at least one CSVCS in an cascading arrangement, further comprising:

at the at least one SVCS, selecting only some or all data of the video signals received from the transmitting endpoints or other SVCSs, and forwarding the selected data to other SVCSs or CSVCSs; and

at a CSVCS, which is the last server in the cascading arrangement, creating the composite coded output picture for transmission to the one or more receiving endpoints.

53. The method of claim 35, further comprising, at the CSVCS generating and transmitting to the one or more receiving endpoints coded picture data for portions of the composite output video picture'"'"'s area that are not assigned to any transmitting endpoint.

54. The method of claim 35, further comprising, at the CSVCS, generating transmitted coded picture data for the portions of the composite output video picture'"'"'s area that are not assigned to any transmitting endpoint by one of:

coding of new content that is generated dynamically;

retrieval of coded data that has been previously computed and stored; and

a combination thereof.

55. The method of claim 35, in the case where there is a feedback channel between an endpoint and the CSVCS, further comprising, responding to endpoint instructions communicated via intra content over the feedback channel.

56. The method of claim 35, in the case where the coding dependency of the scalable video coding technique used is such that some enhancement layers are not used as references by other layers, further comprising:

flagging such layers as discardable.

57. The method of claim 56, further comprising, at the CSVCS, optionally discarding enhancement layers that are flagged as discardable.

58. The method of claim 35, further comprising, providing bidirectional control channels linking a transmitting endpoint with the CSVCS, and the CSVCS with a receiving endpoint.

59. The method of claim 58, further comprising, conducting a capabilities exchange between the transmitting and receiving endpoints over the bi-directional control channel through the CSVCS, whereby the transmitting endpoint indicates its capabilities in terms of spatial, temporal, quality resolutions, and bit rates, and the receiving endpoint indicates which of these capabilities it supports.

60. The method of claim 58, further comprising, receiving requests at the CSVCS from a receiving endpoint for desired spatial resolution of output video signals.

61. The method of claim 60, further comprising, at the CSVCS, modifying the spatial layout in the composite output picture to accommodate different spatial resolution requests.

62. The method of claim 60, further comprising, from the CSVCS, instructing a transmitting endpoint to add or remove a spatial resolution from its transmitted video signal.

63. The method of claim 35, further comprising, including source identifying information and other information over one of in-band and out-of-band bitstreams sent by the CSVCS.

64. The method of claim 35, further comprising, at the CSVCS, overlaying source identifying information or other conveyed information on one of:

(1) the pixels of portions of the composite output picture'"'"'s area that are assigned to each participant in an output picture, and (2) the pixels of portions of the composite output picture'"'"'s area that are not assigned to any of the video signals of the transmitting participants.

65. The method of claim 35, further comprising, using the CSVCS to provide at least one of continuous presence, personalized layout, rate matching, error localization, and random entry features to at least one receiving endpoint linked through the CSVCS by selectively multiplexing the video signal layers received from transmitting endpoints as components of the composite output picture, and forwarding the components to the at least one linked receiving endpoint, together with optional additional data generated at the CSVCS that ensures that the composite output signal received at the at least one linked received endpoint is a valid coded video bitstream

66. The method of claim 35, further comprising, using the CSVCS to respond to bandwidth conditions by at least one of:

statistically multiplexing video signals from a plurality of transmitting endpoints; and

synchronizing the composition and transmission of video signals received from transmitting endpoints to stagger larger-than-average video pictures in the composite output video signal.

67. The method of claim 35, wherein using the CSVCS to respond to bandwidth conditions, further comprises:

modifying the bitrate of the transmitted composite output signal by replacing coded picture data that is received from the at least one transmitting endpoint with coded data that indicates to the at least one receiving endpoint to copy the corresponding pixel data from a previous picture, and

transmitting the replacement coded data,

so that the output bitrate can match desired characteristics.

68. The method of claim 35, further comprising, using the CSVCS to provide at least one of session network border control, media proxy, firewall, and network address translation functions.

69. Computer readable media comprising a set of instructions to perform the steps recited in at least one of the method claims 35-68.