System and method for thinning of scalable video coding bit-streams

• US 8,442,120 B2
• Filed: 08/03/2011
• Issued: 05/14/2013
• Est. Priority Date: 02/16/2006
• Status: Active Grant

• Current Assignee: Vidyo Incorporated (Enghouse Systems Limited)
• Original Assignee: Vidyo Incorporated (Enghouse Systems Limited)
• Inventors: Hong, Danny, Wiegand, Thomas, Eleftheriadis, Alexandros, Shapiro, Ofer
• Primary Examiner(s): Truong, Thanhnga B
• Application Number:
  US13/197,352
• Publication Number:
  US 20120014434A1
• Time in Patent Office: 650 Days
• Field of Search: 375/240.02, 375/240.1, 375/240.24, 375/240.25, 380/201
• US Class Current: 375/240.24
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/159 Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/31 in the temporal domain
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/34 Scalability techniques involving progressive bit-plane based encoding of the enhancement layer, e.g. fine granular scalability [FGS]
• CPC Class Codes: H04N 19/36 Scalability techniques involving formatting the layers as a function of picture distortion after decoding, e.g. signal-to-noise [SNR] scalability
• CPC Class Codes: H04N 19/37 with arrangements for assigning different transmission priorities to video input data or to video coded data
• CPC Class Codes: H04N 19/40 using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
• CPC Class Codes: H04N 19/521 for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

A system for videoconferencing that offers, among other features, extremely low end-to-end delay as well as very high scalability. The system accommodates heterogeneous receivers and networks, as well as the best-effort nature of networks such as those based on the Internet Protocol. The system relies on scalable video coding to provide a coded representation of a source video signal at multiple temporal, quality, and spatial resolutions. These resolutions are represented by distinct bitstream components that are created at each end-user encoder. System architecture and processes called SVC Thinning allow the separation of data into data used for prediction in other pictures and data not used for prediction in other pictures. SVC Thinning processes, which can be performed at video conferencing endpoints or at MCUs, can selectively remove or replace with fewer bits the data not used for prediction in other pictures from transmitted bit streams. This separation and selective removal or replacement of data for transmission allows a trade-off between scalability support (i.e. number of decodable video resolutions), error resiliency and coding efficiency.

Claims

1. A digital video communication system comprising:

at least one encoding device for transmitting coded digital video using a scalable video coding format that supports one or more of temporal, spatial or quality scalability,

at least one decoding device for decoding at least one digital video stream coded in a scalable video coding format, and

a Scalable Video Coding Server (SVCS) linked to the at least one decoding device and the at least one encoding device by at least one communication channel each,

wherein the video signal transmitted from the at least one encoding device is partitioned into distinct data sets comprising;

a first data set corresponding to the target layer intended for decoding by the at least one decoding device,

a second data set corresponding to layers that correspond to lower temporal, spatial, or quality resolutions that the target layer intended for decoding by the at least one decoding device,

a third auxiliary data set corresponding to layers that correspond to lower temporal, spatial, or quality resolutions than the target layer intended for decoding by the at least one decoding device and containing at least information that is used for prediction by the target layer intended for decoding, and

an optional fourth data set corresponding to layers that correspond to higher temporal, spatial, or quality resolutions that the target layer intended for decoding by the at least one decoding device,

such that the SVCS can selectively multiplex data from the second and third data sets to the at least one decoding device in conjunction with that of the first data set and optionally the fourth data set.

2. The system of claim 1, further comprising:

one or more feedback channels over the communication network for transmitting information from the at least one decoding device and the SVCS to the at least one encoding device,

wherein the at least one decoding device and SVCS are configured to communicate network condition indicators to the at least one encoding device over the at least one feedback channels, and wherein the at least one encoding device is configured to adapt the construction of the third data set according to the reported network conditions.

3. The system of claim 1 wherein the third data set of the video signal transmitted from the at least one encoding device is generated and transmitted on a picture-adaptive basis.

4. The system of claim 1, wherein an encoder of the at least one encoding device is configured to encode transmitted media as frames in a threaded coding structure having a number of different temporal levels, wherein a subset of the frames (“

R”

) is particularly selected for reliable transport and includes at least the frames of the lowest temporal layer in the threaded coding structure and such that the decoder can decode at least a portion of received media based on a reliably received frame of the type R after packet loss or error and thereafter is synchronized with the encoder, and wherein the third data set of the video signal transmitted from the at least one encoding device is generated and transmitted for non-R frames only.

5. A digital video communication system comprising:

at least one encoding device for transmitting coded digital video using a scalable video coding format that supports one or more of temporal, spatial or quality scalability,

at least one decoding device for decoding at least one digital video stream coded in a scalable video coding format, and

an SVCS linked to the at least one decoding and the at least one encoding device by at least one communication channel each,

wherein the video signal transmitted from the at least one encoding device is partitioned into distinct data sets comprising;

a first data set corresponding to the target layer intended for decoding by the at least one decoding device,

a second data set corresponding to layers that correspond to lower temporal, spatial, or quality resolutions than the target layer intended for decoding by the at least one decoding device and containing information that is used for prediction by the target layer intended for decoding,

a third data set corresponding to layers that correspond to lower temporal, spatial, or quality resolutions than the target layer intended for decoding by the at least one decoding device and containing information that is not used for prediction by the target layer intended for decoding, and

an optional fourth data set corresponding to layers that correspond to higher temporal, spatial, or quality resolutions that the target layer intended for decoding by the at least one decoding device,

such that the SVCS can selectively multiplex data from the second and third data sets to the at least one decoding device in conjunction with that of the first data set and optionally the fourth data set.

6. The system of claim 5, further comprising:

at least one feedback channels over the communication network for transmitting information from the at least one decoding device and the SVCS to the at least one encoding device,

wherein the at least one decoding device and SVCS are configured to communicate network condition indicators to the at least one encoding device over the at least one feedback channel, and wherein the at least one encoding device is configured to adapt the construction of the third data set according to the reported network conditions.

7. The system of claim 5 wherein the separation of the data corresponding to layers that correspond to lower temporal, spatial, or quality layer resolutions than the target layer intended for decoding by the at least one decoding device into a second and third data set is performed on a picture-adaptive basis.

8. The system of claim 5 wherein an encoder of the at least one encoding device is configured to encode transmitted media as frames in a threaded coding structure having a number of different temporal levels, wherein a subset of the frames (“

R”

) is particularly selected for reliable transport and includes at least the frames of the lowest temporal layer in the threaded coding structure and such that the decoder can decode at least a portion of received media based on a reliably received frame of the type R after packet loss or error and thereafter is synchronized with the encoder, and wherein the separation of the data corresponding to layers that correspond to lower temporal, spatial, or quality layer resolutions than the target layer intended for decoding by the at least one decoding device into a second and third data set is performed for non-R frames only.

9. A method for digital video communication in a system comprising:

at least one encoding device that transmits coded digital video using a scalable video coding format that supports one or more of temporal, spatial or quality scalability,

at least one decoding device that decodes at least one digital video stream coded in a scalable video coding format, and

an SVCS linked to the at least one decoding device and the at least one encoding device by at least one communication channel each,

the method comprising;

partitioning by the SVCS a video signal transmitted from the at least one encoding device into distinct data sets comprising;

a first data set corresponding to the target layer intended for decoding by the at least one decoding device,

a second data set corresponding to layers that correspond to lower temporal, spatial, or quality resolutions that the target layer intended for decoding by the at least one decoding device,

a third auxiliary data set corresponding to layers that correspond to lower temporal, spatial, or quality resolutions that the target layer intended for decoding by the at least one decoding device and containing at least information that is used for prediction by the target layer intended for decoding, and

an optional fourth data set corresponding to layers that correspond to higher temporal, spatial, or quality resolutions that the target layer intended for decoding by the at least one decoding device,

such that the SVCS can selectively multiplex data from the second and third data sets to the at least one decoding device in conjunction with that of the first data set and optionally the fourth data set.

10. The method of claim 9, wherein there is one or more feedback channels over the communication network for transmitting information from the at least one decoding device and the SVCS to the at least one encoding device, and wherein the at least one decoding device and SVCS communicate network condition indicators to the at least one encoding device over the at least one feedback channels, the method further comprising:

at least one encoding device, adapting the construction of the third data set according to the reported network conditions.

11. The method of claim 9 further comprising:

at least one encoding device generating and transmitting the third data set of the output video signal on a picture-adaptive basis.

12. The method of claim 9, wherein an encoder of the at least one encoding device encodes transmitted media as frames in a threaded coding structure having a number of different temporal levels, wherein a subset of the frames (“

R”

) is particularly selected for reliable transport and includes at least the frames of the lowest temporal layer in the threaded coding structure and such that the decoder can decode at least a portion of received media based on a reliably received frame of the type R after packet loss or error and thereafter is synchronized with the encoder, and wherein the third data set of the video signal transmitted from the at least one encoding device is generated and transmitted for non-R frames only.

17. A non-transitory computer readable medium comprising a set of instructions to direct a processor to perform the methods recited in one of claims 9-16.

13. A method for digital video communication in a system comprising:

at least one encoding device that transmits coded digital video using a scalable video coding format that supports one or more of temporal, spatial or quality scalability,

at least one decoding device that decodes at least one digital video stream coded in a scalable video coding format, and

an SVCS linked to the at least one decoding and the at least one encoding device by at least one communication channel each,

the method comprising;

partitioning by the SVCS a video signal transmitted from the at least one encoding device is partitioned into distinct data sets comprising;

a first data set corresponding to the target layer intended for decoding by the at least one decoding device,

a second data set corresponding to layers that correspond to lower temporal, spatial, or quality resolutions that the target layer intended for decoding by the at least one decoding device and containing information that is used for prediction by the target layer intended for decoding,

a third data set corresponding to layers that correspond to lower temporal, spatial, or quality resolutions that the target layer intended for decoding by the at least one decoding device and containing information that is not used for prediction by the target layer intended for decoding, and

an optional fourth data set corresponding to layers that correspond to higher temporal, spatial, or quality resolutions that the target layer intended for decoding by the at least one decoding device,

such that the SVCS can selectively multiplex data from the second and third data sets to the at least one decoding device in conjunction with that of the first data set and optionally the fourth data set.

14. The method of claim 13, wherein there is at least one feedback channel over the communication network for transmitting information from the at least one decoding device and the SVCS to the at least one encoding device,

indicators to the at least one encoding device over the at least one feedback channel, the method, further comprising:

at the at least one encoding device, adapting the construction of the third data set according to the reported network conditions.

15. The method of claim 13 further comprising:

at least one encoding device separating the data corresponding to layers that correspond to lower temporal, spatial, or quality layer resolutions than the target layer intended for decoding by the at least one decoding device into a second and third data set on a picture-adaptive basis.

16. The method of claim 13, wherein an encoder of the at least one encoding device encodes transmitted media as frames in a threaded coding structure having a number of different temporal levels, wherein a subset of the frames (“

R”

) is particularly selected for reliable transport and includes at least the frames of the lowest temporal layer in the threaded coding structure and such that a decoder can decode at least a portion of received media based on a reliably received frame of the type R after packet loss or error and thereafter is synchronized with the encoder, the method further comprising;

at least one encoding device separating the data corresponding to layers that correspond to lower temporal, spatial, or quality layer resolutions than the target layer intended for decoding by the at least one decoding device into a second and third data set for non-R frames only.