System and method for thinning of scalable video coding bit-streams

• US 8,619,865 B2
• Filed: 02/16/2007
• Issued: 12/31/2013
• Est. Priority Date: 02/16/2006
• Status: Expired due to Fees

• Current Assignee: Vidyo Incorporated (Enghouse Systems Limited)
• Original Assignee: Vidyo Incorporated (Enghouse Systems Limited)
• Inventors: Hong, Danny, Wiegand, Thomas, Eleftheriadis, Alexandros, Shapiro, Ofer
• Primary Examiner(s): Mehedi, Morshed
• Assistant Examiner(s): LEE, JASON T
• Application Number:
  US11/676,215
• Publication Number:
  US 20070263087A1
• Time in Patent Office: 2,510 Days
• Field of Search: 375/240.24, 375/240.2, 375/240.11, 348/14.13
• US Class Current: 375/240.24
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/159 Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/31 in the temporal domain
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/34 Scalability techniques involving progressive bit-plane based encoding of the enhancement layer, e.g. fine granular scalability [FGS]
• CPC Class Codes: H04N 19/36 Scalability techniques involving formatting the layers as a function of picture distortion after decoding, e.g. signal-to-noise [SNR] scalability
• CPC Class Codes: H04N 19/37 with arrangements for assigning different transmission priorities to video input data or to video coded data
• CPC Class Codes: H04N 19/40 using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
• CPC Class Codes: H04N 19/521 for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

A system for videoconferencing that offers, among other features, extremely low end-to-end delay as well as very high scalability. The system accommodates heterogeneous receivers and networks, as well as the best-effort nature of networks such as those based on the Internet Protocol. The system relies on scalable video coding to provide a coded representation of a source video signal at multiple temporal, quality, and spatial resolutions. These resolutions are represented by distinct bitstream components that are created at each end-user encoder. System architecture and processes called SVC Thinning allow the separation of data into data used for prediction in other pictures and data not used for prediction in other pictures. SVC Thinning processes, which can be performed at video conferencing endpoints or at MCUs, can selectively remove or replace with fewer bits the data not used for prediction in other pictures from transmitted bit streams. This separation and selective removal or replacement of data for transmission allows a trade-off between scalability support (i.e. number of decodable video resolutions), error resiliency and coding efficiency.

Claims

1. A method for processing an input digital video signal, encoded in a scalable video coding format that supports one or more of spatial and quality scalability in order to produce an output digital video signal at an intended resolution, the method comprising:

for the output signal, replacing information in a plurality of scalable layers of at least one input video signal lower than a target layer that corresponds to the intended resolution such that information not required to decode the output video signal at the intended resolution is replaced by information that requires fewer bits in the output video signal, and wherein the output video signal with the replaced information is still conforming to the scalable video coding format, wherein a side information input data enables the replacement of the information not required to decode the output video signal at the intended resolution of the side information input data without fully parsing the entire input video signal.

2. The method of claim 1 wherein the scalable video coding format is H.264 SVC and wherein the information not required to decode the output video signal at the intended resolution:

comprises macroblocks that are not used for predicting the target layer, wherein the replacing comprises signaling each of the macroblocks are skipped;

and wherein encoding of neighboring blocks is modified if the information replacement affects the encoding of the neighboring blocks.

5. The method of claim 1 wherein the scalable video coding format is H.264 SVC and wherein the information not required to decode the output video signal at the intended resolution

comprises intra blocks where mode prediction is not used and either each of the intra blocks is not used for intra prediction by neighboring blocks, or none of the neighboring blocks are used for predicting the target layer, wherein the replacing comprises setting coefficients of each of the intra blocks to zero and accordingly modifying a coded block pattern of each of the intra blocks;

and wherein encoding of neighboring blocks is modified if the information replacement affects the encoding of the neighboring blocks.

6. The method of claim 1 wherein the scalable video coding format is H.264 SVC and wherein the information not required to decode the output video signal at the intended resolution

comprises inter blocks where no mode prediction or no motion prediction are used, wherein the replacing comprises setting motion information of each of the inter blocks to zero;

and wherein encoding of neighboring blocks is modified if the information replacement affects the encoding of the neighboring blocks.

7. The method of claim 1 wherein the scalable video coding format is H.264 SVC and wherein the information not required to decode the output video signal at the intended resolution

comprises inter blocks where residual prediction is not used, wherein the replacing comprises setting coefficients of each of the inter blocks to zero and accordingly modifying a coded block pattern of each of the inter blocks;

and wherein encoding of neighboring blocks is modified if the information replacement affects the encoding of the neighboring blocks.

3. A method for processing an input digital video signal, encoded in a scalable video coding format that supports one or more of spatial and quality scalability in order to produce an output digital video signal at an intended resolution, the method comprising:

for the output signal, removing information in a plurality of scalable layers of at least one input video signal lower than a target layer that corresponds to the intended resolution such that information not required to decode the output video signal at the intended resolution is removed in the output video signal, wherein a side information input data enables the elimination of the information not required to decode the output video signal at the intended resolution of the side information input data without fully parsing the entire input video signal.

4. The method of claim 3, wherein the information not required to decode the output video signal at the intended resolution

comprises macroblocks that are not used for predicting the target layer, wherein the removing comprises removing each of the macroblocks;

and wherein encoding of neighboring blocks is modified if the information replacement affects the encoding of the neighboring blocks.

8. The method of claim 3, wherein the information not required to decode the output video signal at the intended resolution

comprises intra blocks where mode prediction is not used and either each of the intra blocks is not used for intra prediction by neighboring blocks or none of the neighboring blocks are used for predicting the target layer, wherein the removing comprises inferring coefficients of each of the intra blocks to be zero for further prediction inside a layer corresponding to each of the intra blocks;

and wherein encoding of neighboring blocks is modified if the information replacement affects the encoding of the neighboring blocks.

9. The method of claim 3, wherein the information not required to decode the output video signal at the intended resolution

comprises inter blocks where no mode prediction or motion prediction are used, wherein the removing comprises removing motion information from each of the inter blocks and inferring motion vector differences to be 0 for further prediction inside a layer corresponding to each of the inter blocks;

and wherein encoding of neighboring blocks is modified if the information replacement affects the encoding of the neighboring blocks.

10. The method of claim 3, wherein the information not required to decode the output video signal at the intended resolution

comprises inter blocks where residual prediction is not used, wherein the removing comprises removing all syntax elements relating to residual coding and inferring the syntax elements to be 0 for prediction inside a layer corresponding to each of the inter blocks;

and wherein encoding of neighboring blocks is modified if the information replacement affects the encoding of the neighboring blocks.

11. A non-transitory computer readable medium comprising a set of instructions to direct a processor to:

process an input digital video signal, encoded in a scalable video coding format that supports one or more of spatial and quality scalability in order to produce an output digital video signal at an intended resolution, by;

for the output signal, replacing information in a plurality of scalable layers of at least one input video signal lower than a target layer that corresponds to the intended resolution such that information not required to decode the output video signal at the intended resolution is replaced by information that requires fewer bits in the output video signal, and wherein the output video signal with the replaced information is still conforming to the scalable video coding format, wherein a side information input data enables the replacement of the information not required to decode the output video signal at the intended resolution of the side information input data without fully parsing the entire input video signal.

12. The non-transitory computer readable medium of claim 11, wherein the scalable video coding format is H.264 SVC;

wherein the information not required to decode the output video signal at the intended resolution comprises macroblocks that are not used for predicting the target layer;

wherein the replacing comprises signaling each of the macroblocks are skipped; and

wherein encoding of neighboring blocks is modified if the information replacement affects the encoding of the neighboring blocks.

13. The non-transitory computer readable medium of claim 11, wherein the scalable video coding format is H.264 SVC;

wherein the information not required to decode the output video signal at the intended resolution comprises intra blocks where mode prediction is not used and either each of the intra blocks is not used for intra prediction by neighboring blocks, or none of the neighboring blocks are used for predicting the target layer;

wherein the replacing comprises setting coefficients of each of the intra blocks to zero and accordingly modifying a coded block pattern of each of the intra blocks; and

wherein encoding of neighboring blocks is modified if the information replacement affects the encoding of the neighboring blocks.

14. The non-transitory computer readable medium of claim 11, wherein the scalable video coding format is H.264 SVC;

wherein the information not required to decode the output video signal at the intended resolution comprises inter blocks where no mode prediction or no motion prediction are used;

wherein the replacing comprises setting motion information of each of the inter blocks to zero; and

wherein encoding of neighboring blocks is modified if the information replacement affects the encoding of the neighboring blocks.

15. The non-transitory computer readable medium of claim 11, wherein the scalable video coding format is H.264 SVC;

wherein the information not required to decode the output video signal at the intended resolution comprises inter blocks where residual prediction is not used;

wherein the replacing comprises setting coefficients of each of the inter blocks to zero and accordingly modifying a coded block pattern of each of the inter blocks;

and wherein encoding of neighboring blocks is modified if the information replacement affects the encoding of the neighboring blocks.

16. A non-transitory computer readable medium comprising a set of instructions to direct a processor to:

process an input digital video signal, encoded in a scalable video coding format that supports one or more of spatial and quality scalability in order to produce an output digital video signal at an intended resolution, by;

for the output signal, removing information in a plurality of scalable layers of at least one input video signal lower than a target layer that corresponds to the intended resolution such that information not required to decode the output video signal at the intended resolution is removed in the output video signal, wherein a side information input data enables the elimination of the information not required to decode the output video signal at the intended resolution of the side information input data without fully parsing the entire input video signal.

17. The non-transitory computer readable medium of claim 16, wherein the information not required to decode the output video signal at the intended resolution comprises macroblocks that are not used for predicting the target layer;

wherein the removing comprises removing each of the macroblocks; and

wherein encoding of neighboring blocks is modified if the information replacement affects the encoding of the neighboring blocks.

18. The non-transitory computer readable medium of claim 16, wherein the information not required to decode the output video signal at the intended resolution comprises intra blocks where mode prediction is not used and either each of the intra blocks is not used for intra prediction by neighboring blocks or none of the neighboring blocks are used for predicting the target layer;

wherein the removing comprises inferring coefficients of each of the intra blocks to be zero for further prediction inside a layer corresponding to each of the intra blocks; and

wherein encoding of neighboring blocks is modified if the information replacement affects the encoding of the neighboring blocks.

19. The non-transitory computer readable medium of claim 16, wherein the information not required to decode the output video signal at the intended resolution comprises inter blocks where no mode prediction or motion prediction are used;

wherein the removing comprises removing motion information from each of the inter blocks and inferring motion vector differences to be 0 for further prediction inside a layer corresponding to each of the inter blocks; and

wherein encoding of neighboring blocks is modified if the information replacement affects the encoding of the neighboring blocks.

20. The non-transitory computer readable medium of claim 16, wherein the information not required to decode the output video signal at the intended resolution comprises inter blocks where residual prediction is not used;

wherein the removing comprises removing all syntax elements relating to residual coding and inferring the syntax elements to be 0 for prediction inside a layer corresponding to each of the inter blocks; and

wherein encoding of neighboring blocks is modified if the information replacement affects the encoding of the neighboring blocks.