SYSTEM AND METHOD FOR SCALABLE AND LOW-DELAY VIDEOCONFERENCING USING SCALABLE VIDEO CODING

• US 20080211901A1
• Filed: 01/17/2008
• Published: 09/04/2008
• Est. Priority Date: 07/20/2005
• Status: Active Grant

• Current Assignee: Vidyo Incorporated (Enghouse Systems Limited)
• Original Assignee: Vidyo Incorporated (Enghouse Systems Limited)
• Inventors: Eleftheriadis, Alexandros, Shapiro, Ofer, Hong, Danny, Civanlar, Mehmet Reha
• Granted Patent: US 8,289,370 B2
• Time in Patent Office: Days
• US Class Current: 348/14.09
• CPC Class Codes: H04N 19/105 Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/146 Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/164 Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 19/187 the unit being a scalable video layer
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/36 Scalability techniques involving formatting the layers as a function of picture distortion after decoding, e.g. signal-to-noise [SNR] scalability
• CPC Class Codes: H04N 19/37 with arrangements for assigning different transmission priorities to video input data or to video coded data
• CPC Class Codes: H04N 19/573 Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 21/234327 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

Scalable video codecs are provided for use in videoconferencing systems and applications hosted on heterogeneous endpoints/receivers and network environments. The scalable video codecs provide a coded representation of a source video signal at multiple temporal, quality, and spatial resolutions.

Claims

1. A system for video communication between a plurality of endpoints over an electronic communications network and one or more servers, which network provides different quality of service and bandwidth channels linking the plurality of endpoints and servers, the channels including designated high reliability channels (HRCs) and low reliability channels (LRCs), the system comprising:

transmitting and receiving terminals disposed at the endpoints,

wherein at least one transmitting terminal is configured to prepare at least one scalably coded video signal for transmission to other terminals in a base layer and enhancement layers format, and to transmit at least the base layer over a designated HRC, and

wherein at least one receiving terminal is configured to decode the scalably coded video signal layers that are received over network channels including a designated HRC, and to reconstruct a video for local use by combining the decoded video signal layers, and

wherein the servers are configured to mediate the transfer of the scalably coded video signal layers transmitted by the transmitting terminal to a receiving terminal over electronic communications network channels leading to the latter terminal.

2. The system of claim 1 wherein at least one terminal is configured to access at least one of:

a live video signal for encoding and transmission;

a stored video signal for encoding and transmission;

a synthesized video signal for encoding and transmission and a pre-coded video signal for transmission.

3. An endpoint terminal for video communication with other endpoints over one or more servers disposed in an electronics communication network, which network provides different quality of service and bandwidth channels linking the plurality of endpoints, the channels including designated HRCs, the endpoint terminal comprising:

at least one scalable video encoder, which is configured to scalably encode at least one video signal in a base layer and enhancement layers format; and

a packet multiplexer, which is configured to multiplex layers of the video signal encoded in the base layer and enhancement layers format for transmission over the electronics communication network,

wherein the endpoint terminal is configured to designate at least the base layer from the base layer and enhancement layers of the video signal for transmission over the designated HRCs via a network interface controller.

4. The endpoint terminal of claim 3, further comprising an audio signal encoder whose output is connected to the packet multiplexer.

5. The terminal of claim 3, wherein the scalable video encoder is a motion-compensated, block-based codec comprised of:

a frame memory where one or more decoded frames are stored for future reference; and

a reference controller configured to select the picture type (I, P, or B) as well as the pictures in the frame memory that will be used as prediction references for the picture currently encoded,

wherein the codec is further configured to perform picture prediction using threads as a means of implementing temporal scalability layers.

6. The terminal of claim 5, wherein the scalable video encoder is configured to create one continuous prediction chain path for the base layer.

7. The terminal of claim 5 where the threads are picture threads comprising:

a base layer thread further comprised of pictures that are a number of pictures apart, and in which temporal prediction is performed using one or more previous pictures of the same thread; and

a temporal enhancement layer thread that is comprised of the remaining pictures, and in which prediction is performed from one or more preceding base layer pictures and/or one or more preceding temporal enhancement layer pictures.

8. The terminal of claim 5 where the picture threads are comprised of:

a base layer thread further comprised of pictures that are a fixed number of pictures apart, and in which temporal prediction is performed using the immediately previous frame of the same thread;

a first temporal enhancement layer thread that is comprised of frames that are half way between those of the base layer thread, and in which prediction is performed from either the immediately preceding base layer picture or the immediately preceding first temporal enhancement layer pictures; and

a second temporal enhancement layer thread that is comprised of the remaining pictures, and in which prediction is performed from either the immediately preceding second temporal enhancement layer pictures, the immediately preceding first temporal enhancement layer picture, or the immediately preceding base layer thread picture.

9. The terminal of claim 5, wherein the scalable video encoder is configured to code the base temporal layer frames using a finer quantization than quantizations used for other temporal layers, whereby the said base layer is coded more accurately than the other layers.

10. The terminal of claim 5, wherein the scalable video encoder is configured to create at least one prediction chain which terminates at an enhancement layer.

11. The terminal of claim 5, wherein the temporally predictive codec further comprises an SNR quality scalability layer encoder.

12. The terminal of claim 11 wherein the SNR quality scalability enhancement layer encoder is configured with an input the residual coding error of the base layer obtained by subtracting the decoded base layer frames from the original frames and applying a positive offset and then encoding the difference in the same manner as the base layer encoder.

13. The terminal of claim 11, wherein the SNR enhancement layers encoder is further configured to use a prediction path which is different than the prediction path for the base layer or a lower enhancement layer.

14. The terminal of claim 12, wherein the SNR enhancement layers encoder is further configured to drop the DC components of direct cosine transform (DCT) coefficients when predicting a coded video frame for a SNR quality enhancement layer.

15. The terminal of claim 12, wherein SNR enhancement layers encoder is further configured to quantize the DC and surrounding AC DCT coefficients at coarser level than the remaining DCT coefficients when encoding a video frame for an SNR quality enhancement layer.

16. The terminal of claim 11, wherein the SNR quality scalability layer decoder at the receiving endpoint is configured to display the decoded base layer frame in a desired reduced resolution by applying low-pass filtering and downsampling after decoding.

17. The terminal of claim 11, wherein the SNR quality scalability enhancement layer codec comprises an H.264 SVC FGS codec with threading.

18. The terminal of claim 17, wherein the spatial scalability layers codec comprises an H.264 SVC FGS codec that is configured to use a weighted average of previous enhancement layer pictures and a current base layer picture in a motion compensated prediction, wherein the weighting is dynamically varied to include the zero value at which value the prediction chain is terminated, whereby drift is eliminated.

19. The terminal of claim 11, wherein the SNR quality scalability layer encoder is configured to encode the differences in DCT coefficients before and after quantization by requantizing the differences and applying entropy coding to the requantized differences.

20. The terminal of claim 5, wherein the temporally predictive codec further comprises a spatial scalability layer encoder, and which is configured to low-pass filter and downsample the original input signal, wherein the lower resolution can be different than the intended display resolution, and use it as input to the base layer encoder.

21. The terminal of claim 20, wherein the spatial scalability layers encoder is configured so that a prediction path for an enhancement layer is different from that the prediction path for the base layer or a lower enhancement layer.

22. The terminal of claim 20, where the spatial scalability layers encoder is configured to:

upsample the decoded low resolution signal to the original input signal resolution;

subtract the original input signal from the upsampled decoded low resolution signal to obtain a difference signal;

apply an offset to the difference signal; and

encode the offset difference signal.

23. The terminal of claim 22, wherein the spatial scalability layers encoder is configured so that the DC and surrounding DCT AC coefficients can be quantized more coarsely than the remaining DCT AC coefficients.

24. The terminal of claim 20 wherein the spatial scalability layer encoder is configured to use bi-predictive coding when predicting a high resolution video frame, and in which the first reference picture is a decoded past full resolution picture and the second reference picture is obtained by first encoding and decoding the downsampled base layer signal, and then upsampling it to the original resolution.

25. The terminal of claim 24 wherein the spatial scalability layer encoder is comprised of an H.264 AVC encoder with bi-predictive coding, in which the upsampled decoded base layer frames are inserted as additional reference frames, and in which the temporal and spatial direct modes of motion vector prediction are used to increase compression efficiency.

26. The terminal of claim 5 comprising a base layer encoder, which further comprises at least one of a SNR quality layers encoder, spatial scalability layers encoder, and temporal enhancement layers encoder.

34. The terminal of claim 5, wherein the scalable video encoder is configured to encode the input signal in two or more spatial and/or quality resolutions that may be transmitted simultaneously.

35. The terminal of claim 3, wherein the scalable coding structure may change dynamically depending on network conditions or indication of preferences by receiving endpoints, in any of the scalability dimensions.

27. An endpoint terminal for video communication with other endpoints over one or more servers disposed in an electronics communication network, which network provides different quality of service and bandwidth channels linking the plurality of endpoints, the channels including designated HRCs, the endpoint terminal comprising:

a scalable video decoder or decoders, which is configured to scalably decode one or more video signals in a base layer and enhancement layers format; and

a packet demultiplexer, which is configured to demultiplex layers of the video signal encoded in the base layer and enhancement layers format after reception over the electronics communication network via a network interface controller.

28. The terminal of claim 27, wherein the decoder is comprised of an SNR quality scalability decoder.

29. The terminal of claim 28, where the SNR quality scalability decoder is configured to display the decoded base layer frame in a desired reduced resolution by applying low-pass filtering and downsampling after decoding.

30. The terminal of claim 28 wherein the SNR quality scalability enhancement layer decoder is configured to add to the decoded base layer frames the decoded residual error carried by the enhancement layer data after subtracting a positive offset.

31. The terminal of claim 27, wherein the decoder further comprises a spatial scalability decoder.

32. The terminal of claim 31, where the spatial scalability layers decoder is configured to:

upsample the decoded base resolution signal to the enhancement layer resolution;

decode the offset difference signal carried by the enhancement layer;

subtract an offset from the decoded enhancement layer signal and add the result to the upsampled decoded base resolution signal.

33. The terminal of claim 31, where the spatial scalability layer decoder comprises an H.264 AVC decoder with bi-predictive coding support, in which the upsampled decoded base layer frames are inserted as additional reference frames.

36. A method for communication between a plurality of endpoints over an electronic communications network and one or more servers, which network provides different quality of service and bandwidth channels linking the plurality of endpoints, the channels including designated HRCs, the method comprising:

scalably coding a video signal in a base layer and enhancement layers format;

multiplexing layers of the video signal for transmission over the electronics communication network,

transmitting at least the base layer from the base layer and enhancement layers of the video signal for transmission over the designated HRCs.

37. The method of claim 36, wherein multiplexing layers of the video signal for transmission over the electronics communication network further comprising multiplexing an audio signal for transmission over the electronics communication network.

71. Computer readable media comprising a set of instructions to perform the steps recited in at least one of claims 36-70.

38. A method for communicating encoded video signal pictures, which are scalably coded in a base and enhancement layers format, between a plurality of endpoints over an electronic communications network, the method comprising:

selecting a picture type (I, P, or B) for the picture currently being encoded as well as a prediction reference picture from the decoded pictures stored in a frame memory; and

creating a temporal scalability layer by performing picture prediction using threads.

39. The method of claim 38 further comprising creating a continuous prediction chain path for the base layer.

40. The method of claim 38 wherein the picture threads include a base layer thread that includes pictures that are a number of pictures apart, and the method further comprises performing temporal prediction in each base layer picture using one or more of the preceding pictures of the said base layer picture,

wherein a temporal enhancement layer thread includes the remaining pictures, and the method further comprises performing prediction in each enhancement layer picture using either one or more of the preceding base layer pictures, or one or more preceding temporal enhancement layer pictures,

41. The method of claim 38 wherein the picture threads include a base layer thread that includes pictures that are a number of pictures apart, and the method further comprises performing temporal prediction using the immediately previous frame of the same,

wherein a first temporal enhancement layer thread includes that frames are half way between those of the base layer thread, and the method further comprises performing prediction from either the immediately preceding base layer frame or an immediately preceding first temporal enhancement layer thread picture, and

wherein a second temporal enhancement layer thread includes the remaining frames, and the method further comprises performing temporal prediction using a frame from either the immediately preceding second temporal enhancement layer thread picture, the immediately preceding first temporal enhancement layer thread picture, or the immediately preceding base layer thread picture.

42. The method of claim 38, further comprising coding the base temporal layer frames using a finer quantization than quantizations used for other temporal layers, whereby the said base layer is coded more accurately than the other layers.

43. The method of claim 38, further comprising creating at least one prediction chain, which terminates at an enhancement layer.

44. The method of claim 38, wherein scalably coding a temporal scalability layer by performing picture prediction using threads further comprises coding a SNR quality scalability enhancement layer.

45. The method of claim 44, wherein coding a SNR quality scalability enhancement layer comprises applying a positive offset to a residual coding error of the base layer obtained by subtracting the decoded base layer frames from the original frames and then encoding the difference in the same manner as coding the base layer.

46. The method of claim 44, wherein coding a SNR quality scalability enhancement layer comprises using a prediction path which is different than the prediction path for the base layer or a lower enhancement layer.

47. The method of claim 45, wherein coding a SNR quality scalability enhancement layer comprises dropping the DC components of Discrete Cosine Transform (DCT) coefficients when encoding a picture for the SNR quality scalability enhancement layer.

48. The method of claim 45, wherein coding a SNR quality scalability enhancement layer comprises quantizing the DC and surrounding AC DCT coefficients at coarser level than the remaining DCT coefficients when encoding a video frame for the SNR quality scalability enhancement layer.

49. The method of claim 44, wherein coding a SNR quality scalability enhancement layer further comprises at the receiving endpoint displaying the decoded base layer frame in a desired reduced resolution by applying low-pass filtering and downsampling after decoding.

50. The method of claim 44, wherein coding a SNR quality scalability enhancement layer further comprises an H.264 SVC FGS codec with threading.

51. The method of claim 50, further comprising using an H.264 SVC FGS codec that is configured to use a weighted average of previous enhancement layer pictures and a current base layer picture in a motion compensated prediction, wherein the weighting is dynamically varied to include the zero value at which value the prediction chain is terminated, whereby drift is eliminated.

52. The method of claim 44, wherein coding a SNR quality scalability enhancement layer comprises encoding the differences in DCT coefficients before and after quantization by requantizing the differences and applying entropy coding to the requantized differences.

53. The method of claim 38, wherein coding the temporal quality layers further comprises encoding a spatial scalability layer by applying a low-pass filter and downsampling the original input signal, wherein the lower resolution can be different than the intended display resolution, and encoding the downsampled signal in the same manner as the base layer.

54. The method of claim 53, wherein encoding a spatial scalability layer comprises using a prediction path for an enhancement layer is different from the prediction path for the base layer or a lower enhancement layer.

55. The method of claim 53, wherein encoding a spatial scalability layer comprises:

upsampling the decoded low resolution signal to the original input signal resolution;

subtracting the original input signal from the upsampled decoded low resolution signal to obtain a difference signal;

applying an offset to the difference signal; and

encoding the offset difference signal.

56. The method of claim 55, wherein encoding a spatial scalability layer comprises:

quantizing the DC and surrounding DCT AC coefficients more coarsely than the remaining DCT AC coefficients.

57. The method of claim 53, wherein encoding a spatial scalability layer comprises:

using bi-predictive coding when predicting a high resolution video frame, and in which the first reference picture is a decoded past full resolution picture and the second reference picture is obtained by first encoding and decoding the downsampled base layer signal and then upsampling it to the original resolution.

58. The method of claim 57, wherein encoding a spatial scalability layer comprises:

comprises using an H.264 AVC encoder with bi-predictive coding, in which the upsampled decoded base layer frames are inserted as additional reference frames, and in which the temporal and spatial direct modes of motion vector prediction are used to increase compression efficiency.

59. The method of claim 38 comprising using a base layer encoder, which further comprises at least one of a SNR quality layers encoder, spatial scalability layers encoder, and temporal enhancement layers encoder.

67. The method of claim 38, wherein scalably coding a video signal comprises encoding the signal in two or more spatial and/or quality resolutions that may be transmitted simultaneously.

68. The method of claim 38, wherein the scalable coding structure may change dynamically depending on network conditions or indication of preferences indicated by receiving endpoints, in any of the scalability dimensions.

60. A method for communicating encoded video signal pictures, which are scalably coded in a base and enhancement layers format, between a plurality of endpoints and one or more servers over an electronic communications network, the method comprising using:

a scalable video decoder or decoders, which is configured to scalably decode one or more video signals in a base layer and enhancement layers format; and

a packet demultiplexer, which is configured to demultiplex layers of the video signal encoded in the base layer and enhancement layers format after reception over the electronics communication network via a network interface controller.

61. The method of claim 60, wherein the decoder is comprised of an SNR quality scalability decoder.

62. The method of claim 61, further comprising using the SNR quality scalability decoder to display the decoded base layer frame in a desired reduced resolution by applying low-pass filtering and downsampling after decoding.

63. The method of claim 61, further comprising using the SNR quality scalability decoder to add to the decoded base layer frames the decoded residual error carried by the enhancement layer data after subtracting a positive offset.

64. The method of claim 60, wherein the decoder further comprises a spatial scalability decoder.

65. The method of claim 64, further comprising using the spatial scalability layers decoder to:

upsample the decoded base resolution signal to the enhancement layer resolution;

decode the offset difference signal carried by the enhancement layer;

subtract an offset from the decoded enhancement layer signal and add the result to the upsampled decoded base resolution signal.

66. The method of claim 64, wherein the spatial scalability layer decoder comprises an H.264 AVC decoder with bi-predictive coding support, the method further comprising inserting upsampled decoded base layer frames as additional reference frames.

69. A method for video communication between a plurality of endpoints over an electronic communications network and one or more servers, which network provides different quality of service and bandwidth channels linking the plurality of endpoints and servers, the channels including designated high reliability channels (HRCs) and low reliability channels (LRCs), the method comprising:

disposing transmitting and receiving terminals at the endpoints,

configuring at least one transmitting terminal to prepare at least one scalably coded video signal for transmission to other terminals in a base layer and enhancement layers format, and to transmit at least the base layer over a designated HRC,

configuring at least one receiving terminal to decode the scalably coded video signal layers that are received over network channels including a designated HRC, and to reconstruct a video for local use by combining the decoded video signal layers, and

configuring the servers to mediate the transfer of the scalably coded video signal layers transmitted by the transmitting terminal to a receiving terminal over electronic communications network channels leading to the latter terminal

70. The method of claim 69, wherein configuring at least one transmitting terminal comprises configuring the terminal to access at least one of:

a live video signal for encoding and transmission;

a stored video signal for encoding and transmission;

a synthesized video signal for encoding and transmission and a pre-coded video signal for transmission.